python中的Selenium无法定位元素错误404

Question

我尝试在 python 中使用 selenium 单击“加载更多”按钮以显示特定网页中的所有评论，但是，在 selenium 中查找按钮元素时遇到一些问题，该元素始终返回 ** 错误：404 - 否此类元素：无法定位元素**

import requests
from urllib.parse import urljoin
import time
from bs4 import BeautifulSoup as bs
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)


url = 'https://www.walgreens.com/store/c/allegra-adult-24hr-tablet-180-mg,-allergy-relief/ID=300409806-product'

driver.get(url)
time.sleep(5)

while True:
    try:
        # loadMoreBtn1 = driver.find_element(By.CSS_SELECTOR , 'button.bv-rnr__sc-16j1lpy-3 bv-rnr__sc-17t5kn5-1.jSgrbb.gSrzYA')
        # loadMoreBtn2 = driver.find_element(By.XPATH,'//*[@class="bv-rnr__sc-16j1lpy-3 bv-rnr__sc-17t5kn5-1 jSgrbb gSrzYA"]')
        loadMoreBtn3 = driver.find_element(By.CLASS_NAME,'bv-rnr__sc-16j1lpy-3 bv-rnr__sc-17t5kn5-1 jSgrbb gSrzYA')
        loadMoreBtn3.click()
        time.sleep(2)
    except:
        break

我已经尝试通过 css、xpath 和类名查找元素，但没有成功。 我检查了网页，似乎类名也已修复

<div class="bv-rnr__sc-17t5kn5-0 gcGJRh">
  <button aria-label="Load More , This action will load more reviews on screen" class="bv-rnr__sc-16j1lpy-3 bv-rnr__sc-17t5kn5-1 jSgrbb gSrzYA">Load More</button>
</div>

我还尝试了 CTRL+F 来测试页面源中的 css 和 xpath，它也返回零匹配。

Answer 1

我会简化代码并使用他们的 Ajax API 来获得更多评论：

import requests

url = "https://api.bazaarvoice.com/data/reviews.json"

params = {
    "resource": "reviews",
    "action": "REVIEWS_N_STATS",
    "filter": [
        "productid:eq:300409806",   # <--- change this to product ID
        "contentlocale:eq:en,en_AU,en_CA,en_GB,en_US,en_US",
        "isratingsonly:eq:false",
    ],
    "filter_reviews": "contentlocale:eq:en,en_AU,en_CA,en_GB,en_US,en_US",
    "include": "authors,products,comments",
    "filteredstats": "reviews",
    "Stats": "Reviews",
    "limit": "30",
    "offset": "8",
    "limit_comments": "3",
    "sort": "submissiontime:desc",
    "passkey": "tpcm2y0z48bicyt0z3et5n2xf",
    "apiversion": "5.5",
    "displaycode": "2001-en_us",
}

for page in range(3):  # <-- change this for required number of pages
    params["offset"] = 30 * page

    data = requests.get(url, params=params).json()
    for r in data["Results"]:
        print(r["UserNickname"], " -> ", r["ReviewText"])
        print()

打印：

...

AllergyMan2  ->  [This review was collected as part of a promotion.] This was the first time I used Allegra and I'm thoroughly impressed with this product. I've used other allergy pills in the past and this one takes the cake, it's fast acting for all your allergy needs. I recommend if your like me and you have used them all and just want one that works go get some Allegra 24 hour and you'll be in great hands.

pinkdiemond1  ->  [This review was collected as part of a promotion.] I got the chance to try Allegra 24Hr 60ct Tablets thru Home Tester.  The pills were easy to swallow and fast acting.  I did not get sleep after a few hours of ingestion.  Yes I would recommend Allegra 24Hr 60ct Tablets to my friends and family.

Rachael7  ->  [This review was collected as part of a promotion.] I have seasonal allergies and my first pick is always Allegra. I love that it works within an hour of using and I feel great all day. A plus that it’s non drowsy. The only suggestion I’d like to make is: have the pill me smaller. Other than that, it’s great to take for those annoying allergy symptoms.

...