美丽的汤找到所有点击按钮后找不到div

Question

我正在尝试废弃这个网站。 当我尝试废弃与汽车设备网格相关的数据时，会出现此问题。我只能提取页面中已显示的元素，即使我单击按钮来显示所有元素也是如此。 总之，我无法获取按钮单击事件后显示的元素。 我注意到在浏览器的检查元素选项卡上标记为“事件”，但我不知道如何在 bs4 中使用该信息。

这是代码：

from bs4 import BeautifulSoup
import requests
import pandas as pd
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("prefs", {"profile.default_content_settings.cookies": 2})
driver = webdriver.Chrome(options=chrome_options)

url='https://www.autoscout24.it/ricerca-avanzata?sort=standard&desc=0&ustate=N,U&atype=C&cy=I&mmm=13||&lat=45.5714&lon=12.10799&zip=30037-scorz%C3%A8&zipr=100&source=homepage_search-mask'
driver.get(url)
driver.find_element(By.CLASS_NAME,'_consent-accept_1fb0r_111').click()

for i in range(2):
    driver.find_element(By.TAG_NAME,'body').send_keys(Keys.PAGE_DOWN)
driver.find_element(By.XPATH, "/html/body/div[1]/div[2]/div/div/div/div[3]/div[2]/div[2]/div/button").click()

h=requests.get(url).text
soup=BeautifulSoup(h, 'html.parser')
l=soup.find_all("div",{"class": "OptionalEquipmentFilter_optionContainer__HsU8p"})
print(len(l))
accl=[]
for i in l:
    accl.append(i.span.text)
print(accl)

我不知道如何改进。 我怀疑转移到剪贴会话的页面源在按钮单击事件后没有更新。

感谢您的宝贵时间:)

Answer 1

您可以尝试直接使用

requests

:

加载带有结果的页面

import requests
from bs4 import BeautifulSoup

url = "https://www.autoscout24.it/lst/bmw/30037-scorz%c3%a8"

params = {
    "atype": "C",
    "cy": "I",
    "desc": "0",
    "lat": "45.5714",
    "lon": "12.10799",
    "sort": "standard",
    "source": "detailsearch",
    "ustate": "N,U",
    "zipr": "100",
}

soup = BeautifulSoup(requests.get(url, params=params).content, "html.parser")

for article in soup.select("article"):
    title = article.h2.text
    price = article.select_one('[data-testid="regular-price"]').text
    print(title)
    print(price)
    print("-" * 80)

打印：

...
--------------------------------------------------------------------------------
BMW 316 - 316d Sport
€ 16.500,-
--------------------------------------------------------------------------------
BMW 320 2.0 PER COMMERCIANTI
€ 1.800,-
--------------------------------------------------------------------------------
BMW 318 i Business Advantage
€ 17.800,-
--------------------------------------------------------------------------------
BMW M2 M240i 340cv AUT VOLANTE CARBONIO MPERFORMANCE
€ 37.750,-
--------------------------------------------------------------------------------
...