我正在尝试废弃这个网站。 当我尝试废弃与汽车设备网格相关的数据时,会出现此问题。我只能提取页面中已显示的元素,即使我单击按钮来显示所有元素也是如此。 总之,我无法获取按钮单击事件后显示的元素。 我注意到在浏览器的检查元素选项卡上标记为“事件”,但我不知道如何在 bs4 中使用该信息。
这是代码:
from bs4 import BeautifulSoup
import requests
import pandas as pd
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("prefs", {"profile.default_content_settings.cookies": 2})
driver = webdriver.Chrome(options=chrome_options)
url='https://www.autoscout24.it/ricerca-avanzata?sort=standard&desc=0&ustate=N,U&atype=C&cy=I&mmm=13||&lat=45.5714&lon=12.10799&zip=30037-scorz%C3%A8&zipr=100&source=homepage_search-mask'
driver.get(url)
driver.find_element(By.CLASS_NAME,'_consent-accept_1fb0r_111').click()
for i in range(2):
driver.find_element(By.TAG_NAME,'body').send_keys(Keys.PAGE_DOWN)
driver.find_element(By.XPATH, "/html/body/div[1]/div[2]/div/div/div/div[3]/div[2]/div[2]/div/button").click()
h=requests.get(url).text
soup=BeautifulSoup(h, 'html.parser')
l=soup.find_all("div",{"class": "OptionalEquipmentFilter_optionContainer__HsU8p"})
print(len(l))
accl=[]
for i in l:
accl.append(i.span.text)
print(accl)
我不知道如何改进。 我怀疑转移到剪贴会话的页面源在按钮单击事件后没有更新。
感谢您的宝贵时间:)
您可以尝试直接使用
requests
: 加载带有结果的页面
import requests
from bs4 import BeautifulSoup
url = "https://www.autoscout24.it/lst/bmw/30037-scorz%c3%a8"
params = {
"atype": "C",
"cy": "I",
"desc": "0",
"lat": "45.5714",
"lon": "12.10799",
"sort": "standard",
"source": "detailsearch",
"ustate": "N,U",
"zipr": "100",
}
soup = BeautifulSoup(requests.get(url, params=params).content, "html.parser")
for article in soup.select("article"):
title = article.h2.text
price = article.select_one('[data-testid="regular-price"]').text
print(title)
print(price)
print("-" * 80)
打印:
...
--------------------------------------------------------------------------------
BMW 316 - 316d Sport
€ 16.500,-
--------------------------------------------------------------------------------
BMW 320 2.0 PER COMMERCIANTI
€ 1.800,-
--------------------------------------------------------------------------------
BMW 318 i Business Advantage
€ 17.800,-
--------------------------------------------------------------------------------
BMW M2 M240i 340cv AUT VOLANTE CARBONIO MPERFORMANCE
€ 37.750,-
--------------------------------------------------------------------------------
...