无法使用python硒刮擦某些DIV标签

问题描述 投票:0回答:3

我正在尝试从本网站中提取有关某个主题(例如机器学习)的文章(它们的标题/链接)。 https://www.semanticscholar.org/search?q=machine%20learning&sort=relevance

我需要访问的div标签是嵌套在其他几个div标签下的。

这是我到目前为止尝试过的。我得到空名单。任何帮助表示赞赏。

import time
from selenium import webdriver

# Get all the paper url in the search result
def paper_crawler():
    driver = webdriver.Firefox('path')
    driver.get ('https://www.semanticscholar.org/search?q=machine%20learning&sort=relevance&fos=chemistry')
    result_counts = driver.find_elements_by_xpath('//*[@class="result-count"]')
    print(result_counts)
    for item in result_counts:
        count = item.text
        print(count)
    #search_result_urls = driver.find_elements_by_xpath('.//div[contains(@class,"result-page")]/article/header/div/a')
    search_result_urls = driver.find_elements_by_xpath('//*[@class="result-page"]/article/header/div/a')
    print(search_result_urls)
    for item in search_result_urls:
        paper_url =  item.get_attribute('href')
        print(paper_url)
    search_result_titles = driver.find_elements_by_xpath('//*[@class="result-page"]/article/header/div/a/span')
    for item in search_result_titles:
        paper_title = item.text
        print(paper_title)
    time.sleep(2)

if __name__ == '__main__':
    paper_crawler () 
python selenium web-scraping
3个回答
1
投票

更好地使用API,让您的生活更轻松。现在,用汤来解析任何内容。

import requests
from bs4 import BeautifulSoup


data = {
    "queryString": "machine learning",
    "page": 1,
    "pageSize": 10,
    "sort": "relevance",
    "authors": [],
    "coAuthors": [],
    "venues": [],
    "yearFilter": None,
    "requireViewablePdf": False,
    "publicationTypes": [],
    "externalContentTypes": []
}
r = requests.post(
    'https://www.semanticscholar.org/api/1/search', json=data).json()

print(r)

0
投票

当您开始搜索元素时,页面已加载但未完全呈现。

之后的“ time.sleep(5)”

driver.get('https://www.semanticscholar.org/search?q=machine%20learning&sort=relevance&fos=chemistry')应该可以作为一种快速的解决方法。

为了获得更好,更强大的解决方案,您应该等待result_counts大于0几秒钟,或者该页面为错误页面(https://www.semanticscholar.org/search?q=learning333&sort=relevance&fos=chemistry)。


0
投票

要提取文章的TitleHREF属性,必须为visibility_of_all_elements_located()引入WebDriverWait,并且可以使用以下Locator Strategies

  • 代码块:

    from selenium import webdriver
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    
    options = webdriver.ChromeOptions() 
    options.add_argument("start-maximized")
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)
    driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
    driver.get('https://www.semanticscholar.org/search?q=machine%20learning&sort=relevance&fos=chemistry')
    my_titles = [my_elem.text for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[data-selenium-selector='title-link']>span")))]
    my_hrefs = [my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[data-selenium-selector='title-link']")))]
    for i,j in zip(my_titles, my_hrefs):
        print("{} link is {}".format(i, j))
    driver.quit()
    
  • 控制台输出:

    UCI Repository of Machine Learning Databases link is https://www.semanticscholar.org/paper/UCI-Repository-of-Machine-Learning-Databases-Blake/e068be31ded63600aea068eacd12931efd2a1029
    Energy landscapes for machine learning. link is https://www.semanticscholar.org/paper/Energy-landscapes-for-machine-learning.-Ballard-Das/735d4099d3be0d919ddedb054043e6763205e0f7
    Finding Nature′s Missing Ternary Oxide Compounds Using Machine Learning and Density Functional Theory. link is https://www.semanticscholar.org/paper/Finding-Nature%E2%80%B2s-Missing-Ternary-Oxide-Compounds-Hautier-Fischer/e3ab9e1162fc8f63d215dfdb21801ef5e1fde7b5
    Distributed secure quantum machine learning link is https://www.semanticscholar.org/paper/Distributed-secure-quantum-machine-learning-Sheng-Zhou/ef944614bfc82b1dedfea19ff249a97ceea5ad90
    Neural-Symbolic Machine Learning for Retrosynthesis and Reaction Prediction. link is https://www.semanticscholar.org/paper/Neural-Symbolic-Machine-Learning-for-Retrosynthesis-Segler-Waller/71cc9eefb17d7c4d1062162523b5fdad7ca66a2a
    Transferable Machine-Learning Model of the Electron Density link is https://www.semanticscholar.org/paper/Transferable-Machine-Learning-Model-of-the-Electron-Grisafi-Fabrizio/f809258b65a00a06f9584e76620e6c6395cf81eb
    Crystal structure representations for machine learning models of formation energies link is https://www.semanticscholar.org/paper/Crystal-structure-representations-for-machine-of-Faber-Lindmaa/1bdca98dc8c730ee92d5b19d2973a5bf461a500a
    Machine learning for quantum mechanics in a nutshell link is https://www.semanticscholar.org/paper/Machine-learning-for-quantum-mechanics-in-a-Rupp/29b9ff8f4a26acc90e6182e1e749f15f688bc7cf
    Machine-Learning-Augmented Chemisorption Model for CO2 Electroreduction Catalyst Screening. link is https://www.semanticscholar.org/paper/Machine-Learning-Augmented-Chemisorption-Model-for-Ma-Li/d6f30032c8fac43a8eabf2b67d2e84db6d3d0409
    Adaptive machine learning framework to accelerate ab initio molecular dynamics link is https://www.semanticscholar.org/paper/Adaptive-machine-learning-framework-to-accelerate-Botu-Ramprasad/c9934d684fcc0b8ac6ed25b34d96e726cf2d7b99
    
© www.soinside.com 2019 - 2024. All rights reserved.