使用 BeautifulSoup 抓取所有链接

Question

我试图从页面上抓取所有比赛报告链接，但有“加载更多”按钮，而且我不想使用硒。有没有解决方案可以收集所有不使用硒的链接。 预先感谢。

这是我尝试过的：

 from bs4 import BeautifulSoup as bs
 import requests
 r =requests.get('https://www.iplt20.com/news/match-reports')
 soup = bs(r.text,'lxml')

 for match in soup.find_all('div',class_='latest-slider-wrap 
position-relative'):
      links = match.find('a')
      print(links['href'])

Answer 1

尝试：

import requests
from bs4 import BeautifulSoup

url = "https://www.iplt20.com/news/match-reports"

soup = BeautifulSoup(requests.get(url).content, "html.parser")

for a in soup.select("#div-match-report a:has(li)"):
    print(a["href"])

打印：

https://www.iplt20.com/news/4014/tata-ipl-2024-match-11-lsg-vs-pbks-match-report
https://www.iplt20.com/news/4012/tata-ipl-2024-match-10-rcb-vs-kkr-match-report
https://www.iplt20.com/news/4011/tata-ipl-2024-match-09-rr-vs-dc-match-report
https://www.iplt20.com/news/4009/tata-ipl-2024-match-08-srh-vs-mi-match-report
https://www.iplt20.com/news/4007/tata-ipl-2024-match-07-csk-vs-gt-match-report
https://www.iplt20.com/news/4006/tata-ipl-2024-match-06-rcb-vs-pbks-match-report
https://www.iplt20.com/news/4004/tata-ipl-2024-match-05-gt-vs-mi-match-report
https://www.iplt20.com/news/4003/tata-ipl-2024-match-04-rr-vs-lsg-match-report
https://www.iplt20.com/news/4001/tata-ipl-2024-match-03-kkr-vs-srh-match-report
https://www.iplt20.com/news/4000/tata-ipl-2024-match-02-pbks-vs-dc-match-report
https://www.iplt20.com/news/3999/tata-ipl-2024-match-01-csk-vs-rcb-match-report
https://www.iplt20.com/news/3976/tata-ipl-2023-final-csk-vs-gt-match-report
https://www.iplt20.com/news/3974/tata-ipl-2023-qualifier-2-gt-vs-mi-match-report
https://www.iplt20.com/news/3973/tata-ipl-2023-eliminator-lsg-vs-mi-match-report
https://www.iplt20.com/news/3972/tata-ipl-2023-qualifier-1-gt-vs-csk-match-report
https://www.iplt20.com/news/3971/tata-ipl-2023-match-70-rcb-vs-gt-match-report
https://www.iplt20.com/news/3970/tata-ipl-2023-match-69-mi-vs-srh-match-report
https://www.iplt20.com/news/3969/tata-ipl-2023-match-68-kkr-vs-lsg-match-report
https://www.iplt20.com/news/3968/tata-ipl-2023-match-67-dc-vs-csk-match-report
https://www.iplt20.com/news/3967/tata-ipl-2023-match-66-pbks-vs-rr-match-report
https://www.iplt20.com/news/3966/tata-ipl-2023-match-65-srh-vs-rcb-match-report