如何从HTML中提取链接？

Question

我正在尝试在SF编年史中获取此类别中每篇文章的链接，但我不确定我应该从哪里开始提取URL。这是我目前的进展：

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = 'https://www.sfchronicle.com/local/'

# opening up connection, grabbing the page
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()

# html parsing
page_soup = soup(page_html, "html.parser")
zone2_container = page_soup.findAll("div",{"class":"zone zone-2"})
zone3_container = page_soup.findAll("div",{"class":"zone zone-3"})
zone4_container = page_soup.findAll("div",{"class":"zone zone-4"})
right_rail_container = page_soup.findAll("div",{"class":"right-rail"})

我想要的所有链接都位于zone2-4_container和right_rail_container中。

Answer 1

您可以使用以下代码获取所有链接：

all_zones = [zone2_container, zone3_container, zone4_container, right_rail_container]
urls = []
for i in all_zones:
    links = i[0].findAll('a')
    for link in links:
        urls.append(link['href'])

---

我已将所有列表合并到一个列表中，但您也可以定义一个函数来实现相同的功能。

def get_urls(zone):
    urls = []
    for i in zone:
        links = i.findAll('a')
        for link in links:
            urls.append(link['href'])
    return urls 

get_urls(zone2_container)

Answer 2

现在听起来你基本上都想要所有文章链接，在这种情况下你可以使用带有运算符的attribute = value css选择器来定位其值包含子字符串'href'的article属性。

import requests
from bs4 import BeautifulSoup as bs
from urllib.parse import urljoin
base = 'https://www.sfchronicle.com/'
url = 'https://www.sfchronicle.com/local/'
res = requests.get(url)
soup = bs(res.content, 'lxml')
links = [urljoin(base,link['href']) for link in soup.select('[href*=article]')]
print(links)
print(len(links))