如何一次打印美丽汤的所有结果？

Question

我有一个Twitter用户名列表。我需要获得他们的关注者数量。我使用了BS和请求。但是，我每次只收到一个帐户。

from bs4 import BeautifulSoup
import requests
import pandas as pd
purcsv = pd.read_csv('pureeng.csv', engine= 'python')
followers = purcsv['username']
followers.head(10)

handle = purcsv['username'][0:40]
temp = ("https://twitter.com/"+handle)
temp = temp.tolist() 

for url in temp:
    page = requests.get(url)

bs = BeautifulSoup(page.text,'lxml')

follow_box = bs.find('li',{'class':'ProfileNav-item ProfileNav-item--followers'})
followers = follow_box.find('a').find('span',{'class':'ProfileNav-value'})
print("Number of followers: {} ".format(followers.get('data-count')))

Answer 1

这是因为您首先遍历了url，并在同一变量page中获取了每个URL的内容：

for url in temp:
    page = requests.get(url)

因此页面将始终包含最后访问的URL页面，要解决此问题，您需要在提取后处理页面

followers_list = []
for url in temp:
    page = requests.get(url)

    bs = BeautifulSoup(page.text, "html.parser")

    follow_box = bs.find('li',{'class':'ProfileNav-item ProfileNav-item--followers'})
    followers = follow_box.find('a').find('span',{'class':'ProfileNav-value'})
    print("Number of followers: {} ".format(followers.get('data-count')))
    followers_list.append(followers.get('data-count'))
print(followers_list)

这里有一个完整的示例可供验证

from bs4 import BeautifulSoup
import requests
import pandas as pd
purcsv = pd.read_csv('pureeng.csv')

followers = purcsv['username']

handles = purcsv['username'][0:40].tolist()

followers_list = []
for handle in handles:
    url = "https://twitter.com/" + handle
    try:
        page = requests.get(url)
    except Exception as e:
        print(f"Failed to fetch page for url {url} due to: {e}")
        continue

    bs = BeautifulSoup(page.text, "html.parser")

    follow_box = bs.find('li',{'class':'ProfileNav-item ProfileNav-item--followers'})
    followers = follow_box.find('a').find('span',{'class':'ProfileNav-value'})
    print("Number of followers: {} ".format(followers.get('data-count')))
    followers_list.append(followers.get('data-count'))
print(followers_list)

输出：

Number of followers: 13714085 
Number of followers: 4706511 
['13714085', '4706511']

如果您有两个URL，可以考虑使用async函数来获取和处理这些URL。