编写包含从HTML表中抓取的数据的CSV文件

Question

我正在尝试下一步将Webscrape转换为CSV的步骤。

我已经尝试将每一列放入单独的列表中，但是我觉得这不是解决方案。

from bs4 import BeautifulSoup
import requests

url = 'https://www.pro-football-reference.com/years/2018/passing.htm'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')

for row in tb.find_all('tr'):
    i = row.get_text()
    print(i)

Answer 1

这应该工作

import csv      #quite crucial

final_table = []
for row in tb.findall('tr'):
    next_line = row.get_text()
    final_table.append([next_line])

with open('output.csv', 'w') as f:
    writer = csv.writer(f)
    writer.writerows(final_table)

Answer 2

使用csv模块。我们将使用soup.find("tr").find_all("th")抓取标题，然后循环遍历正文并将其写入文本文件。请注意，每30行交错的标题将被省略。

import csv
import requests
from bs4 import BeautifulSoup

url = "https://www.pro-football-reference.com/years/2018/passing.htm"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

with open("output.csv", "w") as f:
    writer = csv.writer(f)
    writer.writerow([x.get_text() for x in soup.find("tr").find_all("th")])

    for row in soup.find_all("tr"):
        data = [x.get_text() for x in row.find_all("td")]

        if data: 
            writer.writerow(data)

输出（仅前几行）：

Rk,Player,Tm,Age,Pos,G,GS,QBrec,Cmp,Att,Cmp%,Yds,TD,TD%,Int,Int%,Lng,Y/A,AY/A,Y/C,Y/G,Rate,QBR,Sk,Yds,NY/A,ANY/A,Sk%,4QC,GWD
Ben Roethlisberger,PIT,36,QB,16,16,9-6-1,452,675,67.0,5129,34,5.0,16,2.4,97,7.6,7.5,11.3,320.6,96.5,71.0,24,166,7.10,7.04,3.4,2,3
Andrew Luck*,IND,29,QB,16,16,10-6-0,430,639,67.3,4593,39,6.1,15,2.3,68,7.2,7.4,10.7,287.1,98.7,69.4,18,134,6.79,6.95,2.7,3,3
Matt Ryan,ATL,33,QB,16,16,7-9-0,422,608,69.4,4924,35,5.8,7,1.2,75,8.1,8.7,11.7,307.8,108.1,68.5,42,296,7.12,7.71,6.5,1,1
Kirk Cousins,MIN,30,QB,16,16,8-7-1,425,606,70.1,4298,30,5.0,10,1.7,75,7.1,7.3,10.1,268.6,99.7,58.2,40,262,6.25,6.48,6.2,1,0
Aaron Rodgers*,GNB,35,QB,16,16,6-9-1,372,597,62.3,4442,25,4.2,2,0.3,75,7.4,8.1,11.9,277.6,97.6,54.4,49,353,6.33,6.96,7.6,3,3
Case Keenum,DEN,30,QB,16,16,6-10-0,365,586,62.3,3890,18,3.1,15,2.6,64,6.6,6.1,10.7,243.1,81.2,45.5,34,235,5.90,5.39,5.5,3,4

[Check this thread，如果您在Windows的CSV结果中看到额外的换行符。