我如何使用BeautifulSoup从网页上抓取结构化表？

Question

我有以下代码可抓取网站（https://www.vesselfinder.com/vessels/STENAWECO-ENERGY-IMO-9683984-MMSI-538005270）。由于类名相似，因此很难查明表类名以将数据刮到CSV文件中。如何确保我抓取了正确的信息？

我的代码是

agent = {"User-Agent":'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36'} 
urlFile = requests.get('https://www.vesselfinder.com/vessels/STENAWECO-ENERGY-IMO-9683984-MMSI-538005270', headers = agent)

soupHtml = BeautifulSoup(urlFile.content, 'lxml')

rowsFind = soupHtml.find_all("table",{"class": "tparams"})
print(rowsFind)
for i in rowsFind:
    z = i.find_all("tr")
for r in z:
    cols = r.find_all('td' , 'v3')
    cols = [x.text.strip() for x in cols]
    print(cols)
    AISVessel.append(cols[0])

AIStable.append(AISVessel)    

现在我有这个错误：

IndexError: list index out of range

所需的输出将是：

[['Tanker' , 'Marshall Islands' , 'USHOU > DOSPM' , 'Jan 3, 19:00' , '9683984 / 538005270' , '  V7CJ5', '183 / 32 m' ,  '11.4 m' ,' 115.4° / 13.5 kn ' , '19.60436 N/80.84751 W' , 'Jan 1, 2020 07:38 UTC']]

我想将上述相关数据附加到嵌套列表中，以支持将其写入CSV文件。

Answer 1

要找到正确的表，可以使用CSS选择器h2:contains("AIS Data") ~ table.tparams td.v3-这将获得表内所有带有标题“ AIS Data”的<td>：

import requests
from bs4 import BeautifulSoup

agent = {"User-Agent":'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36'}
urlFile = requests.get('https://www.vesselfinder.com/vessels/STENAWECO-ENERGY-IMO-9683984-MMSI-538005270', headers = agent)

soupHtml = BeautifulSoup(urlFile.content, 'lxml')

out = [td.get_text(strip=True) for td in soupHtml.select('h2:contains("AIS Data") ~ table.tparams td.v3')]

print(out)

打印：

['Tanker', 'Marshall Islands', 'USHOU > DOSPM', 'Jan 3, 19:00', '9683984 / 538005270', 'V7CJ5', '183 / 32 m', '11.4 m', '115.4° / 13.5 kn', '19.60436 N/80.84751 W', 'Jan 1, 2020 07:38 UTC']