如何从维基百科抓取列表？

Question

我面临着与如何从维基百科中抓取列表并传输到数据框提出的问题类似的问题。我想从 维基百科页面 中的列表“死亡人数少于 25,000 人的现代战争”创建一个数据框，列名称为“死亡人数”、“战争”、“日期”。

另一篇文章中提出的解决方案对我不起作用，因为维基百科代码不同，我似乎找不到列表的类名。

我使用BeautifulSoup的方式如下：

url = "https://en.wikipedia.org/wiki/List_of_wars_by_death_toll"

import requests
from bs4 import BeautifulSoup
import pandas as pd

response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")

我尝试了多种方法，包括搜索

soup.find_all('lu')

或

soup.find_all("div", {"class": "mw-content-ltr"})[0].find_all("li")

还有

ul_elements = soup.find_all("ul")
for ul in ul_elements:
    # Find all <li> elements within the <ul> element
    li_elements = ul.find_all("li")
    for li in li_elements:
        # Print the text content of each <li> element
        print(li.get_text())

似乎没有什么效果。例如，最后一个选项打印的内容不仅仅是我的列表，而且我不知道如何将结果限制为我的列表。

谢谢！

Answer 1

尝试：

import pandas as pd
import requests
from bs4 import BeautifulSoup

url = "https://en.wikipedia.org/wiki/List_of_wars_by_death_toll"

soup = BeautifulSoup(requests.get(url).content, "html.parser")


data = []
for li in soup.select_one(
    'h3:-soup-contains("Modern wars with fewer than 25,000 deaths by death toll") ~ ul'
).find_all("li"):
    if " – " in li.text:
        a, b = li.text.split(" – ", maxsplit=1)
    else:
        a, b = li.text.split(" ", maxsplit=1)

    if "[" in b:
        b = b.split("[")[0]

    link = None
    if li.a:
        link = li.a["href"]

    data.append((a, b, link))

df = pd.DataFrame(data, columns=["Deathtoll", "Name", "Link"])
print(df.head(10))

打印：

  Deathtoll                                       Name                                         Link
0    22,211   Croatian War of Independence (1991–1995)           /wiki/Croatian_War_of_Independence
1   22,000+      Dominican Restoration War (1863–1865)              /wiki/Dominican_Restoration_War
2   21,000+                         Six-Day War (1967)                            /wiki/Six-Day_War
3    20,068                     Reform War (1857–1860)                             /wiki/Reform_War
4   20,000+                     Yaqui Wars (1533–1929)                             /wiki/Yaqui_Wars
5   20,000+  War of the Quadruple Alliance (1718–1720)          /wiki/War_of_the_Quadruple_Alliance
6   20,000+                 Ragamuffin War (1835–1845)                         /wiki/Ragamuffin_War
7   20,000+              Italo-Turkish War (1911–1912)                      /wiki/Italo-Turkish_War
8    20,000              Anglo-Spanish War (1727–1729)  /wiki/Anglo-Spanish_War_(1727%E2%80%931729)
9   19,619+             Rhodesian Bush War (1964–1979)                     /wiki/Rhodesian_Bush_War