我正在寻找欧洲志愿服务的公开列表：我不需要完整的地址 - 但需要名称和网站。我想到了数据... XML、CSV ... 包含这些字段：名称、国家/地区 - 以及一些其他字段，对于每个存在国家/地区来说，一条记录会很好。 顺便说一句：欧洲志愿服务是年轻人的绝佳选择
我发现了一个非常非常全面的很棒的页面； 想要从欧洲网站上托管的欧洲志愿服务收集数据：
参见：https://youth.europa.eu/go-abroad/volunteering/opportunities_en
@HedgeHog 向我展示了正确的方法以及如何找到正确的选择器 在此线程中：BeatuifulSoup 迭代超过 10k 页面并获取数据，解析：欧洲志愿服务：一个从 EU-Site 收集机会的小型爬虫
# Extracting relevant data
title = soup.h1.get_text(', ',strip=True)
location = soup.select_one('p:has(i.fa-location-arrow)').get_text(', ',strip=True)
start_date,end_date = (e.get_text(strip=True)for e in soup.select('span.extra strong')[-2:])
但是我们在那里有数百个志愿服务机会 - 这些机会存储在如下网站中：
https://youth.europa.eu/solidarity/placement/39020_en
https://youth.europa.eu/solidarity/placement/38993_en
https://youth.europa.eu/solidarity/placement/38973_en
https://youth.europa.eu/solidarity/placement/38972_en
https://youth.europa.eu/solidarity/placement/38850_en
https://youth.europa.eu/solidarity/placement/38633_en
想法：
我认为收集数据会很棒 - 即使用基于
BS4requestsdataframe嗯 - 我认为我们可以迭代所有的网址：
placement/39020_en
placement/38993_en
placement/38973_en
placement/38850_en
想法：我认为我们可以在存储中从零迭代到100 000以获取存储在展示位置中的所有结果。 但这个想法没有代码支持。换句话说 - 目前我不知道如何实现在如此大的范围内迭代的特殊想法：
目前我认为 - 这是从这里开始的基本方法：
import requests
from bs4 import BeautifulSoup
import pandas as pd
# Function to generate placement URLs based on a range of IDs
def generate_urls(start_id, end_id):
base_url = "https://youth.europa.eu/solidarity/placement/"
urls = [base_url + str(id) + "_en" for id in range(start_id, end_id+1)]
return urls
# Function to scrape data from a single URL
def scrape_data(url):
response = requests.get(url)
if response.status_code == 200:
soup = BeautifulSoup(response.content, 'html.parser')
title = soup.h1.get_text(', ', strip=True)
location = soup.select_one('p:has(i.fa-location-arrow)').get_text(', ', strip=True)
start_date, end_date = (e.get_text(strip=True) for e in soup.select('span.extra strong')[-2:])
website_tag = soup.find("a", class_="btn__link--website")
website = website_tag.get("href") if website_tag else None
return {
"Title": title,
"Location": location,
"Start Date": start_date,
"End Date": end_date,
"Website": website,
"URL": url
}
else:
print(f"Failed to fetch data from {url}. Status code: {response.status_code}")
return None
# Set the range of placement IDs we want to scrape
start_id = 1
end_id = 100000
# Generate placement URLs
urls = generate_urls(start_id, end_id)
# Scrape data from all URLs
data = []
for url in urls:
placement_data = scrape_data(url)
if placement_data:
data.append(placement_data)
# Convert data to DataFrame
df = pd.DataFrame(data)
# Print DataFrame
print(df)
这给了我以下信息
Failed to fetch data from https://youth.europa.eu/solidarity/placement/154_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/156_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/157_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/159_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/161_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/162_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/163_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/165_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/166_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/169_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/170_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/171_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/173_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/174_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/176_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/177_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/178_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/179_en. Status code: 404
Failed to fetch data from https://youth.europa.eu/solidarity/placement/180_en. Status code: 404
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-5-d6272ee535ef> in <cell line: 42>()
41 data = []
42 for url in urls:
---> 43 placement_data = scrape_data(url)
44 if placement_data:
45 data.append(placement_data)
<ipython-input-5-d6272ee535ef> in scrape_data(url)
16 title = soup.h1.get_text(', ', strip=True)
17 location = soup.select_one('p:has(i.fa-location-arrow)').get_text(', ', strip=True)
---> 18 start_date, end_date = (e.get_text(strip=True) for e in soup.select('span.extra strong')[-2:])
19 website_tag = soup.find("a", class_="btn__link--website")
20 website = website_tag.get("href") if website_tag else None
ValueError: not enough values to unpack (expected 2, got 0)
有什么想法吗？
参见基本网址：https://youth.europa.eu/go-abroad/volunteering/opportunities_en
与其自己创建 ids，我宁愿选择 API 方法并检索已经结构化为 JSON 的信息。这可以通过
dataframe 转换为 pandas.json_normalize()
import requests
import pandas as pd
data = requests.get('https://youth.europa.eu/d8/api/rest/eyp/v1/search_en?type=Opportunity&size=100&from=0&filters%5Bstatus%5D=open&filters%5Bdate_end%5D%5Boperator%5D=%3E%3D&filters%5Bdate_end%5D%5Bvalue%5D=2024-03-14&filters%5Bdate_end%5D%5Btype%5D=must').json().get('hits').get('hits')
pd.json_normalize(data)