使用 requests 和 BeautifulSoup 进行解析。为什么无法从链接中抓取电子邮件

Question

我的脚本没有返回我想从网站上抓取的电子邮件字段。有什么解决办法吗？？

from  bs4 import BeautifulSoup

import requests

url = 'https://www.kw.com/agent/UPA-6587385179144187908-1'

res = requests.get(url)

soup = BeautifulSoup(res.content,'html.parser')

name  = soup.find('div',class_='AgentContent__name').text.strip()

location = soup.find('div',class_='AgentContent__location').text.strip()

website = soup.find('a',class_='AgentInformation__factBody').attrs['href']

print(website)

print(name)

print(location)

这就是我得到的

/cdn-cgi/l/电子邮件保护#f18394909d94828590859482b199949895989093949d94df929e9c

海蒂·阿贝尔

加利福尼亚州坎贝尔

Answer 1

所有信息都在脚本块中，你可以得到你想要的一切。例如：

import requests
from bs4 import BeautifulSoup
import json


response = requests.get('https://www.kw.com/agent/UPA-6587385179144187908-1')
soup = BeautifulSoup(response.text, 'lxml')
json_data = json.loads(soup.find('script', {'id': '__NEXT_DATA__'}).get_text())
name = json_data['props']['pageProps']['agentData']['name']['full']
city = json_data['props']['pageProps']['agentData']['location']['city']
state = json_data['props']['pageProps']['agentData']['location']['state']
email = json_data['props']['pageProps']['agentData']['email']
website = json_data['props']['pageProps']['agentData']['website']
print(f"{name}, {city}, {state}, {email}, {website}")

输出：

Heidi Abele, Campbell, CA, [email protected], https://heidiabelerealtor.com/