使用 Python 在 DIV 内实现景观跨度

Question

尝试使用 Python 从网站收集 Amazon Prime 可用电影的列表：

https://www.amazon.co.uk/s?i=instant-video&bbn=3010085031&rh=n%3A3010085031%2Cp_85%3A3282143031%2Cp_72%3A3289786031&dc&adult-product=0&field-genre=-family&field-review_计数=3-&字段-ways_to_watch=7448662031&p_n_entity_type=9739952031&qid=1557596014&qs-av_request_type=4&qs-is-prime-customer=0&rnid=3289782031&ref=sr_nr_p_72_4

我试图找到电影的标题，但返回 0。这一定是来自首先加载 JavaScript 的页面，我在 NETWORK 中查找过，但无法找出我要找的内容。

我尝试了以下代码：

from requests import get
url = 'https://www.amazon.co.uk/s?i=instant-video&bbn=3010085031&rh=n%3A3010085031%2Cp_85%3A3282143031%2Cp_72%3A3289786031&dc&adult-product=0&field-genre=-family&field-review_count=3-&field-ways_to_watch=7448662031&p_n_entity_type=9739952031&qid=1557596014&qs-av_request_type=4&qs-is-prime-customer=0&rnid=3289782031&ref=sr_nr_p_72_4'
response = get(url)
from bs4 import BeautifulSoup
html_soup = BeautifulSoup(response.text, 'lxml')
type(html_soup)
movie_containers = html_soup.find_all('span', class_ = 'a-size-medium     a-color-base a-text-normal')
print(type(movie_containers))
print(len(movie_containers))

然后我尝试循环：

for n in soup.find_all('span', {'class':'a-size-medium     a-color-base a-text-normal'}):
    title.append (n.text)

任何帮助都会很棒。谢谢。

结果应提取每部电影的标题和链接。

Answer 1

您必须包含获取请求的标头。

import requests
from bs4 import BeautifulSoup

header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"}
a = requests.get("https://www.amazon.co.uk/s?i=instant-video&bbn=3010085031&rh=n%3A3010085031%2Cp_85%3A3282143031%2Cp_72%3A3289786031&dc&adult-product=0&field-genre=-family&field-review_count=3-&field-ways_to_watch=7448662031&p_n_entity_type=9739952031&qid=1557596014&qs-av_request_type=4&qs-is-prime-customer=0&rnid=3289782031&ref=sr_nr_p_72_4", headers=header)
b = BeautifulSoup(a.text, "html.parser")
for c in b.find_all("span", class_="a-size-medium"):
    print(c.text)