我正在尝试使用下面的代码获取https://www.salewa.com/de-de/herren的产品。问题是，当
next_page/de-de/herren?p=4p=9if len(products) > 0:import scrapy
from scrapy.selector import Selector
import re
import json
from scrapy import Spider, Request
from datetime import datetime as dt
import csv
class Salewa_Spider(Spider):
name = "salewa"
allowed_domains = ["salewa.com"]
start_urls = ["https://www.salewa.com/de-de/herren"]
def parse(self, response):
products = response.css('div.product--info')
for product in products:
yield{
'name' : product.css('h2.product--title::text').get().strip(),
'price': product.css('span.price--default::text').get().strip(),
'url' : product.css('a.product--information-box').attrib['href'],
}
if len(products) > 0:
try:
next_page = response.css('a[class^="listing-page--nav page--next"]').attrib['href']
except:
next_page = []
if next_page is not None:
next_page_url = 'https://www.salewa.com' + next_page
yield response.follow(next_page_url, callback=self.parse)
这是因为无限滚动正在从 ajax 调用获取信息到不同的 url，以填充产品信息。
可以通过浏览器开发工具的网络选项卡找到中间页面的 URL。您需要发现该 url 是什么并将其复制到 scrapy 请求中，以便从无限滚动中获取其余项目。
对于此网站，具体的 api 网址为“https://www.salewa.com/de-de/widgets/listing/listingCount/sCategory/316582?p=1&c=316582&part={页码}&o=1&n=36&loadProducts= 1" 返回一个 json 对象，该对象保存该页面的所有 html 元素。
您可以做的是为每个页面发送单独的请求，从 json 对象中提取 html，将其转换为 scrapy 选择器，然后您就可以像解析第一页一样解析信息。
例如：
from scrapy.selector import Selector
from scrapy import Spider, Request
class Salewa_Spider(Spider):
name = "salewa"
allowed_domains = ["salewa.com"]
def start_requests(self):
yield Request("https://www.salewa.com/de-de/herren") # request for the first page
for i in range(2, 10):
# request for remaining pages
url = "https://www.salewa.com/de-de/widgets/listing/listingCount/sCategory/316582?p=1&c=316582&part=" + str(i) + "&o=1&n=36&loadProducts=1"
yield Request(url)
def parse(self, response):
try:
# if parsing the first page this will fail otherwise this part is needed
html = Selector(text="<html>" + response.json()['listing'] + "</html>")
response = html
except:
pass
products = response.css('div.product--info')
for product in products:
yield{
'name' : product.css('h2.product--title::text').get().strip(),
'price': product.css('span.price--default::text').get().strip(),
'url' : product.css('a.product--information-box').attrib['href'],
}