嗨,我是scrapy的新手,我想从html元素中提取属性值。那么什么可能是从html中提取属性值的正确方法。我想提取“data-next-url”属性
<div class="loading_more_jobs" data-type="loading_more_jobs" style="display:none;" data-next-url="https://www.ziprecruiter.com/candidate/search?search=restaurant&page=2&location=Atlanta%2C+Georgia"></div>
我正在使用该xpath,但它无法正常工作
response.xpath('//*[@class="loading_more_jobs"]/@data-next-url').extract()
如果您查看源HTML,您会发现:
<button class="load_more_jobs" data-type="load_more_jobs" data-next-url="">Load More Job Results</button>
<div class="loading_more_jobs" data-type="loading_more_jobs" style="display:none;"></div>
但是你仍然可以获得下一页网址:
<div class="job_results" data-this-url="/candidate/search?search=restaurant&location=Atlanta%2C+Georgia" data-next-url="/candidate/search?location=Atlanta%2C+Georgia&page=2&search=restaurant" data-type="job_results">
=>
response.xpath('//div[@class="job_results"]/@data-next-url').extract_first()
要么
<link rel="next" href="https://www.ziprecruiter.com/candidate/search?location=Atlanta%2C+Georgia&page=2&search=restaurant">
=>
response.xpath('//link[@rel="next"]/@href').extract_first()