绝对的相对路径

Question

我正在尝试搜寻一个论坛，以便最终找到帖子中包含链接的帖子。 现在，我只是想抓取帖子的用户名。 但是我认为网址不是静态的存在问题。

spider.py

from scrapy.spiders import CrawlSpider
from scrapy.selector import Selector
from scrapy.item import Item, Field


class TextPostItem(Item):
    title = Field()
    url = Field()
    submitted = Field()


class RedditCrawler(CrawlSpider):
    name = 'post-spider'
    allowed_domains = ['flashback.org']
    start_urls = ['https://www.flashback.org/t2637903']


    def parse(self, response):
        s = Selector(response)
        next_link = s.xpath('//a[@class="smallfont2"]//@href').extract()[0]
        if len(next_link):
            yield self.make_requests_from_url(next_link)
        posts =   Selector(response).xpath('//div[@id="posts"]/div[@class="alignc.p4.post"]')
        for post in posts:
            i = TextPostItem()
            i['title'] = post.xpath('tbody/tr[1]/td/span/text()').extract() [0]
            #i['url'] = post.xpath('div[2]/ul/li[1]/a/@href').extract()[0]
            yield i

提供以下错误：

raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: /t2637903p2

任何想法？

Answer 1

您需要将response.url与使用urljoin()提取的相对URL“连接” urljoin() ：

from urlparse import urljoin

urljoin(response.url, next_link)

---

另请注意，无需实例化Selector对象-您可以直接使用response.xpath()快捷方式：

def parse(self, response):
    next_link = response.xpath('//a[@class="smallfont2"]//@href').extract()[0]
    # ...