在Scrapy中，如何使用JSON加载的项目填充新字段？

Question

我正在尝试从工作清单网站的HTML中找到的LD + JSON元数据生成数据列。我使用了草率的Item Loader来清理HTML字符串并将此元数据转换为JSON对象。然后，我想使用JSON中包含的信息来填充搜寻器中的其他字段。

到目前为止，这是蜘蛛，它爬行了最近的100个作业列表：

import scrapy, json
from ..items import EthjobsScrapyItem, EthJobsLoader

class EthioJobsSpider(scrapy.Spider):
    name = "EthioJobs"
    allowed_domains = ["ethiojobs.net"]
    start_urls = ["http://www.ethiojobs.net/latest-jobs-in-ethiopia/?searchId=1573067457.6526&action=search&page=1&listings_per_page=100&view=list"]

    def parse(self, response):
        for listing_url in response.xpath('/html/body/div[4]/section/div/div/div/div[4]/div/div[1]/div[4]/div/div/div/table/tbody//@href').getall():
            yield response.follow(listing_url, callback=self.parse_listing)

    def parse_listing(self, response):
        loader = EthJobsLoader(item = EthjobsScrapyItem(), response=response)
        loader.add_xpath('JSON_LD', '//script[@type="application/ld+json"]/text()')

        yield loader.load_item() 

items.py在哪里：

import scrapy, re, json
from scrapy.loader import ItemLoader

class EthjobsScrapyItem(scrapy.Item):
    JSON_LD     = scrapy.Field()
    datePosted  = scrapy.Field() # an example of a field that would populate data from the JSON data


def cleanJsonVar(self, jsonvar): # Clean HTML markup
    for TEXT in jsonvar:
        if jsonvar:
            try:
                jsonvar = re.sub(r"\r+|\n+|\t+|  |&nbsp;|amp;|</?.{,6}>", " ", TEXT).strip()
                jsonvar = re.sub(r"Job\sDescription", "", jsonvar)
                jsonvar = re.sub(r"\A\s+", "", jsonvar) 
                jsonvar = re.sub(r"( ){2,}", r" ", jsonvar)
                jsonvar = re.sub(r"\u2019", r" '", jsonvar)
                jsonvar = re.sub(r"\u25cf", r" -", jsonvar)
                jsonvar = re.sub(r"\\",r"/", jsonvar)

            except Exception as e:
                jsonvar = None
                print("ERROR: ", str(e))
        else:
            pass
        return jsonvar

def intoJsonVar(self, jsonvar): # Convert from string to JSON
    for TEXT in jsonvar: 
        return json.loads(TEXT)


class EthJobsLoader(ItemLoader):
    JSON_LD_in  =  cleanJsonVar
    JSON_LD_out =  intoJsonVar

[JSON_LD像这样从爬虫输出：

{'JSON_LD': ["{
    '@context': 'http://schema.org/',
    '@type': 'JobPosting',
    'title': 'Terms of Reference',
    'description': ' Terms of Reference for developing General Management Plan...,'
    'identifier': {
        '@type': 'PropertyValue',
        'name': 'Population Health and Environment – Ethiopia Consortium (PHE EC)',
        'value': '65264'
    },
    'datePosted': '2019-12-10 04:13:31',
    'validThrough': '2019-12-20 23:59:59',
    'employmentType': 'Full Time',
    'hiringOrganization': {
        '@type': 'Organization',
        'name': 'Population Health and Envir...'
    },
    'jobLocation': {
        '@type': 'Place',
        'address': {
            '@type': 'PostalAddress',
            'addressLocality': 'ETH Region',
            'addressRegion': ' Addis Ababa ',
            'addressCountry': 'ETH'
        }
    }
}"]
}  

我的问题是：我如何从上述JSON中获取信息，并使用它来填充搜寻器中的新字段？

任何人和所有意见/批评都将受到欢迎！

Answer 1

首先，您可能希望将[1]的json ld文件展平，因为scrapy.Item应该是平坦的，或者至少与其他scrapy.Items嵌套在一起。您还应该摆脱私有变量（与@在一起的变量），因为这对于python中的变量名是无效字符：

{
    '@context': 'http://schema.org/',
    '@type': 'JobPosting',
    'title': 'Terms of Reference',
    'description': ' Terms of Reference for developing General Management Plan...,'
    'identifier': {
        '@type': 'PropertyValue',
        'name': 'Population Health and Environment – Ethiopia Consortium (PHE EC)',
        'value': '65264'
    }

至：

{
    'title': 'Terms of Reference',
    'description': ' Terms of Reference for developing General Management Plan...,'
    'identifier_name': 'Population Health and Environment – Ethiopia Consortium (PHE EC)',
    'identifier_value': '65264'
    },

您有物品：

class MyItem(Item):
    title = Field()
    description = Field()
    identifier_name = Field()
    identifier_value = Field()

最后，您可以通过简单地重新创建项目对象来合并多个项目：

first = MyItem()
first['title'] = 'foo'
json_ld = {
    'description': 'bar'
}
yield MyItem({**first, **json_ld})
# {'title': 'foo', 'description': 'bar'}

[1]在stackoverflow上有许多字典拼合功能和说明，例如：Flatten nested dictionaries, compressing keys