在EC2上使用请求和BeautifulSoup时发生内存错误

Question

我正在使用请求和BeautifulSoup解析Wikidata来构建Person对象。我能够成功完成此操作，但是在进行迭代时，在创建了约3,000个Person对象之后，我遇到了下面的MemoryError。

MemoryError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "TreeBuilder.py", line 11, in <module>
    ancestor = Person(next['id'])
  File "/home/ec2-user/Person.py", line 14, in __init__
    html = soup (data , 'lxml')
  File "/usr/local/lib/python3.7/site-packages/bs4/__init__.py", line 325, in __init__
    self._feed()
  File "/usr/local/lib/python3.7/site-packages/bs4/__init__.py", line 399, in _feed
    self.builder.feed(self.markup)
  File "/usr/local/lib/python3.7/site-packages/bs4/builder/_lxml.py", line 324, in feed
    self.parser.feed(markup)
  File "src/lxml/parser.pxi", line 1242, in lxml.etree._FeedParser.feed
  File "src/lxml/parser.pxi", line 1285, in lxml.etree._FeedParser.feed
  File "src/lxml/parser.pxi", line 855, in lxml.etree._BaseParser._getPushParserContext
  File "src/lxml/parser.pxi", line 871, in lxml.etree._BaseParser._createContext
  File "src/lxml/parser.pxi", line 528, in lxml.etree._ParserContext.__cinit__
SystemError: <class 'lxml.etree._ErrorLog'> returned a result with an error set

我尝试使用以下命令捕获无法正常工作的异常；

try:
  data = requests.get (url).text
  html = soup(data, 'lxml')
except MemoryError:
  return None

仅在我的AWS EC2服务器上在Pycharm中运行程序时，在本地计算机上不会发生此错误。

UPDATE

请参见下面的代码。我每隔100次迭代就添加gc.collect()，似乎没有帮助。

Person.py

import requests
from bs4 import BeautifulSoup as soup

class Person:

    def __init__(self, id):
        url = 'https://www.wikidata.org/wiki/' + id
        data = requests.get (url).text
        html = soup (data , 'lxml')

        ### id ###
        self.id = id

        ### Name ###
        if html.find ("span" , {"class": "wikibase-title-label"}) != None:
            self.name = html.find ("span" , {"class": "wikibase-title-label"}).string
        else:
            self.name = ""

        ### Born ###
        self.birth = ""

        birth = html.find ("div" , {"id": "P569"})
        if birth != None:
            birth = birth.findAll ("div" , {"class": "wikibase-snakview-variation-valuesnak"})
            if len(birth) > 0:
                self.birth = birth[0].string

        ### Death ###
        self.death = ""

        death = html.find ("div" , {"id": "P570"})
        if death != None:
            death = death.findAll ("div" , {"class": "wikibase-snakview-variation-valuesnak"})
            if len(death) > 0:
                self.death = death[0].string

        #### Sex ####
        sex = html.find ("div" , {"id": "P21"})

        if sex != None:
            for item in sex.strings:
                if item == 'male' or item == 'female':
                    self.sex = item

        ### Mother ###
        self.mother = ""

        mother = html.find ("div" , {"id": "P25"})
        if mother != None:
            mother = mother.findAll ("div" , {"class": "wikibase-snakview-variation-valuesnak"})
            if len(mother) > 0:
                self.mother = {"name": mother[0].string , "id": mother[0].find ('a')['title']}

        ### Father ###
        self.father = ""

        father = html.find ("div" , {"id": "P22"})
        if father != None:
            father = father.findAll ("div" , {"class": "wikibase-snakview-variation-valuesnak"})
            if len(father) > 0:
                self.father = {"name": father[0].string , "id": father[0].find ('a')['title']}

        ### Children ###
        self.children = []
        x = html.find("div" , {"id": "P40"})
        if x != None:
            x = x.findAll("div" , {"class": "wikibase-statementview"})

            for i in x:
                a = i.find ('a')
                if a != None and a['title'][0] == 'Q':
                    self.children.append ({'name': a.string , 'id': a['title']})

    def __str__(self):
        return self.name + "\n\tBirth: " + self.birth + "\n\tDeath: " + self.death + "\n\n\tMother: " + \
               self.mother['name'] + "\n\tFather: " + self.father['name'] + "\n\n\tNumber of Children: " + \
               str(len(self.children))

TreeBuilder.py] >>

from Person import Person
import gc, sys

file = open('ancestors.txt', 'w+')

ancestors = [{'name':'Charlemange', 'id':'Q3044'}]
all = [ancestors[0]['id']]
i = 1

while ancestors != []:
    next = ancestors.pop(0)
    ancestor = Person(next['id'])

    for child in ancestor.children:
        if child['id'] not in all:
            all.append(child['id'])
            ancestors.append(child)

    if ancestor.mother != "" and ancestor.mother['id'] not in all:
        all.append(ancestor.mother['id'])
        ancestors.append(ancestor.mother)

    if ancestor.father != "" and ancestor.father['id'] not in all:
        all.append(ancestor.father['id'])
        ancestors.append(ancestor.father)


    file.write(ancestor.id + "*" + ancestor.name + "*" + "https://www.wikidata.org/wiki/" + ancestor.id + "*" + str(ancestor.birth) + "*" + str(ancestor.death) + "\n")

    if i % 100 == 0:
        print (ancestor.name + " (" + ancestor.id + ")" + " - " + str(len(all)) + " - " + str (len(ancestors)) + " - " + str (sys.getsizeof(all)))
        gc.collect()

    i += 1

file.close()
print("\nDone!")

我正在使用请求和BeautifulSoup解析Wikidata来构建Person对象。我能够成功完成此操作，但是在进行迭代操作时，在创建〜...

Answer 1

解决方案，但是很烦人，是在str()调用中包装对BS对象的所有引用。似乎在执行birth[0].string时仅存储对裸字符串的引用，但事实证明，对BS对象的引用只是在循环执行时才继续在内存中构建。试一试，我刚刚在您的代码中尝试过，内存占用仍然很低。