如何修复可能损坏的json文件？大括号字符“ {”（Python3）

Question

这是一个很奇怪的问题，我什至不知道该怎么问，但我会尽力的。我有一些json文件，其中包含网络抓取数据，每个文件有多个条目，它们看起来像这样：

{
"doc_id": "some_number",
"url": "www.seedurl1.com",
"scrape_date": "2019-10-22 16:17:22",
"publish_date": "unknown",
"author": "unknown",
"urls_out": [
"https://www.something.com",
"https://www.sometingelse.com/smth"
],
"text": "lots of text here"
}
{
"doc_id": "some_other_number",
"url": "www.seedurl2.com/smth",
"scrape_date": "2019-10-22 17:44:40",
"publish_date": "unknown",
"author": "unknown",
"urls_out": [
"www.anotherurl.com/smth",
"http://urlx.com/smth.htm"
],
"text": "lots more text over here."
}

我正在尝试格式化它们，以便每个条目都位于它们自己的行上，如下所示：

{"doc_id": blah blah....} 
{"doc_id": blah blah blah...}

所以我这样做：

    # Read the file
    f = codecs.open(file, 'r', encoding='utf-8-sig', errors='replace')
    text = f.read()
    f.close()

    # Check if }{ was found; 
    # this prints nothing for original files but finds everything in a hand written file
    pattern = '}{'
    print('Before editing: ', (re.findall(pattern, text)))

    # Getting rid of excess newlines and whitespaces
    newtext = " ".join(text.split())

    # Check if } { was found;
    # this prints nothing for original files but finds everything in a hand written file
    pattern = '} {'
    print('After editing: ', (re.findall(pattern, newtext)))

    # Put newlines in the right places
    finaltext = re.sub('} {', '}\n{', newtext)

    # Write the new JSON
    newfile = file[:-5]+'_ED.json'
    nf = codecs.open(newfile, 'w', encoding='utf-8', errors='replace')
    nf.write(finaltext)
    nf.close()

事实是，该代码可以在具有相同结构的手写测试文件上完美运行，但不能与原始文件或从原始文件派生的较小测试文件一起使用。

我试图在文本编辑器中分别对“}”和“ {”进行简单搜索，结果很好。但是，如果我尝试搜索“} {”或“} {”，则找不到任何内容。虽然我可以看到他们显然在那里。

[最后发现：我试图在Linux的Nano中打开我的小型测试文件的编辑版本，然后移至问题区域。由于某种原因，需要两次按右箭头键才能在“ {”大括号上移动。所以显然那里有些奇怪。我如何找出什么？或其他可能有帮助的建议？

Answer 1

最简单的解决方案就是以JSON数组开头...

否则，我建议您不要更换任何东西，而只需计算匹配的括号即可。

count = 0
objects = 0
with open('file.txt') as f:
    for i, c in enumerate(f.read()):
      if c == '\n':
        continue
      elif c == '{':
        if i > 0 and count == 0:
          print()  # start new line before printing bracket
        count += 1
      elif c == '}':
        count -= 1
        if count == 0:  # found a complete JSON object
          objects += 1

      print(c, end='')
    print(f'\n\nfound {objects} objects')  # for debugging 

对于给定的文本，我最终得到了这个

{"doc_id": "some_number","url": "www.seedurl1.com","scrape_date": "2019-10-22 16:17:22","publish_date": "unknown","author": "unknown","urls_out": ["https://www.something.com","https://www.sometingelse.com/smth"],"text": "lots of text here"}
{"doc_id": "some_other_number","url": "www.seedurl2.com/smth","scrape_date": "2019-10-22 17:44:40","publish_date": "unknown","author": "unknown","urls_out": ["www.anotherurl.com/smth","http://urlx.com/smth.htm"],"text": "lots more text over here."}

found 2 objects

Answer 2

这是一种方法。

Ex：

import json

with open(filename) as infile:
    data = json.loads("[" + infile.read().replace("}\n{", "},\n{") + "]")
    for i in data:
        print(i)

输出：

{'doc_id': 'some_number', 'url': 'www.seedurl1.com',.....
{'doc_id': 'some_other_number', 'url': 'www.seedurl2.com/smth',.....

Answer 3

这里是另一种解决方案，与您的尝试差不多

import json

with open('test.txt') as f:
    file = f.readlines()
file = ['{'+i+'}'for i in "".join("".join(file).split("\n"))[1:-1].split("}{")]

for i in file:
    print(json.loads(i))

json在这里仅用于验证单个JSON。这给

{'doc_id': 'some_number', 'url': 'www.seedurl1.com', 'scrape_date': '2019-10-22 16:17:22', 'publish_date': 'unknown', 'author': 'unknown', 'urls_out': ['https://www.something.com', 'https://www.sometingelse.com/smth'], 'text': 'lots of text here'}
{'doc_id': 'some_number', 'url': 'www.seedurl1.com', 'scrape_date': '2019-10-22 16:17:22', 'publish_date': 'unknown', 'author': 'unknown', 'urls_out': ['https://www.something.com', 'https://www.sometingelse.com/smth'], 'text': 'lots of text here'}