我正在使用在VSCode编辑器中看起来像这样的jsonl文件:
first.jsonl
1.{"ConnectionTime": 730669.644775033,"objectId": "eHFvTUNqTR","CustomName": "Relay Controller","FirmwareRevision": "FW V1.96","DeviceID": "F1E4746E-DCEC-495B-AC75-1DFD66527561","PeripheralType": 9,"updatedAt": "2016-12-13T15:50:41.626Z","Model": "DF Bluno","HardwareRevision": "HW V1.7","Serial": "0123456789","createdAt": "2016-12-13T15:50:41.626Z","Manufacturer": "DFRobot"}
2.{"ConnectionTime": 702937.7616419792, "objectId": "uYuT3zgyez", "CustomName": "Relay Controller", "FirmwareRevision": "FW V1.96", "DeviceID": "F1E4746E-DCEC-495B-AC75-1DFD66527561", "PeripheralType": 9, "updatedAt": "2016-12-13T08:08:29.829Z", "Model": "DF Bluno", "HardwareRevision": "HW V1.7", "Serial": "0123456789", "createdAt": "2016-12-13T08:08:29.829Z", "Manufacturer": "DFRobot"}
3.
4.
5.
6.
second.jsonl
1.{"ConnectionTime": 730669.644775033,"objectId": "eHFvTUNqTR","CustomName": "Relay Controller","FirmwareRevision": "FW V1.96","DeviceID": "F1E4746E-DCEC-495B-AC75-1DFD66527561","PeripheralType": 9,"updatedAt": "2016-12-13T15:50:41.626Z","Model": "DF Bluno","HardwareRevision": "HW V1.7","Serial": "0123456789","createdAt": "2016-12-13T15:50:41.626Z","Manufacturer": "DFRobot"}
2.{"ConnectionTime": 702937.7616419792, "objectId": "uYuT3zgyez", "CustomName": "Relay Controller", "FirmwareRevision": "FW V1.96", "DeviceID": "F1E4746E-DCEC-495B-AC75-1DFD66527561", "PeripheralType": 9, "updatedAt": "2016-12-13T08:08:29.829Z", "Model": "DF Bluno", "HardwareRevision": "HW V1.7", "Serial": "0123456789", "createdAt": "2016-12-13T08:08:29.829Z", "Manufacturer": "DFRobot"}
3.
4.
然后更多,它们具有随机数量的末端线/ EOF标记。我想在每个文件的末尾使用单行或空行。我一直收到此错误raise JSONDecodeError("Expecting value", s, err.value) from Nonejson.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
使用此方法:
filenames = glob.glob("folder_with_all_jsonl/*.jsonl")
#read file by file, write file by file. Simple.
for f in filenames:
#path to the jsonl file/s
data_json = io.open(f, mode='r', encoding='utf-8-sig') # Opens in the JSONL file
data_python = extract_json(data_json)
#.....code omitted
for line in data_python: # it would fail here because of an empty line
print(line.get(objectId))
#and so on
我手动删除了一些额外的行,因此能够处理我的2个jsonl文件。
我看过这些SO板:1> Removing a new line feed in json file using Python.
2> Replace multiple newlines with single newlines during reading file
这是我想要重写它们的最佳尝试,但是
result = []
for f in glob.glob("folder_with_all_jsonl/*.jsonl"):
with open(f, 'r+', encoding='utf-8-sig') as infile:
for line in infile.readlines():
try:
result.append(json.loads(line)) # read each line of the file
except ValueError:
print(f)
with open(f,'w', encoding= 'utf-8-sig') as outfile:
json.dump(result, outfile, indent=None)
这也不起作用,因为它将所有行都放在[{....}]
中,当我逐行阅读它们时,这很不好,请提供提示/帮助。我将不胜感激!
我希望每个文件都采用以下格式:first.jsonl
1.{"ConnectionTime": 730669.644775033,"objectId": "eHFvTUNqTR","CustomName": "Relay Controller","FirmwareRevision": "FW V1.96","DeviceID": "F1E4746E-DCEC-495B-AC75-1DFD66527561","PeripheralType": 9,"updatedAt": "2016-12-13T15:50:41.626Z","Model": "DF Bluno","HardwareRevision": "HW V1.7","Serial": "0123456789","createdAt": "2016-12-13T15:50:41.626Z","Manufacturer": "DFRobot"}
2.{"ConnectionTime": 702937.7616419792, "objectId": "uYuT3zgyez", "CustomName": "Relay Controller", "FirmwareRevision": "FW V1.96", "DeviceID": "F1E4746E-DCEC-495B-AC75-1DFD66527561", "PeripheralType": 9, "updatedAt": "2016-12-13T08:08:29.829Z", "Model": "DF Bluno", "HardwareRevision": "HW V1.7", "Serial": "0123456789", "createdAt": "2016-12-13T08:08:29.829Z", "Manufacturer": "DFRobot"}
编辑:我用过宋正阳的答案和建议者的建议我实际上有两个4gb文件,执行此操作:
results = []
for f in glob.glob("folder_with_all_jsonl/*.jsonl"):
with open(f, 'r', encoding='utf-8-sig') as infile:
for line in infile:
try:
results.append(json.loads(line)) # read each line of the file
except ValueError:
print(f)
with open(f,'w', encoding= 'utf-8-sig') as outfile:
for result in results:
outfile.write(json.dumps(result) + "\n")
导致错误line 852, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
仅响应您的最后一个代码段。
您可以更改行
json.dump(result, outfile, indent=None)
到类似的东西:
for one_item in result:
outfile.write(json.dumps(one_item)+"\n")