如何以编程方式获取 csv 文件中的字段以相互对齐

问题描述 投票：0回答：2

我有一组非常旧的数据，正在清理。我已将其放入 csv 文件中，但数据逐行不一致。我想让每行中的每个字段与其之前或之后的行中的其他字段相匹配。

data = """
30,1204,PO,71100,147130,I09,B10,OC,350,20105402
31,1221,PO,70400,147170,I09,B10,OC,500,20105402
32,1223,SI,70384,147122,I09,B10,OC,500,PN,3,BO,OI,20105402
33,1224,SI,70392,147032,I09,B10,OC,500,PN,1,BO,OI,20105402
34,1227,PO,70400,146430,I09,B10,PF,500,20105402
35,1241,PO,71100,146420,I09,B10,PF,500,20105402
36,1249,PO,71100,146000,I09,B10,SN,500,20105402
37,1305,PO,70400,146000,I09,B10,OC,500,20105402
38,1307,SI,70379,146041,I09,B10,OC,500,21,BH,1,BO,195,40,SW,20105402
39,1312,SD,70372,146062,I09,B10,OC,500,20105402
40,1332,SI,70334,146309,I09,B10,OC,500,PN,4,BO,OI,20105402
41,1332,SI,70334,146309,I09,B10,OC,500,PN,5,BO,OI,20105403
42,1333,SI,70333,146324,I09,B10,OC,500,PN,2,BO,OI,20105403
43,1334,SI,70328,146348,I09,B10,OC,500,PN,1,BO,OI,20105403
44,1335,SI,70326,146356,I09,B10,OC,500,PN,1,BO,OI,20105403
45,1336,SI,70310,146424,I09,B10,OC,500,PN,1,BO,OI,20105403
46,1338,SI,70302,146457,I10,B10,OC,500,PN,1,BO,OI,20105403
47,1338,SI,70301,146464,I10,B10,OC,500,PN,1,BO,OI,20105403
48,1340,SI,70295,146503,I10,B10,OC,500,PN,8,BO,OI,20105403
49,1405,LD,2,70119,148280,I10,B10,OC,0000,20105403
01,1024,LA,1R,70120,148280,B10,OC,0000,21105501
02,1039,PO,70340,149400,I10,B10,OC,500,21105501
03,1045,SI,70378,149025,I10,B07,PF,300,PN,17,BO,OI,21105501
"""

这是数据的示例。正如您所看到的，一些线的长度相同，而另一些线的长度不同。当线的长度相同时，某些字段是逐行一致的，但当线的长度不同时，字段位于不同的位置。

我真的只是在寻找一些关于如何在Python中清理这个的想法，这样我就可以得到尽可能多的字段排列并实际上能够使用这些数据。

python csv data-cleaning
2个回答
0
投票

您可能需要考虑做的事情之一是了解额外数据的用途。

您可以首先向 csv 文件添加标题。这将有助于字段的映射。如果您不需要这些数据，您可以随时在其中添加某种行，因为 

csv.reader
中的读者查看一行，看到的是额外的数据，将其删除或移动到该行的末尾，然后您也可以将该数据的标题放在行尾。

0
投票

如果没有上下文，我们就不可能做出明确的答案。开始的方法如下：根据 

,
分割每一行，并根据每行中的项目数创建数据框。

然后您可以根据您的需求进一步清理您的数据：

from io import StringIO

import pandas as pd

data = """
30,1204,PO,71100,147130,I09,B10,OC,350,20105402
31,1221,PO,70400,147170,I09,B10,OC,500,20105402
32,1223,SI,70384,147122,I09,B10,OC,500,PN,3,BO,OI,20105402
33,1224,SI,70392,147032,I09,B10,OC,500,PN,1,BO,OI,20105402
34,1227,PO,70400,146430,I09,B10,PF,500,20105402
35,1241,PO,71100,146420,I09,B10,PF,500,20105402
36,1249,PO,71100,146000,I09,B10,SN,500,20105402
37,1305,PO,70400,146000,I09,B10,OC,500,20105402
38,1307,SI,70379,146041,I09,B10,OC,500,21,BH,1,BO,195,40,SW,20105402
39,1312,SD,70372,146062,I09,B10,OC,500,20105402
40,1332,SI,70334,146309,I09,B10,OC,500,PN,4,BO,OI,20105402
41,1332,SI,70334,146309,I09,B10,OC,500,PN,5,BO,OI,20105403
42,1333,SI,70333,146324,I09,B10,OC,500,PN,2,BO,OI,20105403
43,1334,SI,70328,146348,I09,B10,OC,500,PN,1,BO,OI,20105403
44,1335,SI,70326,146356,I09,B10,OC,500,PN,1,BO,OI,20105403
45,1336,SI,70310,146424,I09,B10,OC,500,PN,1,BO,OI,20105403
46,1338,SI,70302,146457,I10,B10,OC,500,PN,1,BO,OI,20105403
47,1338,SI,70301,146464,I10,B10,OC,500,PN,1,BO,OI,20105403
48,1340,SI,70295,146503,I10,B10,OC,500,PN,8,BO,OI,20105403
49,1405,LD,2,70119,148280,I10,B10,OC,0000,20105403
01,1024,LA,1R,70120,148280,B10,OC,0000,21105501
02,1039,PO,70340,149400,I10,B10,OC,500,21105501
03,1045,SI,70378,149025,I10,B07,PF,300,PN,17,BO,OI,21105501
"""

all_data = {}
for line in map(str.strip, data.splitlines()):
    if line == "":
        continue
    line = line.split(",")
    all_data.setdefault(len(line), []).append(line)


for v in all_data.values():
    df = pd.DataFrame(v)
    print(df)
    print("-" * 80)

打印：

    0     1   2      3       4       5    6   7     8         9
0  30  1204  PO  71100  147130     I09  B10  OC   350  20105402
1  31  1221  PO  70400  147170     I09  B10  OC   500  20105402
2  34  1227  PO  70400  146430     I09  B10  PF   500  20105402
3  35  1241  PO  71100  146420     I09  B10  PF   500  20105402
4  36  1249  PO  71100  146000     I09  B10  SN   500  20105402
5  37  1305  PO  70400  146000     I09  B10  OC   500  20105402
6  39  1312  SD  70372  146062     I09  B10  OC   500  20105402
7  01  1024  LA     1R   70120  148280  B10  OC  0000  21105501
8  02  1039  PO  70340  149400     I10  B10  OC   500  21105501
--------------------------------------------------------------------------------
    0     1   2      3       4    5    6   7    8   9   10  11  12        13
0   32  1223  SI  70384  147122  I09  B10  OC  500  PN   3  BO  OI  20105402
1   33  1224  SI  70392  147032  I09  B10  OC  500  PN   1  BO  OI  20105402
2   40  1332  SI  70334  146309  I09  B10  OC  500  PN   4  BO  OI  20105402
3   41  1332  SI  70334  146309  I09  B10  OC  500  PN   5  BO  OI  20105403
4   42  1333  SI  70333  146324  I09  B10  OC  500  PN   2  BO  OI  20105403
5   43  1334  SI  70328  146348  I09  B10  OC  500  PN   1  BO  OI  20105403
6   44  1335  SI  70326  146356  I09  B10  OC  500  PN   1  BO  OI  20105403
7   45  1336  SI  70310  146424  I09  B10  OC  500  PN   1  BO  OI  20105403
8   46  1338  SI  70302  146457  I10  B10  OC  500  PN   1  BO  OI  20105403
9   47  1338  SI  70301  146464  I10  B10  OC  500  PN   1  BO  OI  20105403
10  48  1340  SI  70295  146503  I10  B10  OC  500  PN   8  BO  OI  20105403
11  03  1045  SI  70378  149025  I10  B07  PF  300  PN  17  BO  OI  21105501
--------------------------------------------------------------------------------
   0     1   2      3       4    5    6   7    8   9   10 11  12   13  14  15        16
0  38  1307  SI  70379  146041  I09  B10  OC  500  21  BH  1  BO  195  40  SW  20105402
--------------------------------------------------------------------------------
   0     1   2  3      4       5    6    7   8     9         10
0  49  1405  LD  2  70119  148280  I10  B10  OC  0000  20105403
--------------------------------------------------------------------------------
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.