我如何使用python或scala将非标准的csv文件读入数据帧

Question

我下面有一个要用python或scala处理的数据集采样：

FWD,13032009:09:01,10.56| FWD,13032009:10:53,11.23| FWD,13032009:15:40,23.20
SPOT,13032009:09:04,11.56| FWD,13032009:11:45,11.23| SPOT,13032009:12:30,23.20
FWD,13032009:08:01,10.56| SPOT,13032009:12:30,11.23| FWD,13032009:13:20,23.20| FWD,13032009:14:340,56.00
FWD,13032009:08:01,10.56| SPOT,13032009:12:30,11.23| FWD,13032009:13:20,23.20

每行都将被拆分为多个较小的字符串，可以进一步拆分。

我正在寻找一种有效的方法来生成具有以下内容的RDD或数据框：

FWD,13032009:09:01,10.56 
FWD,13032009:10:53,11.23
FWD,13032009:15:40,23.20
SPOT,13032009:09:04,11.56
FWD,13032009:11:45,11.23
SPOT,13032009:12:30,23.20
FWD,13032009:08:01,10.56
SPOT,13032009:12:30,11.23
FWD,13032009:13:20,23.20
FWD,13032009:14:340,56.00
FWD,13032009:08:01,10.56
SPOT,13032009:12:30,11.23
FWD,13032009:13:20,23.20

注意，效率越高，生产中的总行数可能高达一百万个就越好

非常感谢。

Answer 1

Python解决方案：如果以字符串形式获取文本，则可以用换行符（replace()）将| 序列\n，然后将其作为DataFrame读取：

import pandas as pd
from io import StringIO

data_set = """FWD,13032009:09:01,10.56| FWD,13032009:10:53,11.23| FWD,13032009:15:40,23.20
SPOT,13032009:09:04,11.56| FWD,13032009:11:45,11.23| SPOT,13032009:12:30,23.20
FWD,13032009:08:01,10.56| SPOT,13032009:12:30,11.23| FWD,13032009:13:20,23.20| FWD,13032009:14:340,56.00
FWD,13032009:08:01,10.56| SPOT,13032009:12:30,11.23| FWD,13032009:13:20,23.20
"""
data_set *= 100000  # Make it over a million elements to ensure performance is adequate
data_set = data_set.replace("| ", "\n")

data_set_stream = StringIO(data_set)  # Pandas needs to read a file-like object, so need to turn our string into a buffer
df = pd.read_csv(data_set_stream)
print(df)  # df is our desired DataFrame

Answer 2

假设您正在读取一个csv文件，则可以读取列表的每一行。展平值，然后将它们作为单独的行进行处理。

将文件读入列表-一百万行应该不会太多：

import csv
import itertools

import pandas as pd

with open('test.csv','r') as f:
    reader = csv.reader(f, delimiter = '|')
    rows = list(reader)

从单个列表中分割并拆分-Python标准库中出色的itertools库返回一个生成器，该生成器有助于内存并且高效。

flat_rows = itertools.chain.from_iterable(rows)
list_rows = [i.strip().split(',') for i in flat_rows]

嵌套列表，list_rows现在为您提供了一个干净且格式化的列表，如果您要创建pandas，则可以将其发送到dataframe。

list_rows
>>
[['FWD', '13032009:09:01', '10.56'],
 ['FWD', '13032009:10:53', '11.23'],
 ['FWD', '13032009:15:40', '23.20'],
 ['SPOT', '13032009:09:04', '11.56'],
 ['FWD', '13032009:11:45', '11.23'],
 ['SPOT', '13032009:12:30', '23.20'],
 ['FWD', '13032009:08:01', '10.56'],
 ['SPOT', '13032009:12:30', '11.23'],
 ['FWD', '13032009:13:20', '23.20'],
 ['FWD', '13032009:14:340', '56.00'],
 ['FWD', '13032009:08:01', '10.56'],
 ['SPOT', '13032009:12:30', '11.23'],
 ['FWD', '13032009:13:20', '23.20']]

df = pd.DataFrame(list_rows)