从非结构化文本创建pandas DataFrame

问题描述 投票:0回答:2

很抱歉,这个问题已经解决了。我正在尝试分析一些Facebook消息。到目前为止,我下载了一个html文件,并使用BeautifulSoup将其转换为一个整洁的列表,现在我正尝试从中创建一个数据框。

我正在查看此资源:https://datatofish.com/list-to-dataframe/,但是没有用。

这里是我现在的格式:

list = ['2019-01-07 12:51 PM', 'name1', 'hi how are you', 'im at home', 'wanna come over?', '2019-01-07 01:02 PM', 'name2', 'hell yeah', '🐟', 'ill bring beer', '2019-01-07 01:06 PM', 'name1', 'awesome', 'and so on']

我尝试了几种不同的方法,但我开始认为自己的能力超出了我的咀嚼能力。我正在学习。

这是我希望得到的输出:

index date          time       name            message
0      2019-01-07   12:51 PM   name1           hi how are you
1      2019-01-07   12:51 PM   name1           im at home
2      2019-01-07   12:51 PM   name1           wanna come over?
3      2019-01-07   12:56 PM   name2           hell yeah

我尝试遍历列表并填充列,并点击了日期,名称或消息。

正如我说的,我正在学习,因此,如果不向我指出正确的研究方向,而不是解决方案,那将是惊人的。我将非常感谢。谢谢!

编辑:我尝试了几个现有的消息解析器,但是由于某种原因,它们都在2018年停止了支持。它们也都给我解析错误消息。

python pandas dataframe message
2个回答
0
投票

这有点难看,但是可以。我很乐意赞成一个更精致的解决方案!

l = iter(['2019-01-07 12:51 PM', 'name1', 'hi how are you', 'im at home', 'wanna come over?', '2019-01-07 01:02 PM', 'name2', 'hell yeah', '🐟', 'ill bring beer', '2019-01-07 01:06 PM', 'name1', 'awesome', 'and so on'])
df = pd.DataFrame()

# get first element in list
x = next(l)

# if element is the last, catch the IterationError and stop
try:
    while 1:
        # try to convert element to datetime
        datetime = pd.to_datetime(x, format="%Y-%m-%d %H:%M %p")
        # if successful get next element as name
        x = next(l)
        name = x

        # get next elements as messages while they do not match datetime format
        x = next(l)
        while 1:
            try:
                # if datetime conversion is successful break while 
                pd.to_datetime(x, format="%Y-%m-%d %H:%M %p");
                break
            except ValueError:
                # else add message to dataframe
                df = df.append([{"datetime":datetime,"name":name,"msg":x}])
                x = next(l)
except StopIteration:
    pass

df["date"] = df["datetime"].dt.date
df["time"] = df["datetime"].dt.time
print(df)

             datetime               msg   name        date      time
0 2019-01-07 12:51:00    hi how are you  name1  2019-01-07  12:51:00
0 2019-01-07 12:51:00        im at home  name1  2019-01-07  12:51:00
0 2019-01-07 12:51:00  wanna come over?  name1  2019-01-07  12:51:00
0 2019-01-07 01:02:00         hell yeah  name2  2019-01-07  01:02:00
0 2019-01-07 01:02:00                 🐟  name2  2019-01-07  01:02:00
0 2019-01-07 01:02:00    ill bring beer  name2  2019-01-07  01:02:00
0 2019-01-07 01:06:00           awesome  name1  2019-01-07  01:06:00
0 2019-01-07 01:06:00         and so on  name1  2019-01-07  01:06:00

0
投票

使用正则表达式和列表推导,将列表内容提取并转换为Pandas数据框:


import pandas as pd

import re

datetime_regex = re.compile(r"\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}\sPM")
name_regex = re.compile(r"name\d+")

cols = ["date",
        "time",
        "name",
        "message"
        ]

l = ['2019-01-07 12:51 PM',
     'name1',
     'hi how are you',
     'im at home',
     'wanna come over?',
     '2019-01-07 01:02 PM',
     'name2',
     'hell yeah',
     '🐟',
     'ill bring beer',
     '2019-01-07 01:06 PM',
     'name1',
     'awesome',
     'and so on'
     ]
tmp = ''.join(l)

datetimes = re.findall(datetime_regex, tmp)
dates = [datetime[:11] for datetime in datetimes]
times = [datetime[11:] for datetime in datetimes]
names = re.findall(name_regex, tmp)

messages = [line
            for line in l
            if not line.startswith(('2019', 'name1', 'name2'))
            ]

data = [[[dates[0], times[0], names[0], msg]
        for msg in messages[:3]],
        [[dates[1], times[1], names[1], messages[3]]],
        [[dates[2], times[2], names[2], msg]
        for msg in messages[4:]]
        ]

flatten = [item for sublist in data for item in sublist]

df = pd.DataFrame(flatten, columns=cols)
print(df)

哪个返回:

          date      time   name           message
0  2019-01-07   12:51 PM  name1    hi how are you
1  2019-01-07   12:51 PM  name1        im at home
2  2019-01-07   12:51 PM  name1  wanna come over?
3  2019-01-07   01:02 PM  name2         hell yeah
4  2019-01-07   01:06 PM  name1                 🐟
5  2019-01-07   01:06 PM  name1    ill bring beer
6  2019-01-07   01:06 PM  name1           awesome
7  2019-01-07   01:06 PM  name1         and so on
© www.soinside.com 2019 - 2024. All rights reserved.