熊猫：在事件发生时填充缺失值

问题描述 投票：0回答：2

我已经问了一个类似的问题(see here)，但不幸的是它还不够清楚，所以我决定用更好的数据集创建一个新的更好的例子和对所需输出的新解释 - 编辑本来是一个主要的更改。所以，我有以下数据集（它已按日期和播放器排序）：

d = {'player': ['1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '2', '2', '2', '2', '2', '3', '3', '3', '3', '3', '3'],
'date': ['2018-01-01 00:17:01', '2018-01-01 00:17:05','2018-01-01 00:19:05', '2018-01-01 00:21:07', '2018-01-01 00:22:09', 
         '2018-01-01 00:22:17', '2018-01-01 00:25:09', '2018-01-01 00:25:11', '2018-01-01 00:27:28', '2018-01-01 00:29:29',
          '2018-01-01 00:30:35',  '2018-02-01 00:31:16', '2018-02-01 00:35:22', '2018-02-01 00:38:16', 
         '2018-02-01 00:38:20', '2018-02-01 00:55:15', '2018-01-03 00:55:22', 
         '2018-01-03 00:58:16', '2018-01-03 00:58:21', '2018-03-01 01:00:35', '2018-03-01 01:20:16', '2018-03-01 01:31:16'], 
'id': [np.nan, np.nan, 'a', 'a', 'b', np.nan, 'b', 'c', 'c', 'c', 'c', 'd', 'd', 'e', 'e', np.nan, 'f', 'f', 
       'g', np.nan, 'f', 'g']}

#create dataframe
df = pd.DataFrame(data=d)
#change date to datetime
df['date'] =  pd.to_datetime(df['date']) 
df

   player      date         id
0   1   2018-01-01 00:17:01 NaN
1   1   2018-01-01 00:17:05 NaN
2   1   2018-01-01 00:19:05 a
3   1   2018-01-01 00:21:07 a
4   1   2018-01-01 00:22:09 b
5   1   2018-01-01 00:22:07 NaN
6   1   2018-01-01 00:25:09 b
7   1   2018-01-01 00:25:11 c
8   1   2018-01-01 00:27:28 c
9   1   2018-01-01 00:29:29 c
10  1   2018-01-01 00:30:35 c
11  2   2018-02-01 00:31:16 d
12  2   2018-02-01 00:35:22 d
13  2   2018-02-01 00:38:16 e
14  2   2018-02-01 00:38:20 e
15  2   2018-02-01 00:55:15 NaN
16  3   2018-01-03 00:55:22 f
17  3   2018-01-03 00:58:16 f
18  3   2018-01-03 00:58:21 g
19  3   2018-03-01 01:00:35 NaN
20  3   2018-03-01 01:20:16 f
21  3   2018-03-01 01:31:16 g

所以，这些是我的三个专栏：

  1. 'player' - dtype = object
  2. 'session'（对象）。每个会话id将玩家已在线实现的一组动作（即数据集中的行）组合在一起。
  3. 'date'（日期时间对象）告诉我们每个操作的实施时间。此数据集中的问题是我有每个操作的时间戳，但有些操作缺少其会话ID。我想要做的是以下内容：对于每个玩家，我想根据时间线为缺失值提供一个id标签。如果它们落在特定会话的时间范围（第一个动作 - 最后一个动作）内，则可以标记缺少其id的动作。

好的，所以这里我有我缺少的值：

df.loc[df.id.isnull(),'date']
0     2018-01-01 00:17:01
1     2018-01-01 00:17:05
5     2018-01-01 00:22:07
15    2018-02-01 00:55:15
19    2018-03-01 01:00:35

请注意，我有每个人的播放器代码：我想念的只是会话代码。所以，我想比较每个缺失值的时间戳和相应玩家的sessioncode时间戳。我正在考虑用每个玩家的每个会话的第一个和最后一个动作进行计算（但我不知道这是否是最好的方法）。

my_agg = df.groupby(['player', 'id']).date.agg([min, max])
my_agg
                  min                      max
player  id      
1       a   2018-01-01 00:19:05   2018-01-01 00:21:07
        b   2018-01-01 00:22:09   2018-01-01 00:25:09
        c   2018-01-01 00:25:11   2018-01-01 00:30:35
2       d   2018-02-01 00:31:16   2018-02-01 00:35:22
        e   2018-02-01 00:38:16   2018-02-01 00:38:20
3       f   2018-01-03 00:55:22   2018-03-01 01:20:16
        g   2018-01-03 00:58:21   2018-03-01 01:31:16

然后我想匹配玩家ID的Nan，并将每个缺失值的时间戳与该玩家的每个会话的范围进行比较。

在数据集中，我试图说明我感兴趣的三种可能的场景：

  1. 该操作发生在某个会话的第一个和最后一个日期之间。在这种情况下，我想用该会话的id填充缺失值，因为它显然属于该会话。因此，数据集的第5行应标记为“b”，因为它出现在b的范围内。
  2. 我会将行动发生在任何会话范围之外的会话标记为“0” - 例如前两个Nans和第15行。
  3. 最后，如果无法将操作关联到单个会话，请将其标记为“-99”，因为它发生在不同会话的时间范围内。第19行是最后一个Nan的情况。

期望的输出：总结一下，结果应该看起来像这样：

  player      date         id
0   1   2018-01-01 00:17:01 0
1   1   2018-01-01 00:17:05 0
2   1   2018-01-01 00:19:05 a
3   1   2018-01-01 00:21:07 a
4   1   2018-01-01 00:22:09 b
5   1   2018-01-01 00:22:07 b
6   1   2018-01-01 00:25:09 b
7   1   2018-01-01 00:25:11 c
8   1   2018-01-01 00:27:28 c
9   1   2018-01-01 00:29:29 c
10  1   2018-01-01 00:30:35 c
11  2   2018-02-01 00:31:16 d
12  2   2018-02-01 00:35:22 d
13  2   2018-02-01 00:38:16 e
14  2   2018-02-01 00:38:20 e
15  2   2018-02-01 00:55:15 0
16  3   2018-01-03 00:55:22 f
17  3   2018-01-03 00:58:16 f
18  3   2018-01-03 00:58:21 g
19  3   2018-03-01 01:00:35 -99
20  3   2018-03-01 01:20:16 f
21  3   2018-03-01 01:31:16 g
pandas function pandas-groupby missing-data
2个回答
0
投票

可能不是最好的方法，但确实有效。基本上我使用shift创建一些新列，然后使用你在np.select中提到的条件：

 df['shift'] = df['id'].shift(1)
df['shift-1'] = df['id'].shift(-1)
df['merge'] = df[['shift','shift-1']].values.tolist()
df.drop(columns=['shift','shift-1'], inplace=True)

alpha = {np.nan:0,'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7,'h':8}
diff = []
for i in range(len(df)):
    diff.append(alpha[df['merge'][i][1]] - alpha[df['merge'][i][0]])

df['diff'] = diff

conditions = [(df['id'].shift(1).eq(df['id'].shift(-1)) & (df['id'].isna()) & (df['player'].shift(1).eq(df['player'].shift(-1)))),

              (~df['id'].shift(1).eq(df['id'].shift(-1)) & (df['id'].isna()) & (df['player'].shift(1).eq(df['player']) | 
                                                                                df['player'].shift(-1).eq(df['player'])) &
              (~df['diff'] < 0)),

              (~df['id'].shift(1).eq(df['id'].shift(-1)) & (df['id'].isna()) & (df['player'].shift(1).eq(df['player']) | 
                                                                                df['player'].shift(-1).eq(df['player'])) &
              (df['diff'] < 0)),


             ]
choices = [df['id'].ffill(),
           0,
           -99
          ]
df['id'] = np.select(conditions, choices, default = df['id'])
df.drop(columns=['merge','diff'], inplace=True)
df

出：

    player  date              id
0   1   2018-01-01 00:17:01   0
1   1   2018-01-01 00:17:05   0
2   1   2018-01-01 00:19:05   a
3   1   2018-01-01 00:21:07   a
4   1   2018-01-01 00:22:09   b
5   1   2018-01-01 00:22:07   b
6   1   2018-01-01 00:25:09   b
7   1   2018-01-01 00:25:11   c
8   1   2018-01-01 00:27:28   c
9   1   2018-01-01 00:29:29   c
10  1   2018-01-01 00:30:35   c
11  2   2018-02-01 00:31:16   d
12  2   2018-02-01 00:35:22   d
13  2   2018-02-01 00:38:16   e
14  2   2018-02-01 00:38:20   e
15  2   2018-02-01 00:55:15   0
16  3   2018-01-03 00:55:22   f
17  3   2018-01-03 00:58:16   f
18  3   2018-01-03 00:58:21   g
19  3   2018-03-01 01:00:35  -99
20  3   2018-03-01 01:20:16   f
21  3   2018-03-01 01:31:16   g
0
投票

在我的解决方案中，我只需要正确地应用@ysearka在之前的stackoverflow问题中编写的函数 - see here。基本的挑战是按玩家应用他的功能玩家。

#define a function to sort the missing values (ysearka function from stackoverflow)
def my_custom_function(time):
    #compare every date event with the range of the sessions. 
    current_sessions = my_agg.loc[(my_agg['min']<time) & (my_agg['max']>time)]
    #store length, that is the number of matches. 
    count = len(current_sessions)
    #How many matches are there for any missing id value?
    # if 0 it means that no matches are found: the event lies outside all the possible ranges
    if count == 0:
        return 0
    #if more than one, it is impossible to say to which session the event belongs
    if count > 1:
        return -99
    #equivalent to if count == 1 return: in this case the event belongs clearly to just one session
    return current_sessions.index[0][1]


#create a list storing all the player ids
plist = list(df.player.unique())

#ignore settingcopywarning: https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas
pd.options.mode.chained_assignment = None

# create an empty new dataframe, where to store the results
final = pd.DataFrame()
#with this for loop iterate over the part of the dataset corresponding to one player at a time
for i in plist:
    #slice the dataset by player
    players = df.loc[df['player'] == i]
    #for every player, take the dates where we are missing the id
    mv_per_player = players.loc[players.id.isnull(),'date']
    #for each player, groupby player id, and compute the first and last event
    my_agg = players.groupby(['player', 'id']).date.agg([min, max])
    #apply the function to each chunk of the dataset. You obtain a series, with all the imputed values for the Nan
    ema = mv_per_player.apply(my_custom_function)    
    #now we can sobstitute the missing id with the new imputed values...
    players.loc[players.id.isnull(),'id'] = ema.values    
    #append new values stored in players to the new dataframe
    final = final.append(players)

#...and check the new dataset
final

player  date    id
0   1   2018-01-01 00:17:01 0
1   1   2018-01-01 00:17:05 0
2   1   2018-01-01 00:19:05 a
3   1   2018-01-01 00:21:07 a
4   1   2018-01-01 00:22:09 b
5   1   2018-01-01 00:22:17 b
6   1   2018-01-01 00:25:09 b
7   1   2018-01-01 00:25:11 c
8   1   2018-01-01 00:27:28 c
9   1   2018-01-01 00:29:29 c
10  1   2018-01-01 00:30:35 c
11  2   2018-02-01 00:31:16 d
12  2   2018-02-01 00:35:22 d
13  2   2018-02-01 00:38:16 e
14  2   2018-02-01 00:38:20 e
15  2   2018-02-01 00:55:15 0
16  3   2018-01-03 00:55:22 f
17  3   2018-01-03 00:58:16 f
18  3   2018-01-03 00:58:21 g
19  3   2018-03-01 01:00:35 -99
20  3   2018-03-01 01:20:16 f
21  3   2018-03-01 01:31:16 g

我不认为我的解决方案是最好的，并且仍然会欣赏其他想法，特别是如果它们更容易扩展（我有一个大型数据集）。

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.