我已经识别出一个 pandas 命令
timeseries.loc[z, x] = y
负责迭代中花费的大部分时间。现在我正在寻找更好的方法来加速它。该循环甚至不包含 50k 个元素(生产目标是 ~250k 或更多),但已经需要 20 秒了。
这是我的代码(忽略上半部分,它只是计时助手)
def populateTimeseriesTable(df, observable, timeseries):
"""
Go through all rows of df and
put the observable into the timeseries
at correct row (symbol), column (tsMean).
"""
print "len(df.index)=", len(df.index) # show number of rows
global bf, t
bf = time.time() # set 'before' to now
t = dict([(i,0) for i in range(5)]) # fill category timing with zeros
def T(i):
"""
timing helper: Add passed time to category 'i'. Then set 'before' to now.
"""
global bf, t
t[i] = t[i] + (time.time()-bf)
bf = time.time()
for i in df.index: # this is the slow loop
bf = time.time()
sym = df["symbol"][i]
T(0)
tsMean = df["tsMean"][i]
T(1)
tsMean = tsFormatter(tsMean)
T(2)
o = df[observable][i]
T(3)
timeseries.loc[sym, tsMean] = o
T(4)
from pprint import pprint
print "times needed (total = %.1f seconds) for each command:" % sum(t.values())
pprint (t)
return timeseries
与(不重要,不慢)
def tsFormatter(ts):
"as human readable string, only up to whole seconds"
return time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime(ts))
。 .
--> 待优化的代码在for循环中。
(T 和 t 只是辅助函数和字典,用于计时。)
我已经为每一步计时了。绝大多数时间:
len(df.index)= 47160
times needed (total = 20.2 seconds) for each command:
{0: 1.102,
1: 0.741,
2: 0.243,
3: 0.792,
4: 17.371}
在最后一步花费了
timeseries.loc[sym, tsMean] = o
我已经下载并安装了 pypy - 但遗憾的是,它还不支持 pandas。
有什么想法可以加快填充二维数组吗?
谢谢!
编辑:抱歉,没有提到 -“时间序列”也是一个数据框:
timeseries = pd.DataFrame({"name": titles}, index=index)
更新:从 Pandas 0.20.1 开始.ix 索引器已弃用,转而使用更严格的 .iloc 和 .loc 索引器。
==================================================== ======================
@jezrael 提供了一个有趣的比较,我决定使用更多索引方法和 10M 行 DF 重复它(实际上在这种特殊情况下大小并不重要):
设置:
In [15]: df = pd.DataFrame(np.random.rand(10**7, 5), columns=list('abcde'))
In [16]: df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 5 columns):
a float64
b float64
c float64
d float64
e float64
dtypes: float64(5)
memory usage: 381.5 MB
In [17]: df.shape
Out[17]: (10000000, 5)
时间:
In [37]: %timeit df.loc[random.randint(0, 10**7), 'b']
1000 loops, best of 3: 502 µs per loop
In [38]: %timeit df.iloc[random.randint(0, 10**7), 1]
1000 loops, best of 3: 394 µs per loop
In [39]: %timeit df.at[random.randint(0, 10**7), 'b']
10000 loops, best of 3: 66.8 µs per loop
In [41]: %timeit df.iat[random.randint(0, 10**7), 1]
10000 loops, best of 3: 32.9 µs per loop
In [42]: %timeit df.ix[random.randint(0, 10**7), 'b']
10000 loops, best of 3: 64.8 µs per loop
In [43]: %timeit df.ix[random.randint(0, 10**7), 1]
1000 loops, best of 3: 503 µs per loop
结果为条形图:
计时数据为 DF:
In [88]: r
Out[88]:
method timing
0 loc 502.0
1 iloc 394.0
2 at 66.8
3 iat 32.9
4 ix_label 64.8
5 ix_integer 503.0
In [89]: r.to_dict()
Out[89]:
{'method': {0: 'loc',
1: 'iloc',
2: 'at',
3: 'iat',
4: 'ix_label',
5: 'ix_integer'},
'timing': {0: 502.0,
1: 394.0,
2: 66.799999999999997,
3: 32.899999999999999,
4: 64.799999999999997,
5: 503.0}}
绘图
ax = sns.barplot(data=r, x='method', y='timing')
ax.tick_params(labelsize=16)
[ax.annotate(str(round(p.get_height(),2)), (p.get_x() + 0.2, p.get_height() + 5)) for p in ax.patches]
ax.set_xlabel('indexing method', size=20)
ax.set_ylabel('timing (microseconds)', size=20)
at
是最快的,但事实并非如此。 ix
更快:
import pandas as pd
df = pd.DataFrame({'A':[1,2,3],
'B':[4,5,6],
'C':[7,8,9],
'D':[1,3,5],
'E':[5,3,6],
'F':[7,4,3]})
print (df)
A B C D E F
0 1 4 7 1 5 7
1 2 5 8 3 3 4
2 3 6 9 5 6 3
print (df.at[2, 'B'])
6
print (df.ix[2, 'B'])
6
print (df.loc[2, 'B'])
6
In [77]: %timeit df.at[2, 'B']
10000 loops, best of 3: 44.6 µs per loop
In [78]: %timeit df.ix[2, 'B']
10000 loops, best of 3: 40.7 µs per loop
In [79]: %timeit df.loc[2, 'B']
1000 loops, best of 3: 681 µs per loop
编辑:
MaxU
df
并且导致了差异random.randint
功能:
df = pd.DataFrame(np.random.rand(10**7, 5), columns=list('ABCDE'))
In [4]: %timeit (df.ix[2, 'B'])
The slowest run took 25.80 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 20.7 µs per loop
In [5]: %timeit (df.ix[random.randint(0, 10**7), 'B'])
The slowest run took 9.42 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 28 µs per loop
如果要在循环内添加行,请考虑这些性能问题;对于大约前 1000 到 2000 条记录,“my_df.loc”性能较好,但随着循环中记录数量的增加,性能逐渐变慢。
如果你打算在一个大循环中进行精简(比如 10Mu200c 记录左右),你最好混合使用“iloc”和“append”;用 iloc 填充临时数据帧,直到大小达到 1000 左右,然后将其附加到原始数据帧,并清空临时数据帧。这将使您的表现提高约 10 倍
python 3.12.1 上的 pandas 2.2.2
In [1]: import random; import numpy as np; import pandas as pd
In [2]: df = pd.DataFrame(np.random.rand(10**7, 5), columns=list("abcde"))
In [3]: df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 5 columns):
# Column Dtype
--- ------ -----
0 a float64
1 b float64
2 c float64
3 d float64
4 e float64
dtypes: float64(5)
memory usage: 381.5 MB
In [4]: %timeit df.loc[random.randint(0, 10**7 - 1), "b"]
8.17 µs ± 130 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
In [5]: %timeit df.iloc[random.randint(0, 10**7 - 1), 1]
18.7 µs ± 285 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
In [6]: %timeit df.at[random.randint(0, 10**7 - 1), "b"]
3.91 µs ± 251 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
In [7]: %timeit df.iat[random.randint(0, 10**7 - 1), 1]
14.9 µs ± 371 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
In [8]: %timeit df.values[random.randint(0, 10**7 - 1), 1]
1.7 µs ± 29.1 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
所以,直接访问
.values
numpy 数组仍然是最快的两倍以上,而且令人惊讶的是,如果不使用这种直接访问,通过标签访问比通过索引访问要快。