我正在尝试修改 DataFrame
df
以仅包含列 closing_price
中的值在 99 到 101 之间的行,并尝试使用下面的代码来执行此操作。
但是,我收到错误
ValueError:系列的真值不明确。使用 a.empty、a.bool()、a.item()、a.any() 或 a.all()
我想知道是否有一种方法可以在不使用循环的情况下做到这一点。
df = df[99 <= df['closing_price'] <= 101]
df = df[df['closing_price'].between(99, 101)]
您应该使用
()
对布尔向量进行分组以消除歧义。
df = df[(df['closing_price'] >= 99) & (df['closing_price'] <= 101)]
有一个更好的选择 - 使用 query() 方法:
In [58]: df = pd.DataFrame({'closing_price': np.random.randint(95, 105, 10)})
In [59]: df
Out[59]:
closing_price
0 104
1 99
2 98
3 95
4 103
5 101
6 101
7 99
8 95
9 96
In [60]: df.query('99 <= closing_price <= 101')
Out[60]:
closing_price
1 99
5 101
6 101
7 99
更新:回答评论(已编辑以修复小错误):
我喜欢这里的语法,但在尝试与 表达:
df.query('(mean - 2*sd) <= closing_price <= (mean + 2*sd)')
我的数据均在平均值的 2 个标准差之内,因此我将使用 1 来演示:
In [161]: qry = ("(closing_price.mean() - closing_price.std())" +
...: " <= closing_price <= " +
...: "(closing_price.mean() + closing_price.std())")
...:
In [162]: df.query(qry)
Out[162]:
closing_price
1 99
2 98
5 101
6 101
7 99
9 96
或
In [163]: mean = df['closing_price'].mean()
...: sd = df['closing_price'].std()
...: df.query('(@mean - @sd) <= closing_price <= (@mean + @sd)')
...:
Out [163]:
closing_price
1 99
2 98
5 101
6 101
7 99
9 96
newdf = df.query('closing_price.mean() <= closing_price <= closing_price.std()')
或
mean = closing_price.mean()
std = closing_price.std()
newdf = df.query('@mean <= closing_price <= @std')
pd.Series.between(l,r)
(针对不同的边界 l
和 r
),则会不必要地重复很多工作。在这种情况下,对框架/系列进行一次排序然后使用 pd.Series.searchsorted()
是有益的。我测得加速高达 25 倍,见下文。
def between_indices(x, lower, upper, inclusive=True):
"""
Returns smallest and largest index i for which holds
lower <= x[i] <= upper, under the assumption that x is sorted.
"""
i = x.searchsorted(lower, side="left" if inclusive else "right")
j = x.searchsorted(upper, side="right" if inclusive else "left")
return i, j
# Sort x once before repeated calls of between()
x = x.sort_values().reset_index(drop=True)
# x = x.sort_values(ignore_index=True) # for pandas>=1.0
ret1 = between_indices(x, lower=0.1, upper=0.9)
ret2 = between_indices(x, lower=0.2, upper=0.8)
ret3 = ...
基准
测量
n_reps=100
的重复评估(pd.Series.between()
)以及基于pd.Series.searchsorted()
的方法,针对不同的参数lower
和upper
。在我的配备 Python v3.8.0 和 Pandas v1.0.3 的 MacBook Pro 2015 上,以下代码会产生以下输出
# pd.Series.searchsorted()
# 5.87 ms ± 321 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# pd.Series.between(lower, upper)
# 155 ms ± 6.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
# Logical expressions: (x>=lower) & (x<=upper)
# 153 ms ± 3.52 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
import numpy as np
import pandas as pd
def between_indices(x, lower, upper, inclusive=True):
# Assumption: x is sorted.
i = x.searchsorted(lower, side="left" if inclusive else "right")
j = x.searchsorted(upper, side="right" if inclusive else "left")
return i, j
def between_fast(x, lower, upper, inclusive=True):
"""
Equivalent to pd.Series.between() under the assumption that x is sorted.
"""
i, j = between_indices(x, lower, upper, inclusive)
if True:
return x.iloc[i:j]
else:
# Mask creation is slow.
mask = np.zeros_like(x, dtype=bool)
mask[i:j] = True
mask = pd.Series(mask, index=x.index)
return x[mask]
def between(x, lower, upper, inclusive=True):
mask = x.between(lower, upper, inclusive=inclusive)
return x[mask]
def between_expr(x, lower, upper, inclusive=True):
if inclusive:
mask = (x>=lower) & (x<=upper)
else:
mask = (x>lower) & (x<upper)
return x[mask]
def benchmark(func, x, lowers, uppers):
for l,u in zip(lowers, uppers):
func(x,lower=l,upper=u)
n_samples = 1000
n_reps = 100
x = pd.Series(np.random.randn(n_samples))
# Sort the Series.
# For pandas>=1.0:
# x = x.sort_values(ignore_index=True)
x = x.sort_values().reset_index(drop=True)
# Assert equivalence of different methods.
assert(between_fast(x, 0, 1, True ).equals(between(x, 0, 1, True)))
assert(between_expr(x, 0, 1, True ).equals(between(x, 0, 1, True)))
assert(between_fast(x, 0, 1, False).equals(between(x, 0, 1, False)))
assert(between_expr(x, 0, 1, False).equals(between(x, 0, 1, False)))
# Benchmark repeated evaluations of between().
uppers = np.linspace(0, 3, n_reps)
lowers = -uppers
%timeit benchmark(between_fast, x, lowers, uppers)
%timeit benchmark(between, x, lowers, uppers)
%timeit benchmark(between_expr, x, lowers, uppers)
如果您正在处理多个值和多个输入,您还可以设置像这样的应用函数。在本例中,过滤数据帧以查找特定范围内的 GPS 位置。
def filter_values(lat,lon):
if abs(lat - 33.77) < .01 and abs(lon - -118.16) < .01:
return True
elif abs(lat - 37.79) < .01 and abs(lon - -122.39) < .01:
return True
else:
return False
df = df[df.apply(lambda x: filter_values(x['lat'],x['lon']),axis=1)]
而不是这个
df = df[99 <= df['closing_price'] <= 101]
你应该使用这个
df = df[(99 <= df['closing_price']) & (df['closing_price'] <= 101)]
我们必须使用 NumPy 的按位逻辑运算符
|
、&
、~
、^
来进行复合查询。
此外,括号对于运算符优先级也很重要。
有关更多信息,您可以访问链接:比较、掩码和布尔逻辑(摘自 Jake VanderPlas 的《Python 数据科学手册》)。