numpy.average()
有权重选项,但 numpy.std()
没有。有人对解决方法有建议吗?
下面这个简短的“手工计算”怎么样?
def weighted_avg_and_std(values, weights):
"""
Return the weighted average and standard deviation.
They weights are in effect first normalized so that they
sum to 1 (and so they must not all be 0).
values, weights -- NumPy ndarrays with the same shape.
"""
average = numpy.average(values, weights=weights)
# Fast and numerically precise:
variance = numpy.average((values-average)**2, weights=weights)
return (average, math.sqrt(variance))
statsmodels
中有一个类可以轻松计算加权统计:statsmodels.stats.weightstats.DescrStatsW
。
假设这个数据集和权重:
import numpy as np
from statsmodels.stats.weightstats import DescrStatsW
array = np.array([1,2,1,2,1,2,1,3])
weights = np.ones_like(array)
weights[3] = 100
您初始化类(请注意,此时您必须传入校正因子,即 delta 自由度):
weighted_stats = DescrStatsW(array, weights=weights, ddof=0)
然后你可以计算:
.mean
加权平均值:
>>> weighted_stats.mean
1.97196261682243
.std
加权标准差:
>>> weighted_stats.std
0.21434289609681711
.var
加权方差:
>>> weighted_stats.var
0.045942877107170932
.std_mean
加权平均值的标准误差:
>>> weighted_stats.std_mean
0.020818822467555047
以防万一您对标准误差和标准差之间的关系感兴趣:标准误差(对于
ddof == 0
)计算为加权标准差除以权重总和的平方根减 1 ( GitHub 上 statsmodels
版本 0.9 的对应源码):
standard_error = standard_deviation / sqrt(sum(weights) - 1)
还有另一种选择:
np.sqrt(np.cov(values, aweights=weights))
numpy/scipy 中似乎还没有这样的函数,但有一个 ticket 建议添加此功能。在那里你会发现Statistics.py,它实现了加权标准差。
gaborous提出了一个非常好的例子:
import pandas as pd
import numpy as np
# X is the dataset, as a Pandas' DataFrame
# Compute the weighted sample mean (fast, efficient and precise)
mean = mean = np.ma.average(X, axis=0, weights=weights)
# Convert to a Pandas' Series (it's just aesthetic and more
# ergonomic; no difference in computed values)
mean = pd.Series(mean, index=list(X.keys()))
xm = X-mean # xm = X diff to mean
# fill NaN with 0
# a variance of 0 is just void, but at least it keeps the other
# covariance's values computed correctly))
xm = xm.fillna(0)
# Compute the unbiased weighted sample covariance
sigma2 = 1./(w.sum()-1) * xm.mul(w, axis=0).T.dot(xm);
自从“加权样本标准差Python”谷歌搜索导致这篇文章以来,“频率权重”意义上的“样本”或“无偏”标准差的后续内容:
def frequency_sample_std_dev(X, n):
"""
Sample standard deviation for X and n,
where X[i] is the quantity each person in group i has,
and n[i] is the number of people in group i.
See Equation 6.4 of:
Montgomery, Douglas, C. and George C. Runger. Applied Statistics
and Probability for Engineers, Enhanced eText. Available from:
WileyPLUS, (7th Edition). Wiley Global Education US, 2018.
"""
n_groups = len(n)
n_people = sum(n)
lhs_numerator = sum([ni*Xi**2 for Xi, ni in zip(X, n)])
rhs_numerator = sum([Xi*ni for Xi, ni in zip(X,n)])**2/n_people
denominator = n_people-1
var = (lhs_numerator - rhs_numerator) / denominator
std = sqrt(var)
return std
或者修改@Eric的答案如下:
def weighted_sample_avg_std(values, weights):
"""
Return the weighted average and weighted sample standard deviation.
values, weights -- Numpy ndarrays with the same shape.
Assumes that weights contains only integers (e.g. how many samples in each group).
See also https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Frequency_weights
"""
average = np.average(values, weights=weights)
variance = np.average((values-average)**2, weights=weights)
variance = variance*sum(weights)/(sum(weights)-1)
return (average, sqrt(variance))
print(weighted_sample_avg_std(X, n))
我只是在寻找与 numpy
np.std
函数等效的 API,该函数还允许设置 axis
参数:
(我只是用二维测试了它,所以如果有不正确的地方请随时改进。)
def std(values, weights=None, axis=None):
"""
Return the weighted standard deviation.
axis -- the axis for std calculation
values, weights -- Numpy ndarrays with the same shape on the according axis.
"""
average = np.expand_dims(np.average(values, weights=weights, axis=axis), axis=axis)
# Fast and numerically precise:
variance = np.average((values-average)**2, weights=weights, axis=axis)
return np.sqrt(variance)
感谢 Eric O Lebigot 的原始答案。