NumPy 中的加权标准差

问题描述 投票:0回答:7

numpy.average()
有权重选项,但
numpy.std()
没有。有人对解决方法有建议吗?

python numpy statsmodels standard-deviation weighted
7个回答
186
投票

下面这个简短的“手工计算”怎么样?

def weighted_avg_and_std(values, weights):
    """
    Return the weighted average and standard deviation.

    They weights are in effect first normalized so that they 
    sum to 1 (and so they must not all be 0).

    values, weights -- NumPy ndarrays with the same shape.
    """
    average = numpy.average(values, weights=weights)
    # Fast and numerically precise:
    variance = numpy.average((values-average)**2, weights=weights)
    return (average, math.sqrt(variance))

    

54
投票

statsmodels
中有一个类可以轻松计算加权统计:
statsmodels.stats.weightstats.DescrStatsW

假设这个数据集和权重:

import numpy as np
from statsmodels.stats.weightstats import DescrStatsW

array = np.array([1,2,1,2,1,2,1,3])
weights = np.ones_like(array)
weights[3] = 100

您初始化类(请注意,此时您必须传入校正因子,即 delta 自由度):

weighted_stats = DescrStatsW(array, weights=weights, ddof=0)

然后你可以计算:

  • .mean
    加权平均值

    >>> weighted_stats.mean      
    1.97196261682243
    
  • .std
    加权标准差:

    >>> weighted_stats.std       
    0.21434289609681711
    
  • .var
    加权方差

    >>> weighted_stats.var       
    0.045942877107170932
    
  • .std_mean
    加权平均值的标准误差

    >>> weighted_stats.std_mean  
    0.020818822467555047
    

    以防万一您对标准误差和标准差之间的关系感兴趣:标准误差(对于

    ddof == 0
    )计算为加权标准差除以权重总和的平方根减 1 ( GitHub 上
    statsmodels
    版本 0.9 的对应源码
    ):

    standard_error = standard_deviation / sqrt(sum(weights) - 1)
    

41
投票

还有另一种选择:

np.sqrt(np.cov(values, aweights=weights))

6
投票

numpy/scipy 中似乎还没有这样的函数,但有一个 ticket 建议添加此功能。在那里你会发现Statistics.py,它实现了加权标准差。


2
投票

gaborous提出了一个非常好的例子:

import pandas as pd
import numpy as np
# X is the dataset, as a Pandas' DataFrame
# Compute the weighted sample mean (fast, efficient and precise)
mean = mean = np.ma.average(X, axis=0, weights=weights) 

# Convert to a Pandas' Series (it's just aesthetic and more 
# ergonomic; no difference in computed values)
mean = pd.Series(mean, index=list(X.keys())) 
xm = X-mean # xm = X diff to mean
# fill NaN with 0 
# a variance of 0 is just void, but at least it keeps the other
# covariance's values computed correctly))
xm = xm.fillna(0) 
# Compute the unbiased weighted sample covariance
sigma2 = 1./(w.sum()-1) * xm.mul(w, axis=0).T.dot(xm); 

加权无偏样本协方差的正确方程,URL(版本:2016-06-28)


1
投票

自从“加权样本标准差Python”谷歌搜索导致这篇文章以来,“频率权重”意义上的“样本”或“无偏”标准差的后续内容:

def frequency_sample_std_dev(X, n):
    """
    Sample standard deviation for X and n,
    where X[i] is the quantity each person in group i has,
    and n[i] is the number of people in group i.
    See Equation 6.4 of:
    Montgomery, Douglas, C. and George C. Runger. Applied Statistics 
     and Probability for Engineers, Enhanced eText. Available from: 
      WileyPLUS, (7th Edition). Wiley Global Education US, 2018.
    """
    n_groups = len(n)
    n_people = sum(n)
    lhs_numerator = sum([ni*Xi**2 for Xi, ni in zip(X, n)])
    rhs_numerator = sum([Xi*ni for Xi, ni in zip(X,n)])**2/n_people
    denominator = n_people-1
    var = (lhs_numerator - rhs_numerator) / denominator
    std = sqrt(var)
    return std

或者修改@Eric的答案如下:

def weighted_sample_avg_std(values, weights):
    """
    Return the weighted average and weighted sample standard deviation.

    values, weights -- Numpy ndarrays with the same shape.
    
    Assumes that weights contains only integers (e.g. how many samples in each group).
    
    See also https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Frequency_weights
    """
    average = np.average(values, weights=weights)
    variance = np.average((values-average)**2, weights=weights)
    variance = variance*sum(weights)/(sum(weights)-1)
    return (average, sqrt(variance))

print(weighted_sample_avg_std(X, n))

0
投票

我只是在寻找与 numpy

np.std
函数等效的 API,该函数还允许设置
axis
参数:

(我只是用二维测试了它,所以如果有不正确的地方请随时改进。)

def std(values, weights=None, axis=None):
    """
    Return the weighted standard deviation.
    axis -- the axis for std calculation
    values, weights -- Numpy ndarrays with the same shape on the according axis.
    """
    average = np.expand_dims(np.average(values, weights=weights, axis=axis), axis=axis)
    # Fast and numerically precise:
    variance = np.average((values-average)**2, weights=weights, axis=axis)
    return np.sqrt(variance)

感谢 Eric O Lebigot 的原始答案

© www.soinside.com 2019 - 2024. All rights reserved.