我正在尝试从具有日期的GPS高程位移散点图中过滤掉一些离群值
我正在尝试使用df.rolling计算每个窗口的中位数和标准偏差,如果该点大于3个标准偏差,则将其删除。
但是,我想不出一种方法来遍历该列并比较所计算的中位数滚动。
这里是我到目前为止的代码
import pandas as pd
import numpy as np
def median_filter(df, window):
cnt = 0
median = df['b'].rolling(window).median()
std = df['b'].rolling(window).std()
for row in df.b:
#compare each value to its median
df = pd.DataFrame(np.random.randint(0,100,size=(100,2)), columns = ['a', 'b'])
median_filter(df, 10)
我如何遍历并比较每个点并将其删除?
仅过滤数据框
df['median']= df['b'].rolling(window).median()
df['std'] = df['b'].rolling(window).std()
#filter setup
df = df[(df.b <= df['median']+3*df['std']) & (df.b >= df['median']-3*df['std'])]
可能会有更泛泛的方法来执行此操作-这有点麻烦,它依赖于一种将原始df的索引映射到每个滚动窗口的手动方法。 (我选了6号)。直到第6行的记录与first窗口关联;第7行是第二个窗口,依此类推。
n = 100
df = pd.DataFrame(np.random.randint(0,n,size=(n,2)), columns = ['a','b'])
## set window size
window=6
std = 1 # I set it at just 1; with real data and larger windows, can be larger
## create df with rolling stats, upper and lower bounds
bounds = pd.DataFrame({'median':df['b'].rolling(window).median(),
'std':df['b'].rolling(window).std()})
bounds['upper']=bounds['median']+bounds['std']*std
bounds['lower']=bounds['median']-bounds['std']*std
## here, we set an identifier for each window which maps to the original df
## the first six rows are the first window; then each additional row is a new window
bounds['window_id']=np.append(np.zeros(window),np.arange(1,n-window+1))
## then we can assign the original 'b' value back to the bounds df
bounds['b']=df['b']
## and finally, keep only rows where b falls within the desired bounds
bounds.loc[bounds.eval("lower<b<upper")]
这是我创建中值过滤器的方法:
def median_filter(num_std=3):
def _median_filter(x):
_median = np.median(x)
_std = np.std(x)
s = x[-1]
return (s >= _median - num_std * _std and s <= _median + num_std * _std)
return _median_filter
df.y.rolling(window).apply(median_filter(num_std=3), raw=True)