计算平均值并从数据中删除异常值除以r中的小时数

Question

part of data我试图从一个大型数据集（4个月的数据）计算每小时测量的平均值（大约每小时20个），但是我需要每小时删除异常值，定义为距离小时平均值2SD。

structure(list(YEAR = c(2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 
2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 
2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 
2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 
2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 
2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L, 2018L), MONTH = c(1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L), DAY = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L), HOUR = c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 
1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L), MINUTE = c(1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L
), SECOND = c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L), Tmp = c(25.6984, 25.6967, 25.6962, 25.6962, 
25.6955, 25.6949, 25.6959, 25.6944, 25.6954, 25.6954, 25.6958, 
25.6958, 25.6962, 25.6967, 25.6982, 25.6976, 25.6978, 25.6977, 
25.6975, 25.6979, 25.5552, 25.5577, 25.5579, 25.5573, 25.746, 
25.7248, 25.7164, 25.7249, 25.7379, 25.752, 25.7502, 25.7678, 
25.7805, 25.7871, 25.7863, 25.7856, 25.7948, 25.7939, 25.7953, 
25.7969, 25.7982, 25.7981, 25.7972, 25.7978, 25.644, 25.6451, 
25.6455, 25.6456, 25.6451, 25.6454)), row.names = c(NA, 50L), class = "data.frame")

Answer 1

我正在使用你发布的数据为df。

library(tidyverse)

# manually changing first value to create an outlier
df$Tmp[1] = 60

df %>%
  group_by(HOUR) %>%
  mutate(MEAN = mean(Tmp),
         SD = sd(Tmp),
         IsOutlier = ifelse(Tmp < MEAN-2*SD | Tmp > MEAN+2*SD, 1, 0)) %>%
  ungroup()

# # A tibble: 50 x 10
#    YEAR MONTH   DAY  HOUR MINUTE SECOND   Tmp  MEAN    SD IsOutlier
#   <int> <int> <int> <int>  <int>  <int> <dbl> <dbl> <dbl>     <dbl>
# 1  2018     1     1     0      1      0  60    27.4  7.67         1
# 2  2018     1     1     0      1      0  25.7  27.4  7.67         0
# 3  2018     1     1     0      1      0  25.7  27.4  7.67         0
# 4  2018     1     1     0      1      0  25.7  27.4  7.67         0
# 5  2018     1     1     0      1      0  25.7  27.4  7.67         0
# 6  2018     1     1     0      1      0  25.7  27.4  7.67         0
# 7  2018     1     1     0      1      0  25.7  27.4  7.67         0
# 8  2018     1     1     0      1      0  25.7  27.4  7.67         0
# 9  2018     1     1     0      1      0  25.7  27.4  7.67         0
#10  2018     1     1     0      1      0  25.7  27.4  7.67         0
# # ... with 40 more rows

您可以看到第一行被归类为异常值行，您可以在以后使用... %>% filter(IsOutlier == 0)删除它。

我离开了我创建的列，以便了解该过程的工作原理。

Answer 2

考虑基础R的ave（来自内置的stats库进行内联聚合）来计算异常值：

df$outlier <- ave(df$Tmp, df$HOUR, 
                  FUN=function(x) (x < (mean(x) - sd(x)*2)) | (x > (mean(x) + sd(x)*2)))

然后相应的子集：

subdf <- subset(df, outlier == 0)