在R中使用mlr包，我正在创建随机林模型。为了评估模型的分类准确性，我使用here中描述的嵌套重采样。我的问题是内环中随机森林模型的分类准确度通常比外环结果高15％。我在内环中观察到~85％的分类精度，但外环的准确度通常最终达到70％左右。我无法在这里提供数据，但我正在粘贴我正在使用的代码。
怎么可能？可能是什么原因？
rf_param_set <- makeParamSet(
ParamHelpers::makeDiscreteParam('mtry', values = c(3, 7, 14)),
ParamHelpers::makeDiscreteParam('ntree', values = c(1000, 2000))
)
rf_tune_ctrl <- makeTuneControlGrid()
rf_inner_resample <- makeResampleDesc('Bootstrap', iters = 5)
acc632plus <- setAggregation(acc, b632plus)
rf_learner <- makeTuneWrapper('classif.randomForest',
resampling = rf_inner_resample,
measures = list(acc),
par.set = rf_param_set,
control = rf_tune_ctrl,
show.info = TRUE)
# rf_outer_resample <- makeResampleDesc('Subsample', iters = 10, split = 2/3)
rf_outer_resample <- makeResampleDesc('Bootstrap', iters = 10, predict = 'both')
rf_result_resample <- resample(rf_learner, clf_task,
resampling = rf_outer_resample,
extract = getTuneResult,
measures = list(acc, acc632plus),
show.info = TRUE)
您可以在下面生成结果。
Resampling: OOB bootstrapping
Measures: acc.train acc.test acc.test
[Tune] Started tuning learner classif.randomForest for parameter set:
Type len Def Constr Req Tunable Trafo
mtry discrete - - 3,7,14 - TRUE -
ntree discrete - - 1000,2000 - TRUE -
With control class: TuneControlGrid
Imputation value: -0
[Tune-x] 1: mtry=3; ntree=1000
[Tune-y] 1: acc.test.mean=0.8415307; time: 0.1 min
[Tune-x] 2: mtry=7; ntree=1000
[Tune-y] 2: acc.test.mean=0.8405726; time: 0.1 min
[Tune-x] 3: mtry=14; ntree=1000
[Tune-y] 3: acc.test.mean=0.8330845; time: 0.1 min
[Tune-x] 4: mtry=3; ntree=2000
[Tune-y] 4: acc.test.mean=0.8415809; time: 0.3 min
[Tune-x] 5: mtry=7; ntree=2000
[Tune-y] 5: acc.test.mean=0.8395083; time: 0.3 min
[Tune-x] 6: mtry=14; ntree=2000
[Tune-y] 6: acc.test.mean=0.8373584; time: 0.3 min
[Tune] Result: mtry=3; ntree=2000 : acc.test.mean=0.8415809
[Resample] iter 1: 0.9961089 0.7434555 0.7434555
[Tune] Started tuning learner classif.randomForest for parameter set:
Type len Def Constr Req Tunable Trafo
mtry discrete - - 3,7,14 - TRUE -
ntree discrete - - 1000,2000 - TRUE -
With control class: TuneControlGrid
Imputation value: -0
[Tune-x] 1: mtry=3; ntree=1000
[Tune-y] 1: acc.test.mean=0.8479891; time: 0.1 min
[Tune-x] 2: mtry=7; ntree=1000
[Tune-y] 2: acc.test.mean=0.8578465; time: 0.1 min
[Tune-x] 3: mtry=14; ntree=1000
[Tune-y] 3: acc.test.mean=0.8556608; time: 0.1 min
[Tune-x] 4: mtry=3; ntree=2000
[Tune-y] 4: acc.test.mean=0.8502869; time: 0.3 min
[Tune-x] 5: mtry=7; ntree=2000
[Tune-y] 5: acc.test.mean=0.8601446; time: 0.3 min
[Tune-x] 6: mtry=14; ntree=2000
[Tune-y] 6: acc.test.mean=0.8586638; time: 0.3 min
[Tune] Result: mtry=7; ntree=2000 : acc.test.mean=0.8601446
[Resample] iter 2: 0.9980545 0.7032967 0.7032967
[Tune] Started tuning learner classif.randomForest for parameter set:
Type len Def Constr Req Tunable Trafo
mtry discrete - - 3,7,14 - TRUE -
ntree discrete - - 1000,2000 - TRUE -
With control class: TuneControlGrid
Imputation value: -0
[Tune-x] 1: mtry=3; ntree=1000
[Tune-y] 1: acc.test.mean=0.8772566; time: 0.1 min
[Tune-x] 2: mtry=7; ntree=1000
[Tune-y] 2: acc.test.mean=0.8750990; time: 0.1 min
[Tune-x] 3: mtry=14; ntree=1000
[Tune-y] 3: acc.test.mean=0.8730733; time: 0.1 min
[Tune-x] 4: mtry=3; ntree=2000
[Tune-y] 4: acc.test.mean=0.8782829; time: 0.3 min
[Tune-x] 5: mtry=7; ntree=2000
[Tune-y] 5: acc.test.mean=0.8741619; time: 0.3 min
[Tune-x] 6: mtry=14; ntree=2000
[Tune-y] 6: acc.test.mean=0.8687918; time: 0.3 min
[Tune] Result: mtry=3; ntree=2000 : acc.test.mean=0.8782829
[Resample] iter 3: 0.9902724 0.7329843 0.7329843
你所看到的正是你想要使用嵌套重采样的原因 - 内部重采样循环（在某种程度上）过度拟合数据，并给出了泛化性能的误导性印象。通过外部重采样，您可以检测到（精度较低）。
mlr教程有一个更详细的页面（https://mlr.mlr-org.com/articles/tutorial/nested_resampling.html）。一般来说，你没有看到这些结果，因为你做错了什么（除非你以某种方式手动拆分数据），你只是使用一种强大的优化方法，它可以比它应该更多地优化 - 但是你正在通过嵌套的重采样来检测它。
您可以尝试使用交叉验证而不是引导;这可以提供更一致的结果。