MLR3：ROC 曲线和标准差/IC 的提取？

Question

我想提取在任务的多个学习者基准中获得的结果的标准差和/或 IC95，以确保结果完整。我读到了这个：mlr3 k 倍交叉验证重采样的标准差

但是不知道从2月份开始有没有新的东西...

这是我的代码和 bmr 的图：

resampling_outer = rsmp("cv", folds = 5)
resampling_inner = rsmp("cv", folds = 3)

set.seed(372)
resampling_outer$instantiate(task_wilcox)
resampling_inner$instantiate(task_wilcox)

at_xgboost = auto_tuner(tuner=tnr("mbo"), learner = xgboost,resampling = resampling_inner, measure = msr("classif.auc"),term_evals = 20,store_tuning_instance = TRUE,store_models = TRUE)
at_ranger = auto_tuner(tuner=tnr("mbo"), learner = ranger,resampling = resampling_inner, measure = msr("classif.auc"),term_evals = 20,store_tuning_instance = TRUE,store_models = TRUE)
at_svm = auto_tuner(tuner=tnr("mbo"), learner = svm,resampling = resampling_inner, measure = msr("classif.auc"),term_evals = 20,store_tuning_instance = TRUE,store_models = TRUE)
at_knn = auto_tuner(tuner=tnr("mbo"), learner = knn,resampling = resampling_inner, measure = msr("classif.auc"),term_evals = 20,store_tuning_instance = TRUE,store_models = TRUE)


learners <- c(at_xgboost, at_svm, at_ranger, at_knn)

measures = msrs(c("classif.auc", "classif.bacc", "classif.bbrier"))

#Benchmarking
set.seed(372)
design = benchmark_grid(tasks = task_wilcox, learners = learners, resamplings = resampling_outer)
bmr = benchmark(design, store_models = TRUE)
results <- bmr$aggregate(measures)
print(results)
autoplot(bmr, measure = msr("classif.auc"))
autoplot(bmr, type = "roc")

results
   nr     task_id                learner_id resampling_id iters classif.auc classif.bacc classif.bbrier
1:  1 data_wilcox       scale.xgboost.tuned            cv     5   0.6112939    0.5767294      0.2326787
2:  2 data_wilcox           scale.svm.tuned            cv     5   0.5226407    0.5010260      0.1893202
3:  3 data_wilcox scale.random_forest.tuned            cv     5   0.6200084    0.5614843      0.2229120
4:  4 data_wilcox           scale.knn.tuned            cv     5   0.5731675    0.5002955      0.1917721

extract_inner_tuning_results(bmr)[,list(learner_id, classif.auc)]
                   learner_id classif.auc
 1:       scale.xgboost.tuned   0.6231350
 2:       scale.xgboost.tuned   0.6207103
 3:       scale.xgboost.tuned   0.6175323
 4:       scale.xgboost.tuned   0.6195693
 5:       scale.xgboost.tuned   0.6222398
 6:           scale.svm.tuned   0.5891432
 7:           scale.svm.tuned   0.5837583
 8:           scale.svm.tuned   0.5767444
 9:           scale.svm.tuned   0.6027165
10:           scale.svm.tuned   0.6082825
11: scale.random_forest.tuned   0.6287649
12: scale.random_forest.tuned   0.6165179
13: scale.random_forest.tuned   0.6288599
14: scale.random_forest.tuned   0.6259322
15: scale.random_forest.tuned   0.6234295
16:           scale.knn.tuned   0.5931790
17:           scale.knn.tuned   0.5926835
18:           scale.knn.tuned   0.5931790
19:           scale.knn.tuned   0.5929156
20:           scale.knn.tuned   0.5929156

正如您在ROC曲线上看到的，有色和半透明边缘存在标准偏差或IC，但如何提取它？我想对于标准差，我必须提取嵌套 CV 的所有外部重采样的结果，但没有办法直接提取它（它通过边距出现在 ROC 曲线上，我想它存在于某个地方。 ..).

第二个问题，在我绘制的每个学习者的 AUC 箱形图上，我真的不知道它是如何构建箱形图的，因为它与测试集上的结果不对应（外循环 => 重新采样外层） )...

最后一个问题：你知道如何个性化mlr3中的roc曲线吗？例如，如果我想在方案上添加 AUC，或者删除曲线周围的边距...

谢谢！

Answer 1

1. 您可以通过设置

   predict_type = "se"

   来获取标准差和预测，请参阅 https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-predicting。这是由学习者内部计算的，而不是交叉验证的折叠，但很可能是您想要的。
2. 您得到的结果数字应与绘图相对应；没有完整的可重现示例，我无法告诉你更多...

3. mlr3

   中的所有绘图函数都返回

   ggplot2

   对象；您可以像其他绘图一样自定义它们。