如何量化一系列预测的一致性，包括。预测置信度，使用 sklearn 或类似库中的标准函数

Question

假设我对同一对象有一系列类别预测。然后我想测量该序列的一致性。例如，像

class_a, class_a, class_a, class_a

这样的一致预测序列应该给出高分。像

class_a, class_b, class_a, class_c

这样的不一致序列应该会导致较低的分数。

它最好适用于任意数量的可能类别，并且还考虑到预测置信度。序列

class_a (0.9), class_b (0.9), class_a (0.9), class_c (0.9)

应该给出比

class_a (0.9), class_b (0.2), class_a (0.8), class_c (0.3)

更低的分数，因为当预测与高置信度不一致时，它是不好的。

我可以自己构建一些东西，但我想知道是否有一个标准的 sklearn （或类似）函数？预先感谢！

对这个问题的评论建议使用Spearman相关系数或Kandell相关系数。我也会研究一下。

Answer 1

不确定这是否是您要找的:

import numpy as np
from collections import Counter

def consistency_score(predictions, confidences):
    """
    Calculate a consistency score for a sequence of predictions.
    

    """
    # Calculate base consistency as the frequency of the most common class
    most_common_class, most_common_freq = Counter(predictions).most_common(1)[0]
    base_consistency = most_common_freq / len(predictions)
    
    # Adjust consistency based on confidences
    # Penalize deviations from the most common class, especially with high confidence
    penalty = sum(conf for pred, conf in zip(predictions, confidences) if pred != most_common_class) / len(predictions)
    adjusted_consistency = max(0, base_consistency - penalty)
    
    return adjusted_consistency

• 示例：

    predictions = ["class_a", "class_b", "class_a", "class_c"]
    confidences = [0.9, 0.9, 0.9, 0.9]
    score_high_confidence = consistency_score(predictions, confidences)
  
    predictions_low_confidence = ["class_a", "class_b", "class_a", "class_c"]
    confidences_low_confidence = [0.9, 0.2, 0.8, 0.3]
    score_low_confidence = consistency_score(predictions_low_confidence, confidences_low_confidence)
  
    print(f"High confidence inconsistencies score: {score_high_confidence}")
    print(f"Lower confidence inconsistencies score: {score_low_confidence}")