我试图通过spark mllib试验信用卡欺诈检测数据集。与1(意味着欺诈)相比,我拥有的数据集有很多0(意思是非欺诈)。我想知道如上所述解决类不平衡问题我们在Spark中有任何可用的算法,比如SMOTE。我使用逻辑回归作为模型
您可以在逻辑回归中尝试weightCol,如下所示:
temp = train.groupby("LabelCol").count()
new_train = train.join(temp, "LabelCol", how = 'leftouter')
num_labels = train_data.select(countDistinct(train_data.score)).first()[0]
train1 = new_train.withColumn("weight",(new_train.count()/(num_labels * new_train["count"])))
# Logistic Regrestion Initiation
lr = LogisticRegression(weightCol = "weight", family = 'multinomial')