我正在开发一个利用 Microsoft 的 lightgbm (lgbm) 库的 python 脚本。虽然我的 lgbm 脚本与 xgboost 和随机森林的脚本非常相似(两者都工作正常),但在使用 lgbm 时,我似乎在 Mac Book Pro 和 MacStudios(带有 M1 芯片)上始终收到以下错误:
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.
The exit codes of the workers are {SIGSEGV(-11)}
相关代码:
_train_x, _val_x, _train_y, _val_y = train_test_split(_train_x, _train_y, test_size = 0.2)
lgbm_model = LGBMClassifier(bagging_fraction = 0.75, bagging_freq = 5, random_state=42, verbose=-1, force_col_wise=True)
kfoldcv = StratifiedKFold(n_splits=3, shuffle=True, random_state=7)
lgbm_random_search = RandomizedSearchCV(estimator = lgbm_model, param_distributions = self._param_dict, n_iter = self.num_searches, cv = kfoldcv, verbose=2, random_state=42, n_jobs=-1)
lgbm_random_search.fit(_train_x, _train_y)
self._CrossVal_largest_accscore = lgbm_random_search.best_score_
lgbm_model = LGBMClassifier(n_jobs=-1, verbose=-1, force_col_wise=True, bagging_fraction = 0.75, bagging_freq = 5, **lgbm_random_search.best_params_)
lgbm_model.fit(_train_x, _train_y, callbacks=[early_stopping(50), log_evaluation(50)], eval_set=[(_val_x,_val_y)])
注意,当我简单地删除子句 njobs=-1 时,我的程序在运行该行时就会终止:
lgbm_random_search.fit(_train_x, _train_y)
环境:
系统软件概述:
System Version: macOS 14.0 (23A344)
Kernel Version: Darwin 23.0.0
Boot Volume: Macintosh
HDBoot Mode: Normal
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled
硬件概述:
Model Name: MacBook Pro
Model Number: MK1F3B/A
Chip: Apple M1 Pro
Total Number of Cores: 10 (8 performance and 2 efficiency)
Memory: 16 GB
System Firmware Version: 10151.1.1
OS Loader Version: 10151.1.1
Activation Lock Status: Enabled
应用软件
Visual Studio Code==1.72.2
python==3.10.121
Python 包
anaconda-client==1.12.0
anaconda-navigator==2.4.2
conda==23.7.2
conda-build==3.26.0
joblib==1.3.0
lightgbm==4.0.0
matplotlib==3.7.1
matplotlib-inline==0.1.6
numpy==1.23.5
pandas==2.0.3
scikit-image==0.20.0
scikit-learn==1.3.0
scipy==1.11.1
statsmodels==0.14.0
sympy==1.12
xgboost==2.0.0
感谢您的时间和任何帮助,非常感谢
我已经审查并尝试了以下网站中的一些建议解决方案,但无济于事(例如删除 njobs 参数;添加 'pre_dispatch=2' 参数;重新安装 anaconda、lightgbm、joblib;使用较少数量的估计器 30 到 300 等) :
我想我已经找到了解决方案这里。
该问题似乎已通过使用以下方法安装 lightgbm 得到解决:
$ sudo conda install -c conda-forge lightgbm
而不是通常的 pip install 方法。
我希望这对其他人也有帮助!