Python LightGBM 错误:joblib.externals.loky.process_executor.TermulatedWorkerError {SIGSEGV(-11)}

问题描述 投票:0回答:1

我正在开发一个利用 Microsoft 的 lightgbm (lgbm) 库的 python 脚本。虽然我的 lgbm 脚本与 xgboost 和随机森林的脚本非常相似(两者都工作正常),但在使用 lgbm 时,我似乎在 Mac Book Pro 和 MacStudios(带有 M1 芯片)上始终收到以下错误:

joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGSEGV(-11)}

相关代码:

_train_x, _val_x, _train_y, _val_y = train_test_split(_train_x, _train_y, test_size = 0.2)
    
lgbm_model = LGBMClassifier(bagging_fraction = 0.75, bagging_freq = 5, random_state=42, verbose=-1, force_col_wise=True)
    
kfoldcv = StratifiedKFold(n_splits=3, shuffle=True, random_state=7)
    
lgbm_random_search = RandomizedSearchCV(estimator = lgbm_model, param_distributions = self._param_dict, n_iter = self.num_searches, cv = kfoldcv, verbose=2, random_state=42, n_jobs=-1)
    
lgbm_random_search.fit(_train_x, _train_y)

self._CrossVal_largest_accscore = lgbm_random_search.best_score_
    
lgbm_model = LGBMClassifier(n_jobs=-1, verbose=-1, force_col_wise=True, bagging_fraction = 0.75, bagging_freq = 5, **lgbm_random_search.best_params_)

lgbm_model.fit(_train_x, _train_y, callbacks=[early_stopping(50), log_evaluation(50)], eval_set=[(_val_x,_val_y)])

注意,当我简单地删除子句 njobs=-1 时,我的程序在运行该行时就会终止:

lgbm_random_search.fit(_train_x, _train_y)

环境:

系统软件概述:

System Version: macOS 14.0 (23A344)
Kernel Version: Darwin 23.0.0
Boot Volume: Macintosh 
HDBoot Mode: Normal
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled

硬件概述:

  Model Name: MacBook Pro
  Model Number: MK1F3B/A
  Chip: Apple M1 Pro
  Total Number of Cores: 10 (8 performance and 2 efficiency)
  Memory: 16 GB
  System Firmware Version: 10151.1.1
  OS Loader Version: 10151.1.1
  Activation Lock Status: Enabled

应用软件

Visual Studio Code==1.72.2
python==3.10.121

Python 包


anaconda-client==1.12.0
anaconda-navigator==2.4.2 
conda==23.7.2
conda-build==3.26.0
joblib==1.3.0
lightgbm==4.0.0
matplotlib==3.7.1
matplotlib-inline==0.1.6
numpy==1.23.5
pandas==2.0.3
scikit-image==0.20.0
scikit-learn==1.3.0
scipy==1.11.1
statsmodels==0.14.0
sympy==1.12
xgboost==2.0.0

感谢您的时间和任何帮助,非常感谢

我已经审查并尝试了以下网站中的一些建议解决方案,但无济于事(例如删除 njobs 参数;添加 'pre_dispatch=2' 参数;重新安装 anaconda、lightgbm、joblib;使用较少数量的估计器 30 到 300 等) :

python machine-learning scikit-learn parallel-processing lightgbm
1个回答
0
投票

我想我已经找到了解决方案这里

该问题似乎已通过使用以下方法安装 lightgbm 得到解决:

$ sudo conda install -c conda-forge lightgbm 

而不是通常的 pip install 方法。

我希望这对其他人也有帮助!

© www.soinside.com 2019 - 2024. All rights reserved.