RegressionGLM /泊松测试需要花费很多时间才能完成模型的生成以及如何读取水日志

问题描述 投票:0回答:1

我想知道h2o日志中的条目是什么意思。日志来自一个测试(glm,poisson,y = testcolA),该测试运行了数小时(> = 6个小时并且还在计时),我们必须停止该测试。一个类似的测试(glm,泊松,y = testcolB)在一个小时左右的时间内完成。 Trainingset的大小为131GB。

集群信息:

R is connected to the H2O cluster:
   H2O cluster uptime:         17 seconds 996 milliseconds
   H2O cluster timezone:       Etc/UTC
   H2O data parsing timezone:  UTC
   H2O cluster version:        3.24.0.3
   H2O cluster version age:    5 months and 18 days !!!
   H2O cluster name:           h2o-ca60e2ab-9e9b-4154-bb35-a52fc092abdd
   H2O cluster total nodes:    12
   H2O cluster total memory:   372.00 GB
   H2O cluster total cores:    96
   H2O cluster allowed cores:  96
   H2O cluster healthy:        TRUE
   H2O Connection ip:          localhost
   H2O Connection port:        3000
   H2O Connection proxy:       NA
   H2O Internal Security:      FALSE
   H2O API Extensions:         Amazon S3, XGBoost, Algos, AutoML, Core V3, Core V4
   R Version:                  R version 3.6.0 (2019-04-26)

日志片段:

...snip...
10-25 22:05:30.111 172.31.20.64:3000     206    FJ-1-23   INFO: Building H2O GLM model with these parameters:
10-25 22:05:30.122 172.31.20.64:3000     206    FJ-1-23   INFO: {"_train":{"name":"RTMP_sid_9e49_4","type":"Key"},"_valid":null,"_nfolds":0,"_keep_cross_validation_models":true,"_keep_cross_validation_predictions":false,"_keep_cross_validation_fold_assignment":false,"_parallelize_cross_validation":true,"_auto_rebalance":true,"_seed":-1,"_fold_assignment":"AUTO","_categorical_encoding":"AUTO","_max_categorical_levels":10,"_distribution":"AUTO","_tweedie_power":1.5,"_quantile_alpha":0.5,"_huber_alpha":0.9,"_ignored_columns":[...snip...],"_ignore_const_cols":true,"_weights_column":null,"_offset_column":null,"_fold_column":null,"_check_constant_response":true,"_is_cv_model":false,"_score_each_iteration":false,"_max_runtime_secs":0.0,"_stopping_rounds":3,"_stopping_metric":"deviance","_stopping_tolerance":1.0E-4,"_response_column":snip,"_balance_classes":false,"_max_after_balance_size":5.0,"_class_sampling_factors":null,"_max_confusion_matrix_size":20,"_checkpoint":null,"_pretrained_autoencoder":null,"_custom_metric_func":null,"_export_checkpoints_dir":null,"_standardize":true,"_family":"poisson","_link":"family_default","_solver":"AUTO","_tweedie_variance_power":0.0,"_tweedie_link_power":1.0,"_theta":1.0E-10,"_invTheta":1.0E10,"_alpha":null,"_lambda":null,"_missing_values_handling":"MeanImputation","_prior":-1.0,"_lambda_search":false,"_nlambdas":-1,"_non_negative":false,"_exactLambdas":false,"_lambda_min_ratio":-1.0,"_use_all_factor_levels":false,"_max_iterations":-1,"_intercept":true,"_beta_epsilon":1.0E-4,"_objective_epsilon":-1.0,"_gradient_epsilon":-1.0,"_obj_reg":-1.0,"_compute_p_values":false,"_remove_collinear_columns":false,"_interactions":null,"_interaction_pairs":null,"_early_stopping":true,"_beta_constraints":null,"_max_active_predictors":-1,"_stdOverride":false}
10-25 22:05:30.127 172.31.20.64:3000     206    FJ-1-23   INFO: Dropping ignored columns: [..snip..]
10-25 22:05:30.132 172.31.20.64:3000     206    FJ-1-23   INFO: train dataset already contains 489 (non-empty)  chunks. No need to rebalance. [desiredChunks=8, rebalanceRatio=1.0]
10-25 22:05:35.360 172.31.20.64:3000     206    FJ-1-23   INFO: Starting model MyModel
10-25 22:05:35.557 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=0 lmb=.14E2 obj=530570.0 imp=.1E1 bdf=.11E2] Got 2151 active columns out of 2163 total
10-25 22:05:35.558 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=0 lmb=.14E2 obj=530570.0 imp=.1E1 bdf=.11E2] picked solver IRLSM
10-25 23:20:14.377 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-25 23:20:14.377 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 0.27587151573955293 >  eps = 1.0E-4
10-25 23:20:14.378 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=0 lmb=.14E2 obj=530570.0 imp=.1E1 bdf=.11E2] computed in 4383038+95782=4478820ms, step = 1, l1solver iter = 10000, gerr = 0.27587151573955293
10-26 00:31:02.569 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=1 lmb=.14E2 obj=525370.0 imp=.98E-2 bdf=.94E0] Scoring after 8732476ms
10-26 00:31:20.993 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=1 lmb=.14E2 obj=525370.0 imp=.98E-2 bdf=.94E0] Training metrics computed in 18424ms
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=1 lmb=.14E2 obj=525370.0 imp=.98E-2 bdf=.94E0] Model Metrics Type: RegressionGLM
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 3.63462328E11
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 602878.4
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 525228.0
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 130539.59
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.968928
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.0042488E7
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.275379E12
10-26 00:31:20.994 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2753832E12
10-26 00:31:21.088 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=1 lmb=.14E2 obj=525370.0 imp=.98E-2 bdf=.94E0] computed in 4182012+1+84696=4266709ms, step = 0.33, l1solver iter = 10000, gerr = 0.27587151573955293
10-26 01:37:31.864 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 01:37:31.864 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 55571.13056411838 >  eps = 1.0E-4
10-26 01:38:38.508 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=2 lmb=.14E2 obj=524670.0 imp=.13E-2 bdf=.65E0] Scoring after 4037421ms
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=2 lmb=.14E2 obj=524670.0 imp=.13E-2 bdf=.65E0] Training metrics computed in 17184ms
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=2 lmb=.14E2 obj=524670.0 imp=.13E-2 bdf=.65E0] Model Metrics Type: RegressionGLM
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 3.48753723E11
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 590553.75
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 524416.7
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 130637.945
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.955455
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.0042486E7
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.26723E12
10-26 01:38:55.693 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2672342E12
10-26 01:38:55.704 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=2 lmb=.14E2 obj=524670.0 imp=.13E-2 bdf=.65E0] computed in 3914318+56458+83839=4054615ms, step = 0.33, l1solver iter = 10000, gerr = 55571.13056411838
10-26 02:44:41.208 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 02:44:41.208 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 5768.096703827045 >  eps = 1.0E-4
10-26 02:46:20.563 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=3 lmb=.14E2 obj=523790.0 imp=.17E-2 bdf=.11E0] Scoring after 4044860ms
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=3 lmb=.14E2 obj=523790.0 imp=.17E-2 bdf=.11E0] Training metrics computed in 19030ms
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=3 lmb=.14E2 obj=523790.0 imp=.17E-2 bdf=.11E0] Model Metrics Type: RegressionGLM
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 3.48649226E11
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 590465.25
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 523514.56
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 130344.89
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.952192
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.0042381E7
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.2581693E12
10-26 02:46:39.594 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2581735E12
10-26 02:46:39.601 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=3 lmb=.14E2 obj=523790.0 imp=.17E-2 bdf=.11E0] computed in 3889387+56117+118392=4063896ms, step = 0.10890000000000001, l1solver iter = 10000, gerr = 5768.096703827045
10-26 03:52:19.527 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 03:52:19.527 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 4784.969933090491 >  eps = 1.0E-4
10-26 03:53:25.773 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=4 lmb=.14E2 obj=523120.0 imp=.13E-2 bdf=.46E0] Scoring after 4006173ms
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=4 lmb=.14E2 obj=523120.0 imp=.13E-2 bdf=.46E0] Training metrics computed in 20423ms
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=4 lmb=.14E2 obj=523120.0 imp=.13E-2 bdf=.46E0] Model Metrics Type: RegressionGLM
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 4.97980703E11
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 705677.5
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 522778.22
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 130602.62
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.941541
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.0042353E7
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.2507737E12
10-26 03:53:46.196 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2507779E12
10-26 03:53:46.201 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=4 lmb=.14E2 obj=523120.0 imp=.13E-2 bdf=.46E0] computed in 3886191+53735+86674=4026600ms, step = 0.33, l1solver iter = 10000, gerr = 4784.969933090491
10-26 04:59:31.210 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 04:59:31.211 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 44821.17005179174 >  eps = 1.0E-4
10-26 05:01:09.199 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=5 lmb=.14E2 obj=522540.0 imp=.11E-2 bdf=.98E-1] Scoring after 4042998ms
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=5 lmb=.14E2 obj=522540.0 imp=.11E-2 bdf=.98E-1] Training metrics computed in 20907ms
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=5 lmb=.14E2 obj=522540.0 imp=.11E-2 bdf=.98E-1] Model Metrics Type: RegressionGLM
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 4.94325006E11
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 703082.5
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 522180.72
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 130517.93
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.93738
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.004234E7
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.2447722E12
10-26 05:01:30.107 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2447763E12
10-26 05:01:30.118 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=5 lmb=.14E2 obj=522540.0 imp=.11E-2 bdf=.98E-1] computed in 3888934+56076+118907=4063917ms, step = 0.10890000000000001, l1solver iter = 10000, gerr = 44821.17005179174
10-26 06:07:41.261 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 06:07:41.261 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 56532.11099690192 >  eps = 1.0E-4
10-26 06:08:47.588 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=6 lmb=.14E2 obj=521080.0 imp=.28E-2 bdf=.28E0] Scoring after 4037470ms
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=6 lmb=.14E2 obj=521080.0 imp=.28E-2 bdf=.28E0] Training metrics computed in 21232ms
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=6 lmb=.14E2 obj=521080.0 imp=.28E-2 bdf=.28E0] Model Metrics Type: RegressionGLM
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 5.12239665E11
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 715709.2
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 520660.66
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 130576.516
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.925282
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.0042325E7
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.2295049E12
10-26 06:09:08.822 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2295091E12
10-26 06:09:08.827 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=6 lmb=.14E2 obj=521080.0 imp=.28E-2 bdf=.28E0] computed in 3911274+59869+87566=4058709ms, step = 0.33, l1solver iter = 10000, gerr = 56532.11099690192
10-26 07:15:06.827 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 07:15:06.827 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 60485.20087029953 >  eps = 1.0E-4
10-26 07:16:10.428 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=7 lmb=.14E2 obj=520960.0 imp=.22E-3 bdf=.19E0] Scoring after 4021601ms
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=7 lmb=.14E2 obj=520960.0 imp=.22E-3 bdf=.19E0] Training metrics computed in 22090ms
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=7 lmb=.14E2 obj=520960.0 imp=.22E-3 bdf=.19E0] Model Metrics Type: RegressionGLM
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 5.692472E11
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 754484.75
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 520506.1
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 131249.19
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.91393
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.004232E7
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.2279525E12
10-26 07:16:32.518 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2279567E12
10-26 07:16:32.523 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=7 lmb=.14E2 obj=520960.0 imp=.22E-3 bdf=.19E0] computed in 3892837+65163+85696=4043696ms, step = 0.33, l1solver iter = 10000, gerr = 60485.20087029953
10-26 08:22:34.254 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 08:22:34.254 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 87496.09341881957 >  eps = 1.0E-4
10-26 08:24:11.293 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=8 lmb=.14E2 obj=520840.0 imp=.24E-3 bdf=.42E-1] Scoring after 4058770ms
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=8 lmb=.14E2 obj=520840.0 imp=.24E-3 bdf=.42E-1] Training metrics computed in 21569ms
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=8 lmb=.14E2 obj=520840.0 imp=.24E-3 bdf=.42E-1] Model Metrics Type: RegressionGLM
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 5.503653E11
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 741866.06
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 520372.72
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 131350.28
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.910981
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.0042317E7
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.2266129E12
10-26 08:24:32.863 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2266171E12
10-26 08:24:32.868 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=8 lmb=.14E2 obj=520840.0 imp=.24E-3 bdf=.42E-1] computed in 3904152+57579+118614=4080345ms, step = 0.10890000000000001, l1solver iter = 10000, gerr = 87496.09341881957
10-26 09:30:37.337 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 09:30:37.337 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 46878.36427637116 >  eps = 1.0E-4
10-26 09:31:42.658 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=9 lmb=.14E2 obj=519810.0 imp=.2E-2 bdf=.11E0] Scoring after 4029790ms
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=9 lmb=.14E2 obj=519810.0 imp=.2E-2 bdf=.11E0] Training metrics computed in 21189ms
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=9 lmb=.14E2 obj=519810.0 imp=.2E-2 bdf=.11E0] Model Metrics Type: RegressionGLM
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 4.78805721E11
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 691957.9
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 519314.25
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 131280.0
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.902149
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.0042314E7
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.2159814E12
10-26 09:32:03.847 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2159856E12
10-26 09:32:03.853 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=9 lmb=.14E2 obj=519810.0 imp=.2E-2 bdf=.11E0] computed in 3909384+55085+86516=4050985ms, step = 0.33, l1solver iter = 10000, gerr = 46878.36427637116
10-26 10:38:06.545 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 10:38:06.545 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 153149.48474907267 >  eps = 1.0E-4
10-26 10:39:13.512 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=10 lmb=.14E2 obj=519560.0 imp=.48E-3 bdf=.75E-1] Scoring after 4029658ms
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=10 lmb=.14E2 obj=519560.0 imp=.48E-3 bdf=.75E-1] Training metrics computed in 21204ms
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=10 lmb=.14E2 obj=519560.0 imp=.48E-3 bdf=.75E-1] Model Metrics Type: RegressionGLM
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 5.12623084E11
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 715977.0
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 519042.94
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 131338.34
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.894209
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.0042305E7
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.2132567E12
10-26 10:39:34.716 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2132609E12
10-26 10:39:34.720 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=10 lmb=.14E2 obj=519560.0 imp=.48E-3 bdf=.75E-1] computed in 3906292+56400+88175=4050867ms, step = 0.33, l1solver iter = 10000, gerr = 153149.48474907267
10-26 11:45:29.369 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver stopped after 10000 iterations. (max_iter=10000)
10-26 11:45:29.369 172.31.20.64:3000     206    FJ-1-23   WARN: ADMM solver finished with gerr = 44089.34572744647 >  eps = 1.0E-4
10-26 11:46:35.949 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=11 lmb=.14E2 obj=519210.0 imp=.66E-3 bdf=.83E-1] Scoring after 4021229ms
10-26 11:46:58.380 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=11 lmb=.14E2 obj=519210.0 imp=.66E-3 bdf=.83E-1] Training metrics computed in 22431ms
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=11 lmb=.14E2 obj=519210.0 imp=.66E-3 bdf=.83E-1] Model Metrics Type: RegressionGLM
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  Description: N/A
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  model id: MyModel
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  frame id: RTMP_sid_9e49_4
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  MSE: 4.48644973E11
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  RMSE: 669809.6
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  mean residual deviance: 518686.6
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  mean absolute error: 131270.0
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  root mean squared log error: 10.888633
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  null DOF: 1.0043978E7
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  residual DOF: 1.0042303E7
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  null deviance: 5.3290043E12
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  residual deviance: 5.2096774E12
10-26 11:46:58.381 172.31.20.64:3000     206    FJ-1-23   INFO:  AIC: 5.2096816E12
10-26 11:46:58.385 172.31.20.64:3000     206    FJ-1-23   INFO: GLM[dest=MyModel, iter=11 lmb=.14E2 obj=519210.0 imp=.66E-3 bdf=.83E-1] computed in 3896878+57770+89016=4043664ms, step = 0.33, l1solver iter = 10000, gerr = 44089.34572744647
10-26 11:54:31.824 172.31.20.64:3000     206    #191:3000 ERRR: Got IO error when sending batch UDP bytes: java.io.IOException: Connection reset by peer
10-26 11:54:31.829 172.31.20.64:3000     206    #106:3000 ERRR: Got IO error when sending batch UDP bytes: java.io.IOException: Connection reset by peer
h2o poisson
1个回答
0
投票

您已经说过您的数据在131GB附近,这非常大,因此模型训练和测试将花费大量时间。

下面是在GLM-poisson中训练h2o模型的官方文档,它还包括训练期间出现的所有参数,并说明了如何解释模型输出参数。

http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/glm.html

https://www.h2o.ai/wp-content/uploads/2018/01/GLM-BOOKLET.pdf

© www.soinside.com 2019 - 2024. All rights reserved.