GCMLE上的TPUEstimator“内部错误：作业”主“未在群集中定义”

Question

我正在尝试按照此guide上的说明使用GC版本1.8的GCMLE。

对于TF 1.8，他们说：

tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(
FLAGS.tpu,
zone=FLAGS.tpu_zone,
project=FLAGS.gcp_project)

config = tpu_config.RunConfig(
    cluster=tpu_cluster_resolver,
    model_dir=FLAGS.model_dir,
    save_checkpoints_steps=max(600, FLAGS.iterations_per_loop),
    tpu_config=tpu_config.TPUConfig(
        iterations_per_loop=FLAGS.iterations_per_loop,
        num_shards=FLAGS.num_cores))

然后我将这些传递给TPUEstimator / train_and_evaluate（），如下所示：

estimator = tpu_estimator.TPUEstimator(
    use_tpu=True,
    model_fn=model_fn,
    config=run_config,
    params = params,
    train_batch_size = params.train_batch_size,
    eval_batch_size = params.eval_batch_size,
    )

tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)

对于1.7，他们分别说使用具有“主”标志的不同配置。但是，当我在GCMLE上运行上面的1.8指令并将--runtime_version设置为1.8时，我得到以下回溯错误，这表明InternalError: Job "master" was not defined in cluster

Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 585, in <module> run_experiment(params) File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 127, in run_experiment tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate executor.run() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 546, in run getattr(self, task_to_run)() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 601, in run_master self._start_distributed_training(saving_listeners=saving_listeners) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 719, in _start_distributed_training self._start_std_server(config) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 707, in _start_std_server start=False) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/server_lib.py", line 147, in __init__ self._server_def.SerializeToString(), status) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__ c_api.TF_GetCode(self.status.status)) InternalError: Job "master" was not defined in cluster

这很令人困惑，因为文档说不使用master所以我不确定出了什么问题？

Answer 1

使用estimator.train（input_fn = train_input_fn，max_steps = next_checkpoint）和estimator.evaluate（）调用TPUEstimator

Reference

tf.estimator.train_and_evaluate（estimator，train_spec，eval_spec）仅适用于CPU / GPU。