我正在尝试TensorFlow 2.0并测试keras的分布式解决方案，但我遇到了问题：
embedding_column从variable-length input feature转换不适用于分布式Keras MultiWorkerMirroredStrategy。 通过本地培训（非分布式），一切都很好，或者如果我将可变长度输入功能转换为indicator_column，一切都还可以。
这里是重现该问题的代码：
import tensorflow as tf
import os
import json
os.environ['TF_CONFIG'] = json.dumps({
'cluster': {
'worker': ["localhost:12345", "localhost:23456"]
},
'task': {'type': 'worker', 'index': 1} # {'type': 'worker', 'index': 0}
})
# create fake dataset file
def serialize_example(value):
feature = {
'color': tf.train.Feature(bytes_list=tf.train.BytesList(value=value)),
}
example_proto = tf.train.Example(features=tf.train.Features(feature=feature))
return example_proto.SerializeToString()
tfrecord_writer = tf.io.TFRecordWriter('./color.tfrecord')
for each in [['G', 'R'], ['B'], ['B', 'G'], ['R']]:
tfrecord_writer.write(serialize_example(each))
tfrecord_writer.close()
# build feature column
color_column = tf.feature_column.categorical_column_with_vocabulary_list('color', ['R', 'G', 'B'], dtype=tf.string)
color_embeding = tf.feature_column.embedding_column(color_column, 4) # tf.feature_column.indicator_column(color_column)
inputs = {}
inputs['color'] = tf.keras.layers.Input(name='color', shape=(None, ), sparse=True, dtype='string')
# build model
with tf.distribute.experimental.MultiWorkerMirroredStrategy().scope():
dense = tf.keras.layers.DenseFeatures([color_embeding])(inputs)
output = tf.keras.layers.Dense(1, activation='sigmoid')(dense)
model = tf.keras.Model(inputs, output)
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
# build dataset
def parse(example_proto):
feature_description = {
'color': tf.io.VarLenFeature(tf.string)
}
parsed_features = tf.io.parse_single_example(example_proto, feature_description)
return parsed_features, True
dataset = tf.data.TFRecordDataset('./color.tfrecord').map(parse).repeat().batch(1)
model.fit(dataset, epochs=3, steps_per_epoch=1)
错误日志：
2019-10-14 22:35:48.491329: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-14 22:35:48.504154: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fd065e01d50 executing computations on platform Host. Devices:
2019-10-14 22:35:48.504171: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2019-10-14 22:35:48.506239: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:258] Initialize GrpcChannelCache for job worker -> {0 -> localhost:12345, 1 -> localhost:23456}
2019-10-14 22:35:48.506975: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:365] Started server with target: grpc://localhost:23456
INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:XLA_CPU:0']
INFO:tensorflow:Multi-worker CollectiveAllReduceStrategy with cluster_spec = {u'worker': [u'localhost:12345', u'localhost:23456']}, task_type = 'worker', task_id = 1, num_workers = 2, local_devices = ('/job:worker/task:1',), communication = CollectiveCommunication.AUTO
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
INFO:tensorflow:Running Distribute Coordinator with mode = 'independent_worker', cluster_spec = {u'worker': [u'localhost:12345', u'localhost:23456']}, task_type = u'worker', task_id = 1, environment = None, rpc_layer = 'grpc'
WARNING:tensorflow:`eval_fn` is not passed in. The `worker_fn` will be used if an "evaluator" task exists in the cluster.
WARNING:tensorflow:`eval_strategy` is not passed in. No distribution strategy will be used for evaluation.
INFO:tensorflow:Multi-worker CollectiveAllReduceStrategy with cluster_spec = {u'worker': [u'localhost:12345', u'localhost:23456']}, task_type = u'worker', task_id = 1, num_workers = 2, local_devices = (u'/job:worker/task:1',), communication = CollectiveCommunication.AUTO
INFO:tensorflow:Multi-worker CollectiveAllReduceStrategy with cluster_spec = {u'worker': [u'localhost:12345', u'localhost:23456']}, task_type = u'worker', task_id = 1, num_workers = 2, local_devices = (u'/job:worker/task:1',), communication = CollectiveCommunication.AUTO
WARNING:tensorflow:ModelCheckpoint callback is not provided. Workers will need to restart training if any fails.
2019-10-14 22:36:00.226012: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:400] Cannot find shardable dataset, adding a shard node at the end of the dataset instead. This may have performance implications.
Train for 1 steps
Epoch 1/3
INFO:tensorflow:Collective batch_all_reduce: 2 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce for IndexedSlices: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 2 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce for IndexedSlices: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2
2019-10-14 22:36:01.352450: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingReduce with Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.352478: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.352638: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingGather with Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.352657: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.353012: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingGather with Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.353029: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.353213: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingReduce with Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.353225: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.353325: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:234 : Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.353338: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingReduce with Cancelled: [_Derived_]Cancelled
Additional GRPC error information:
{"created":"@1571063761.352818000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Cancelled","grpc_status":1}
2019-10-14 22:36:01.353352: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Cancelled: [_Derived_]Cancelled
Additional GRPC error information:
{"created":"@1571063761.352818000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Cancelled","grpc_status":1}
2019-10-14 22:36:01.353370: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:125 : Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.353359: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
[[metrics/accuracy/div_no_nan/allreduce_1/CollectiveReduce]]
2019-10-14 22:36:01.353469: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingReduce with Cancelled: [_Derived_]Cancelled
Additional GRPC error information:
{"created":"@1571063761.352917000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Cancelled","grpc_status":1}
2019-10-14 22:36:01.353451: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:234 : Cancelled: [_Derived_]Cancelled
Additional GRPC error information:
{"created":"@1571063761.352818000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Cancelled","grpc_status":1}
2019-10-14 22:36:01.353491: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Cancelled: [_Derived_]Cancelled
Additional GRPC error information:
{"created":"@1571063761.352917000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Cancelled","grpc_status":1}
2019-10-14 22:36:01.353512: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:234 : Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-10-14 22:36:01.353630: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:234 : Cancelled: [_Derived_]Cancelled
Additional GRPC error information:
{"created":"@1571063761.352917000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Cancelled","grpc_status":1}
2019-10-14 22:36:01.353698: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:125 : Internal: Inconsistent output shapes, got [4], but expected is [2].
[[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
1/1 [==============================] - 1s 1s/step
---------------------------------------------------------------------------
InternalError Traceback (most recent call last)
<ipython-input-1-b83aa9a3e019> in <module>()
43
44 dataset = tf.data.TFRecordDataset('./color.tfrecord').map(parse).repeat().batch(1)
---> 45 model.fit(dataset, epochs=3, steps_per_epoch=1)
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training.pyc in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
726 max_queue_size=max_queue_size,
727 workers=workers,
--> 728 use_multiprocessing=use_multiprocessing)
729
730 def evaluate(self,
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_distributed.pyc in fit(self, *args, **kwargs)
787 def fit(self, *args, **kwargs):
788 return train_with_multi_worker(self._single_worker_loop.fit)(
--> 789 *args, **kwargs)
790
791 def evaluate(self, *args, **kwargs):
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_distributed.pyc in wrapper(model, **kwargs)
774 _worker_fn,
775 model._distribution_strategy,
--> 776 mode=dc.CoordinatorMode.INDEPENDENT_WORKER)
777
778 return wrapper
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/distribute/distribute_coordinator.pyc in run_distribute_coordinator(worker_fn, strategy, eval_fn, eval_strategy, mode, cluster_spec, task_type, task_id, session_config, rpc_layer)
851 # All jobs run `worker_fn` if between-graph.
852 return _run_single_worker(worker_fn, strategy, cluster_spec, task_type,
--> 853 task_id, session_config, rpc_layer)
854 else:
855 # Only one node runs `worker_fn` if in-graph.
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/distribute/distribute_coordinator.pyc in _run_single_worker(worker_fn, strategy, cluster_spec, task_type, task_id, session_config, rpc_layer, worker_barrier, coord)
358 return worker_fn(strategy)
359 else:
--> 360 return worker_fn(strategy)
361
362
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_distributed.pyc in _worker_fn(_)
769 filtered_callbacks = dist_utils.filter_distributed_callbacks(callbacks)
770 kwargs['callbacks'] = filtered_callbacks
--> 771 return method(model, **kwargs)
772
773 return dc.run_distribute_coordinator(
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_v2.pyc in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs)
322 mode=ModeKeys.TRAIN,
323 training_context=training_context,
--> 324 total_epochs=epochs)
325 cbks.make_logs(model, epoch_logs, training_result, ModeKeys.TRAIN)
326
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_v2.pyc in run_one_epoch(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)
121 step=step, mode=mode, size=current_batch_size) as batch_logs:
122 try:
--> 123 batch_outs = execution_function(iterator)
124 except (StopIteration, errors.OutOfRangeError):
125 # TODO(kaftan): File bug about tf function and errors.OutOfRangeError?
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.pyc in execution_function(input_fn)
84 # `numpy` translates Tensors to values in Eager mode.
85 return nest.map_structure(_non_none_constant_value,
---> 86 distributed_function(input_fn))
87
88 return execution_function
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/eager/def_function.pyc in __call__(self, *args, **kwds)
455
456 tracing_count = self._get_tracing_count()
--> 457 result = self._call(*args, **kwds)
458 if tracing_count == self._get_tracing_count():
459 self._call_counter.called_without_tracing()
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/eager/def_function.pyc in _call(self, *args, **kwds)
518 # Lifting succeeded, so variables are initialized and we can run the
519 # stateless function.
--> 520 return self._stateless_fn(*args, **kwds)
521 else:
522 canon_args, canon_kwds = \
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/eager/function.pyc in __call__(self, *args, **kwargs)
1821 """Calls a graph function specialized to the inputs."""
1822 graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
-> 1823 return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
1824
1825 @property
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/eager/function.pyc in _filtered_call(self, args, kwargs)
1139 if isinstance(t, (ops.Tensor,
1140 resource_variable_ops.BaseResourceVariable))),
-> 1141 self.captured_inputs)
1142
1143 def _call_flat(self, args, captured_inputs, cancellation_manager=None):
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/eager/function.pyc in _call_flat(self, args, captured_inputs, cancellation_manager)
1222 if executing_eagerly:
1223 flat_outputs = forward_function.call(
-> 1224 ctx, args, cancellation_manager=cancellation_manager)
1225 else:
1226 gradient_name = self._delayed_rewrite_functions.register()
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/eager/function.pyc in call(self, ctx, args, cancellation_manager)
509 inputs=args,
510 attrs=("executor_type", executor_type, "config_proto", config),
--> 511 ctx=ctx)
512 else:
513 outputs = execute.execute_with_cancellation(
/Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/eager/execute.pyc in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
65 else:
66 message = e.message
---> 67 six.raise_from(core._status_to_exception(e.code, message), None)
68 except TypeError as e:
69 keras_symbolic_tensors = [
/Users/felix/Envs/tf2/lib/python2.7/site-packages/six.pyc in raise_from(value, from_value)
735 else:
736 def raise_from(value, from_value):
--> 737 raise value
738
739
InternalError: Inconsistent output shapes, got [4], but expected is [2].
[[node Adam/allreduce_1/CollectiveGather_1 (defined at /Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
Additional GRPC error information:
{"created":"@1571063761.352390000","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Inconsistent output shapes, got [4], but expected is [2].\n\t [[node Adam/allreduce_1/CollectiveGather_1 (defined at /Users/felix/Envs/tf2/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]","grpc_status":13}
[[metrics/accuracy/div_no_nan/allreduce_1/CollectiveReduce]] [Op:__inference_distributed_function_1149]
Function call stack:
distributed_function
这个问题困扰了我一个星期，任何帮助将不胜感激
尝试删除steps_per_epochs参数。由于某种原因，Keras（或TF）在训练tfrecords时不知道如何确定数据集的大小。
编辑：实际上我现在刚刚检查了新近更新的TF 2.0文档，看来现在数据集创建需要在strategy.scope()之内。检查https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras#train_the_model_with_multiworker