Tensorflow 2.1 Dataset From Generator - ValueError: 检查输入时出错：预期Geo的形状为(504,)，但得到的是形状为(1,)的数组。

问题描述 投票：0回答：1

我在让我的模型编译时遇到了麻烦，我似乎经常遇到这个问题，而且错误输出也不是特别有用。

期望Geo有形状(504,)，但得到的是形状为(1,)的数组。

我明白形状在某个地方不匹配，但这个错误并没有将我指向代码中的任何特定点。是来自Input()层、第一个Dense层还是来自Generator本身？

我的模型是一个多输入单输出的模型，它在 "Geo "输入中接收一个504个浮动输入的numpy数组，在 "SDF "输入中接收一个64,64,64,1的SDF。这些都会被连接起来，输出一个12个浮点数的结果。

我的逻辑如下。

# The Data Generator
# Input shape is 42, 12, 1
# SDF shape is 64, 64, 64

def data_generator(train_path, test_path):
    """
    Generator for loading geo data.
    """

    imgListTestSDF = test_path.parent
    imgListTestSDF = imgListTestSDF / "test_sdf"    

    imgListTrain = sorted(train_path.iterdir() )
    imgListTest = sorted(test_path.iterdir() )
    imgListTestSDF = sorted(imgListTestSDF.iterdir() )

    for dist, norm, norm_sdf in zip(imgListTrain, imgListTest, imgListTestSDF):
        norm = np.load(norm)['array']
        norm_sdf = np.load(norm_sdf)['array']
        dist = np.load(dist)['array']

        outshape = norm.shape[0] * 3 * 12
        print ("Out shape: %s" % (outshape,)) # Out shape: 504
        #norm = np.reshape(norm, (504))
        norm = norm.flatten()
        print ("Norm reshaped: %s" % (norm.shape,)) # Norm reshaped: (504,)

        norm_sdf = np.reshape(norm_sdf, IN_SDF_SHAPE)

        # Cast to float32, loaded as float64
        norm = norm.astype(np.float32)
        dist = dist.astype(np.float32)

        norm_combined = {}
        norm_combined['Geo'] = norm
        norm_combined['SDF'] = norm_sdf

        yield norm_combined, dist

NN逻辑

EPOCHS = 100
BATCH_SIZE = 10

IN_GEO_SHAPE = [504]
inGeoTensorShape = tf.TensorShape(IN_GEO_SHAPE)

IN_SDF_SHAPE = [64, 64, 64, 1]
inSDFTensorShape = tf.TensorShape(IN_SDF_SHAPE)

outTypes = ({'Geo': tf.dtypes.float32 , 'SDF' : tf.dtypes.float32}, tf.dtypes.float32)
outTensorShape = ({'Geo': inGeoTensorShape , 'SDF' : inSDFTensorShape }, tf.TensorShape([12,]))

trainGen = partial(data_generator, train, test) 
dataset = tf.data.Dataset.from_generator(trainGen, outTypes, output_shapes=outTensorShape )


# Geo Dense layers.
inGeo = tf.keras.Input(shape=inGeoTensorShape, batch_size=BATCH_SIZE, name="Geo" )
print ("In Geo layer shape: %s" % (inGeo.shape,)) # In Geo layer shape: (10, 504)

dense1 = tf.keras.layers.Dense(512)(inGeo)
dense2 = tf.keras.layers.Dense(512)(dense1)
dense3 = tf.keras.layers.Dense(512)(dense2)
geoOut = tf.keras.layers.Flatten()(dense3)

print ("Geo Out Shape: %s" % (geoOut.shape, )) # Geo Out Shape: (10, 512)

# SDF CNN Net
initializer = tf.random_normal_initializer(0.0, 1.0)

inSDF = tf.keras.Input(shape=inSDFTensorShape, batch_size=BATCH_SIZE, name="SDF")
print ("In SDF Shape: %s" % (inSDF.shape,) )  # In SDF Shape: (10, 64, 64, 64, 1)
conv1 = tf.keras.layers.Conv3D(32, 4, padding='same', kernel_initializer=initializer, input_shape=inSDFTensorShape)(inSDF) # 32
max1 = tf.keras.layers.MaxPool3D((2, 2, 2))(conv1)
... Many CNN layers ...
conv6 = tf.keras.layers.Conv3D(512, 4, padding='same', kernel_initializer=initializer)(max5) # 1
max6 = tf.keras.layers.MaxPool3D((2, 2, 2))(conv6)
sdfOut = tf.keras.layers.Flatten()(max6)

print ("SDF Out Shape: %s" % (sdfOut.shape, )) # SDF Out Shape: (10, 512)

# Concatenation and Output

concat = tf.keras.layers.concatenate([geoOut, sdfOut])
decode1 = tf.keras.layers.Dense(512)(concat)
decode2 = tf.keras.layers.Dense(128)(decode1)
output = tf.keras.layers.Dense(12)(decode2)

print ("Creating Model")
model = tf.keras.Model(inputs=[inGeo, inSDF], outputs=output ) 
model.summary()

optimiser = tf.keras.optimizers.Adam()
#loss = tf.keras.losses.MSE()

print("Compiling Model")
model.compile(optimizer=optimiser, loss='mse', metrics=['accuracy'])

print("Training Model")
#model.fit(data_gener#ator(train, test) , epochs=EPOCHS, steps_per_epoch=30 )
model.fit(dataset, epochs=EPOCHS, steps_per_epoch=10 )

这是模型摘要

Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
SDF (InputLayer)                [(10, 64, 64, 64, 1) 0
__________________________________________________________________________________________________
conv3d (Conv3D)                 (10, 64, 64, 64, 32) 2080        SDF[0][0]
__________________________________________________________________________________________________
max_pooling3d (MaxPooling3D)    (10, 32, 32, 32, 32) 0           conv3d[0][0]
__________________________________________________________________________________________________
conv3d_1 (Conv3D)               (10, 32, 32, 32, 128 262272      max_pooling3d[0][0]
__________________________________________________________________________________________________
max_pooling3d_1 (MaxPooling3D)  (10, 16, 16, 16, 128 0           conv3d_1[0][0]
__________________________________________________________________________________________________
conv3d_2 (Conv3D)               (10, 16, 16, 16, 256 2097408     max_pooling3d_1[0][0]
__________________________________________________________________________________________________
max_pooling3d_2 (MaxPooling3D)  (10, 8, 8, 8, 256)   0           conv3d_2[0][0]
__________________________________________________________________________________________________
conv3d_3 (Conv3D)               (10, 8, 8, 8, 512)   8389120     max_pooling3d_2[0][0]
__________________________________________________________________________________________________
max_pooling3d_3 (MaxPooling3D)  (10, 4, 4, 4, 512)   0           conv3d_3[0][0]
__________________________________________________________________________________________________
Geo (InputLayer)                [(10, 504)]          0
__________________________________________________________________________________________________
conv3d_4 (Conv3D)               (10, 4, 4, 4, 512)   16777728    max_pooling3d_3[0][0]
__________________________________________________________________________________________________
dense (Dense)                   (10, 512)            258560      Geo[0][0]
__________________________________________________________________________________________________
max_pooling3d_4 (MaxPooling3D)  (10, 2, 2, 2, 512)   0           conv3d_4[0][0]
__________________________________________________________________________________________________
dense_1 (Dense)                 (10, 512)            262656      dense[0][0]
__________________________________________________________________________________________________
conv3d_5 (Conv3D)               (10, 2, 2, 2, 512)   16777728    max_pooling3d_4[0][0]
__________________________________________________________________________________________________
dense_2 (Dense)                 (10, 512)            262656      dense_1[0][0]
__________________________________________________________________________________________________
max_pooling3d_5 (MaxPooling3D)  (10, 1, 1, 1, 512)   0           conv3d_5[0][0]
__________________________________________________________________________________________________
flatten (Flatten)               (10, 512)            0           dense_2[0][0]
__________________________________________________________________________________________________
flatten_1 (Flatten)             (10, 512)            0           max_pooling3d_5[0][0]
__________________________________________________________________________________________________
concatenate (Concatenate)       (10, 1024)           0           flatten[0][0]
                                                                 flatten_1[0][0]
__________________________________________________________________________________________________
dense_3 (Dense)                 (10, 512)            524800      concatenate[0][0]
__________________________________________________________________________________________________
dense_4 (Dense)                 (10, 128)            65664       dense_3[0][0]
__________________________________________________________________________________________________
dense_5 (Dense)                 (10, 12)             1548        dense_4[0][0]
==================================================================================================
Total params: 45,682,220
Trainable params: 45,682,220
Non-trainable params: 0
__________________________________________________________________________________________________

这里是完整的错误（不包括完整的堆栈跟踪）。

ValueError: in converted code:

    C:\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py:677 map_fn
        batch_size=None)
    C:\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training.py:2410 _standardize_tensors
        exception_prefix='input')
    C:\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_utils.py:582 standardize_input_data
        str(data_shape))

    ValueError: Error when checking input: expected Geo to have shape (504,) but got array with shape (1,)

最后，如果有人能告诉我哪里出错了，我将感激不尽。如果他们也能帮助我理解为什么我得到这个错误，我将永远感激。对NN和TF中形状的理解让我至今困惑不已。

谢谢!

python machine-learning keras tensorflow2.0 shapes
1个回答
0
投票

该 Geo layer 正在期待一个大小为 (504,) 但你却给它输入了一个大小为 (1,)

也许当你在做 norm = norm.flatten() 在 data_generator 它正在转化为一个大小为1的数组，里面有504个元素。所以也许可以尝试将其重塑为 (504,)

0
投票

所以看来我使用Dataset的方法不正确。

在创建了我的数据集之后，我需要对它调用batch(BATCH_SIZE)，并提供批次大小作为参数。

因为TF的数据集并没有从输入层推断出批次大小，而是要求数据集正确地产生批次。

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.