我正在尝试训练一个模型。我有近 150 个类，我正在使用 ImageDataGenerator 来扩充我的数据集。我还使用模型检查点和 csvlogger 来保存权重。当我开始训练时，它在第一个时期的某个时刻给了我一个错误。如果有帮助的话，我使用的图像是灰度图像。
这是我的代码：
batch_size = 2000
epochs = 10
# Augments dataset 10x
train_batches = ImageDataGenerator(preprocessing_function=preprocess_func, horizontal_flip=True, width_shift_range=0.1, height_shift_range=0.1, shear_range=0.2, zoom_range=0.2, fill_mode='nearest') \
.flow_from_directory(directory=train_path, target_size=image_size, classes=dataset_classes, batch_size=5, color_mode='grayscale')
valid_batches = ImageDataGenerator(preprocessing_function=preprocess_func, horizontal_flip=True, width_shift_range=0.15, height_shift_range=0.1, shear_range=0.2, zoom_range=0.2, fill_mode='nearest') \
.flow_from_directory(directory=valid_path, target_size=image_size, classes=dataset_classes, batch_size=5, color_mode='grayscale')
test_batches = ImageDataGenerator(preprocessing_function=preprocess_func, horizontal_flip=True, width_shift_range=0.15, height_shift_range=0.1, shear_range=0.2, zoom_range=0.2, fill_mode='nearest') \
.flow_from_directory(directory=test_path, target_size=image_size, classes=dataset_classes, batch_size=5, color_mode='grayscale')
这是我的回调：
from keras.callbacks import ModelCheckpoint, CSVLogger
checkpoint_path = "/content/drive/MyDrive/Colab Notebooks/Datasets/Experiment/weights_improvements-epoch:{epoch:02d}-val_accuracy:{val_accuracy:.2f}.hdf5"
checkpoint_dir = os.path.dirname(checkpoint_path)
# Create a callback that saves the model's weights
cp_callback = ModelCheckpoint(checkpoint_path,
verbose=1,
monitor='val_accuracy',
mode='max',
save_best_only=True,
save_weights_only=True)
log_folder = '/content/drive/MyDrive/Colab Notebooks/Datasets/Experiment'
log_path = os.path.join(log_folder, 'FSLR_logs.csv')
log_csv = CSVLogger(log_path, separator=',', append=False)
callback_list = [cp_callback, log_csv]
拟合模型：
# Compile the layers into one model and create a connection
model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.Adam(), metrics=['accuracy'])
# Train the model with the new callback
history = model.fit(x=train_batches,
validation_data=valid_batches,
batch_size=batch_size,
epochs=epochs,
callbacks=callback_list)
我收到的错误是这样的：
Epoch 1/10 3428/4128 [=======================>......] - ETA: 26:10 - loss: 4.8299 - accuracy: 0.0078
--------------------------------------------------------------------------- UnknownError Traceback (most recent call last) <ipython-input-21-86d6207a54c5> in <module>()
4 batch_size=batch_size,
5 epochs=epochs,
----> 6 callbacks=callback_list)
6 frames /usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
58 ctx.ensure_initialized()
59 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60 inputs, attrs, num_outputs)
61 except core._NotOkStatusException as e:
62 if name is not None:
UnknownError: OSError: image file is truncated (30 bytes not processed) Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/script_ops.py", line 249, in __call__
ret = func(*args)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/autograph/impl/api.py", line 645, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 892, in generator_py_func
values = next(generator_state.get_iterator(iterator_id))
File "/usr/local/lib/python3.7/dist-packages/keras/engine/data_adapter.py", line 822, in wrapped_generator
for data in generator_fn():
File "/usr/local/lib/python3.7/dist-packages/keras/engine/data_adapter.py", line 948, in generator_fn
yield x[i]
File "/usr/local/lib/python3.7/dist-packages/keras_preprocessing/image/iterator.py", line 65, in __getitem__
return self._get_batches_of_transformed_samples(index_array)
File "/usr/local/lib/python3.7/dist-packages/keras_preprocessing/image/iterator.py", line 230, in _get_batches_of_transformed_samples
interpolation=self.interpolation)
File "/usr/local/lib/python3.7/dist-packages/keras_preprocessing/image/utils.py", line 138, in load_img
img = img.resize(width_height_tuple, resample)
File "/usr/local/lib/python3.7/dist-packages/PIL/Image.py", line 1886, in resize
self.load()
File "/usr/local/lib/python3.7/dist-packages/PIL/ImageFile.py", line 247, in load
"(%d bytes not processed)" % len(b)
OSError: image file is truncated (30 bytes not processed)
[[{{node PyFunc}}]] [[IteratorGetNext]] [Op:__inference_train_function_1029]
Function call stack: train_function
我尝试在训练两个类时使用相同的代码，效果很好。我不知道为什么当我在所有 140 多个课程中使用它时它不起作用。
如何解决这个问题？
编辑： 我已经运行此代码来验证所有图像。它没有发现任何损坏的文件。
import os
from os import listdir
from PIL import Image
categ = ['Train', 'Valid', 'Test']
dataset = '/content/drive/MyDrive/Colab Notebooks/Datasets/FSLR_Application_Dataset'
for cat in categ:
img_path = os.path.join(dataset, cat)
for foldername in listdir(img_path):
sign_path = os.path.join(img_path, foldername)
print(sign_path)
for sign in listdir(sign_path):
if sign.endswith('.jpg'):
try:
img = Image.open(os.path.join(sign_path, sign)) # open the image file
img.verify() # verify that it is, in fact an image
except (IOError, SyntaxError) as e:
print('Bad file:', sign) # print out the names of corrupt files
我在查找有缺陷的图像文件时也遇到过类似的问题。 ImageDataGenerator 使用 PIL。生成器没有检测到图像文件中的错误，如果有的话，它会打印一条警告消息。因此我建议您尝试使用 PIL 以外的其他工具来检测有缺陷的图像文件。尝试使用 cv2 我发现它有时会检测到 PIL 无法检测到的错误。具体来说
import cv2
your code but replace
img = Image.open(os.path.join(sign_path, sign)) # open the image file
img.verify() # verify that it is, in fact an image
except (IOError, SyntaxError) as e:
print('Bad file:', sign) # print out the names of corrupt files
with
bad_file_list=[]
bad_count=0
try:
img.cv2.imread(os.path.join(sign_path, sign)
shape=img.shape # this will throw an error if the img is not read correctly
except:
bad_file_list.append(os.path.join(sign_path, sign))
bad_count +=1
然后在循环外打印是否发现坏文件
我之前遇到过同样的问题，这对我有用，在拟合模型之前添加这一行：
from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True
history = model.fit(...) #ur fitting code