我将训练数据和验证数据堆叠在两个张量中。首先,我使用keras.model.fit()
函数运行了一个NN。就我的目的而言,我希望移至keras.model.fit()
。我建立了一个生成器,我注意到样本数量不是批次大小的乘积。
我的实现克服了这个问题:
keras.model.fit_generator()
在将每个时期的索引随机化之后,我简单地分批跳过此步骤,并合并正确的数据和标签。
我正在观察模型性能的下降。使用keras.model.fit_generator()
进行训练时,我的训练精度为0.99,验证准确性为0.93,而使用indices = np.arange(len(dataset))# generate indices of len(dataset)
num_of_steps = int(np.ceil(len(dataset)/batch_size)) #number of steps per epoch
extra = num_of_steps *batch_size-len(dataset)#find the size of extra samples needed to complete the next multiplication of batch_size
additional = np.random.randint(len(dataset),size = extra )#complete with random samples
indices = np.append(indices ,additional )
进行训练时,我的精度分别为0.95和0.9。请注意,这是一致的,不是单个实验。我认为可能是由于fit()
以不同方式处理了所需的额外样本。我的实施合理吗? fit_generator()
如何处理大小不同于fit()
乘法的数据集?
共享完整的生成器代码:
fit()
看来您可以大大简化生成器!步数等可以在循环外部设置,因为它们并没有真正改变。而且,batch_size
似乎并没有遍历整个数据集。最后,如果您的数据适合存储在内存中,则可能根本不需要生成器,但是您可以自行决定。
def generator(self,batch_size,train):
"""
Generates batches of samples
:return:
"""
while 1:
nb_of_steps=0
if(train):
nb_of_steps = self._num_of_steps_train
indices = np.arange(len(self._x_train))
additional = np.random.randint(len(self._x_train), size=self._num_of_steps_train*batch_size-len(self._x_train))
else:
nb_of_steps = self._num_of_steps_test
indices = np.arange(len(self._x_test))
additional = np.random.randint(len(self._x_test), size=self._num_of_steps_test*batch_size-len(self._x_test))
indices = np.append(indices,additional)
np.random.shuffle(indices)
# print(indices.shape)
# print(nb_of_steps)
for i in range(nb_of_steps):
batch_indices=indices[i:i+batch_size]
if(train):
feat = self._x_train[batch_indices]
label = self._y_train[batch_indices]
else:
feat = self._x_test[batch_indices]
label = self._y_test[batch_indices]
feat = np.expand_dims(feat,axis=1)
# print(feat.shape)
# print(label.shape)
yield feat, label
对于更强大的生成器,请考虑使用batch_indices
def generator(self, batch_size, train):
nb_of_steps = 0
if (train):
nb_of_steps = self._num_of_steps_train
indices = np.arange(len(self._x_train)) #len of entire dataset
else:
nb_of_steps = self._num_of_steps_test
indices = np.arange(len(self._x_test))
while 1:
np.random.shuffle(indices)
for i in range(nb_of_steps):
start_idx = i*batch_size
end_idx = min(i*batch_size+batch_size, len(indices))
batch_indices=indices[start_idx : end_idx]
if(train):
feat = self._x_train[batch_indices]
label = self._y_train[batch_indices]
else:
feat = self._x_test[batch_indices]
label = self._y_test[batch_indices]
feat = np.expand_dims(feat,axis=1)
yield feat, label
为您的集合创建一个类。它将添加一些额外的代码行,但肯定可以与keras一起使用。