Keras fit vs. fit_generator额外的smaples

问题描述 投票:0回答:1

我将训练数据和验证数据堆叠在两个张量中。首先,我使用keras.model.fit()函数运行了一个NN。就我的目的而言,我希望移至keras.model.fit()。我建立了一个生成器,我注意到样本数量不是批次大小的乘积。

我的实现克服了这个问题:

keras.model.fit_generator()

在将每个时期的索引随机化之后,我简单地分批跳过此步骤,并合并正确的数据和标签。

我正在观察模型性能的下降。使用keras.model.fit_generator()进行训练时,我的训练精度为0.99,验证准确性为0.93,而使用indices = np.arange(len(dataset))# generate indices of len(dataset) num_of_steps = int(np.ceil(len(dataset)/batch_size)) #number of steps per epoch extra = num_of_steps *batch_size-len(dataset)#find the size of extra samples needed to complete the next multiplication of batch_size additional = np.random.randint(len(dataset),size = extra )#complete with random samples indices = np.append(indices ,additional ) 进行训练时,我的精度分别为0.95和0.9。请注意,这是一致的,不是单个实验。我认为可能是由于fit()以不同方式处理了所需的额外样本。我的实施合理吗? fit_generator()如何处理大小不同于fit()乘法的数据集?

共享完整的生成器代码:

fit()
keras
1个回答
0
投票

看来您可以大大简化生成器!步数等可以在循环外部设置,因为它们并没有真正改变。而且,batch_size似乎并没有遍历整个数据集。最后,如果您的数据适合存储在内存中,则可能根本不需要生成器,但是您可以自行决定。

    def generator(self,batch_size,train):
        """
        Generates batches of samples
        :return: 
        """
        while 1:
            nb_of_steps=0
            if(train):    
                nb_of_steps = self._num_of_steps_train
                indices = np.arange(len(self._x_train))
                additional = np.random.randint(len(self._x_train), size=self._num_of_steps_train*batch_size-len(self._x_train))
            else:
                nb_of_steps = self._num_of_steps_test
                indices = np.arange(len(self._x_test))
                additional = np.random.randint(len(self._x_test), size=self._num_of_steps_test*batch_size-len(self._x_test))

            indices = np.append(indices,additional)
            np.random.shuffle(indices)
#            print(indices.shape)
#            print(nb_of_steps)

            for i in range(nb_of_steps):
                batch_indices=indices[i:i+batch_size]
                if(train):
                    feat = self._x_train[batch_indices]
                    label = self._y_train[batch_indices]
                else:
                    feat = self._x_test[batch_indices]
                    label = self._y_test[batch_indices]
                feat = np.expand_dims(feat,axis=1)
#                print(feat.shape)
#                print(label.shape)

                yield feat, label      

对于更强大的生成器,请考虑使用batch_indices def generator(self, batch_size, train): nb_of_steps = 0 if (train): nb_of_steps = self._num_of_steps_train indices = np.arange(len(self._x_train)) #len of entire dataset else: nb_of_steps = self._num_of_steps_test indices = np.arange(len(self._x_test)) while 1: np.random.shuffle(indices) for i in range(nb_of_steps): start_idx = i*batch_size end_idx = min(i*batch_size+batch_size, len(indices)) batch_indices=indices[start_idx : end_idx] if(train): feat = self._x_train[batch_indices] label = self._y_train[batch_indices] else: feat = self._x_test[batch_indices] label = self._y_test[batch_indices] feat = np.expand_dims(feat,axis=1) yield feat, label 为您的集合创建一个类。它将添加一些额外的代码行,但肯定可以与keras一起使用。

© www.soinside.com 2019 - 2024. All rights reserved.