所以我有一个序列到序列的问题,输入是许多不同长度的多变量序列,输出是一个与输入对应的长度相同的二进制向量序列。我把长度相同的序列归在一起,放在一个单独的文件夹里,然后像这样调用拟合函数。
for e in range(epochs):
print('Epoch', e+1)
for i in range(3,19):
train_x_batch,train_y_batch,batch_size= get_data(i)
history=model.fit_(train_x_batch,train_y_batch,
batch_size=batch_size,
validation_split=0.15,
callbacks=[tensorboard_cb])
def get_data(i):
train_x = np.load(os.path.join(cwd, "lab_values","batches",f"f_{i}","train_x.npy"), allow_pickle=True)
train_y = np.load(os.path.join(cwd, "lab_values","batches",f"f_{i}","train_y.npy"), allow_pickle=True)
print(f"batch no {i} Train X size= ", train_x.shape)
print(f"batch no {i} Train Y size= ", train_y.shape)
batch_Size=train_x.shape[0]
return train_x,train_y,batch_size
所以问题是有什么更好的方法吗?我听说我可以使用一个生成器来实现这个功能,可惜我无法实现这样的生成器。
你是想对整个数据进行训练 (npy file)
而不是分批训练模型。
我们可以写一个 Generator
并对模型进行训练 Batches
.
我们使用代码从现有的Numpy文件中提取数据批次。
train_x = np.load(os.path.join(cwd, "lab_values","batches",f"f_{i}","train_x.npy"), mmap_mode='r', allow_pickle=True)
和
x_batch = train_x[start:end].copy()
.
完整的代码 Generator
和代码的 Training
如下图所示。
import numpy as np
for e in range(epochs):
print('Epoch', e+1)
for i in range(3,19):
#train_x_batch,train_y_batch = get_data(i)
batch_size = 32
history=model.fit_(get_data(i),
batch_size=batch_size,
validation_split=0.15,
callbacks=[tensorboard_cb],epochs = 20
steps_per_epoch = 500, val_steps = 10)
def get_data(i):
train_x = np.load(os.path.join(cwd, "lab_values","batches",f"f_{i}","train_x.npy"),
mmap_mode='r', allow_pickle=True)
train_y = np.load(os.path.join(cwd, "lab_values","batches",f"f_{i}","train_y.npy"),
mmap_mode='r', allow_pickle=True)
print(f"batch no {i} Train X size= ", train_x.shape)
print(f"batch no {i} Train Y size= ", train_y.shape)
Number_Of_Rows = train_x.shape[0]
batch_size = 32
start = np.random.choice(Number_of_Rows - batch_size)
end = start + batch_size
x_batch = train_x[start:end].copy()
y_batch = train_y[start:end].copy()
yield x_batch,y_batch