我有一个自动编码器,它将图像作为输入并生成一个新图像作为输出。
输入图像(1x1024x1024x3)在送入网络之前被分割为补丁(1024x32x32x3)。
一旦我有输出,也有一批大小为1024x32x32x3的补丁,我希望能够重建1024x1024x3图像。我以为我只是简单地重塑了这个,但这就是发生的事情。
首先,Tensorflow读取的图像:
我用以下代码修补了图像
patch_size = [1, 32, 32, 1]
patches = tf.extract_image_patches([image],
patch_size, patch_size, [1, 1, 1, 1], 'VALID')
patches = tf.reshape(patches, [1024, 32, 32, 3])
以下是此图片中的几个补丁:
但是,当我将这个补丁数据重新塑造成一个形状为梨状的图像时。
reconstructed = tf.reshape(patches, [1, 1024, 1024, 3])
converted = tf.image.convert_image_dtype(reconstructed, tf.uint8)
encoded = tf.image.encode_png(converted)
在该示例中,在修补和重建之间没有进行任何处理。我做了一个version of the code你可以用来测试这种行为。要使用它,请运行以下命令:
echo "/path/to/test-image.png" > inputs.txt
mkdir images
python3 image_test.py inputs.txt images
代码将为每个输入图像中的1024个补丁中的每个补丁创建一个输入图像,一个补丁图像和一个输出图像,因此如果您只关心保存所有补丁,请注释掉创建输入和输出图像的行。
有人请解释发生了什么:(
使用更新#2 - 您的任务的一个小例子:(TF 1.0)
考虑将尺寸(4,4,1)的图像转换为尺寸(4,2,2,1)的贴片并将其重建为图像。
import tensorflow as tf
image = tf.constant([[[1], [2], [3], [4]],
[[5], [6], [7], [8]],
[[9], [10], [11], [12]],
[[13], [14], [15], [16]]])
patch_size = [1,2,2,1]
patches = tf.extract_image_patches([image],
patch_size, patch_size, [1, 1, 1, 1], 'VALID')
patches = tf.reshape(patches, [4, 2, 2, 1])
reconstructed = tf.reshape(patches, [1, 4, 4, 1])
rec_new = tf.space_to_depth(reconstructed,2)
rec_new = tf.reshape(rec_new,[4,4,1])
sess = tf.Session()
I,P,R_n = sess.run([image,patches,rec_new])
print(I)
print(I.shape)
print(P.shape)
print(R_n)
print(R_n.shape)
输出:
[[[ 1][ 2][ 3][ 4]]
[[ 5][ 6][ 7][ 8]]
[[ 9][10][11][12]]
[[13][14][15][16]]]
(4, 4, 1)
(4, 2, 2, 1)
[[[ 1][ 2][ 3][ 4]]
[[ 5][ 6][ 7][ 8]]
[[ 9][10][11][12]]
[[13][14][15][16]]]
(4,4,1)
仅适用于p = sqrt(h)
import tensorflow as tf
import numpy as np
c = 3
h = 1024
p = 32
image = tf.random_normal([h,h,c])
patch_size = [1,p,p,1]
patches = tf.extract_image_patches([image],
patch_size, patch_size, [1, 1, 1, 1], 'VALID')
patches = tf.reshape(patches, [h, p, p, c])
reconstructed = tf.reshape(patches, [1, h, h, c])
rec_new = tf.space_to_depth(reconstructed,p)
rec_new = tf.reshape(rec_new,[h,h,c])
sess = tf.Session()
I,P,R_n = sess.run([image,patches,rec_new])
print(I.shape)
print(P.shape)
print(R_n.shape)
err = np.sum((R_n-I)**2)
print(err)
输出:
(1024, 1024, 3)
(1024, 32, 32, 3)
(1024, 1024, 3)
0.0
从extract_image_patches的输出重建似乎很困难。使用其他函数来提取补丁并反转重建过程似乎更容易。
import tensorflow as tf
import numpy as np
c = 3
h = 1024
p = 128
image = tf.random_normal([1,h,h,c])
# Image to Patches Conversion
pad = [[0,0],[0,0]]
patches = tf.space_to_batch_nd(image,[p,p],pad)
patches = tf.split(patches,p*p,0)
patches = tf.stack(patches,3)
patches = tf.reshape(patches,[(h/p)**2,p,p,c])
# Do processing on patches
# Using patches here to reconstruct
patches_proc = tf.reshape(patches,[1,h/p,h/p,p*p,c])
patches_proc = tf.split(patches_proc,p*p,3)
patches_proc = tf.stack(patches_proc,axis=0)
patches_proc = tf.reshape(patches_proc,[p*p,h/p,h/p,c])
reconstructed = tf.batch_to_space_nd(patches_proc,[p, p],pad)
sess = tf.Session()
I,P,R_n = sess.run([image,patches,reconstructed])
print(I.shape)
print(P.shape)
print(R_n.shape)
err = np.sum((R_n-I)**2)
print(err)
输出:
(1, 1024, 1024, 3)
(64, 128, 128, 3)
(1, 1024, 1024, 3)
0.0
你可以在这里看到其他酷张量变换函数:https://www.tensorflow.org/api_guides/python/array_ops
由于我也在努力解决这个问题,因此我发布了一个可能对其他人有用的解决方案。诀窍是要认识到tf.extract_image_patches
的逆是它的梯度,如suggested here。由于此操作的梯度在Tensorflow中实现,因此很容易构建重建功能:
import tensorflow as tf
from keras import backend as K
import numpy as np
def extract_patches(x):
return tf.extract_image_patches(
x,
(1, 3, 3, 1),
(1, 1, 1, 1),
(1, 1, 1, 1),
padding="VALID"
)
def extract_patches_inverse(x, y):
_x = tf.zeros_like(x)
_y = extract_patches(_x)
grad = tf.gradients(_y, _x)[0]
# Divide by grad, to "average" together the overlapping patches
# otherwise they would simply sum up
return tf.gradients(_y, _x, grad_ys=y)[0] / grad
# Generate 10 fake images, last dimension can be different than 3
images = np.random.random((10, 28, 28, 3)).astype(np.float32)
# Extract patches
patches = extract_patches(images)
# Reconstruct image
# Notice that original images are only passed to infer the right shape
images_reconstructed = extract_patches_inverse(images, patches)
# Compare with original (evaluating tf.Tensor into a numpy array)
# Here using Keras session
images_r = images_reconstructed.eval(session=K.get_session())
print (np.sum(np.square(images - images_r)))
# 2.3820458e-11
tf.extract_image_patches
安静难以使用,因为它在背景中做了很多东西。
如果你只是需要不重叠,那么自己编写它会容易得多。您可以通过反转qazxsw poi中的所有操作来重建完整图像。
代码示例(绘制原始图像和补丁):
image_to_patches