用填充提取矩形框

Question

我正在尝试从多个矩形区域内的2d Tensor中提取值。我想裁剪矩形区域，同时将框外的所有值设置为零。

例如，从9 x 9图像中，我想获得两个单独的图像，其中两个矩形红色框内的值，同时将其余值设置为零。使用张量流切片有一种方便的方法吗？

我想到的一种方法是定义一个掩码数组，该数组在框内为1，在外部为0，并将其与输入数组相乘。但这需要循环遍历框的数量，每次更改掩码的哪些值设置为0.是否有更快更有效的方法来执行此操作而不使用for循环？张量流中是否存在等效的裁剪和替换功能？这是我用for循环的代码。感谢对此的任何意见。谢谢

import tensorflow as tf
import matplotlib.pyplot as plt
import matplotlib.patches as patches

tf.reset_default_graph()

size = 9 # size of input image
num_boxes = 2 # number of rectangular boxes


def get_cutout(X, bboxs):
    """Returns copies of X with values only inside bboxs"""
    out = []
    for i in range(num_boxes):
        bbox = bboxs[i] # get rectangular box coordinates
        Y = tf.Variable(np.zeros((size, size)), dtype=tf.float32) # define temporary mask
        # set values of mask inside box to 1
        t = [Y[bbox[0]:bbox[2], bbox[2]:bbox[3]].assign(
            tf.ones((bbox[2]-bbox[0], bbox[3]-bbox[2])))]
        with tf.control_dependencies(t):
            mask = tf.identity(Y) 
        out.append(X * mask) # get values inside rectangular box
    return out, X

#define a 9x9 input image X and convert to tensor
in_x = np.eye(size)
in_x[0:3]=np.random.rand(3,9)
X = tf.constant(in_x , dtype=tf.float32)

bboxs = tf.placeholder(tf.int32, [None, 4]) # placeholder for rectangular box

X_outs = get_cutout(X, bboxs)

# coordintes of box ((bottom left x, bottom left y, top right x, top right y))
in_bbox = [[1,3,3,6], [4,3,7,8]] 
feed_dict = {bboxs: in_bbox}

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    x_out= sess.run(X_outs, feed_dict=feed_dict)

# plot results
vmin = np.min(x_out[2])
vmax = np.max(x_out[2])
fig, ax = plt.subplots(nrows=1, ncols=1+len(in_bbox),figsize=(10,2))
im = ax[0].imshow(x_out[2], vmin=vmin, vmax=vmax, origin='lower')
plt.colorbar(im, ax=ax[0])
ax[0].set_title("input X")
for i, bbox in enumerate(in_bbox):
    bottom_left = (bbox[2]-0.5, bbox[0]-0.5)
    width = bbox[3]-bbox[2]
    height = bbox[2]- bbox[0]
    rect = patches.Rectangle(bottom_left, width, height,
                             linewidth=1,edgecolor='r',facecolor='none')
    ax[0].add_patch(rect)
    ax[i+1].set_title("extract values in box {}".format(i+1))
    im = ax[i + 1].imshow(x_out[0][i], vmin=vmin, vmax=vmax, origin='lower')
    plt.colorbar(im,ax=ax[i+1])

Answer 1

可以使用tf.pad创建蒙版。

 crop = tf.ones([3, 3])
 # "before_axis_x" how many padding will be added before cropping zone over the axis x
 # "after_axis_x" how many padding will be added after cropping zone over the axis x
 mask = tf.pad(crop, [[before_axis_0, after_axis_0], [before_axis_1, after_axis_1]]

 tf.mask(image, mask) # creates the extracted image

为了与tf.image.crop_and_resize具有相同的行为，这里有一个函数，它将获取一个框数组，并将返回带有填充的提取图像数组。

def extract_with_padding(image, boxes):
  """
   boxes: tensor of shape [num_boxes, 4]. 
          boxes are the coordinates of the extracted part
          box is an array [y1, x1, y2, x2] 
          where [y1, x1] (respectively [y2, x2]) are the coordinates 
          of the top left (respectively bottom right ) part of the image
   image: tensor containing the initial image
  """
  extracted = []
  shape = tf.shape(image)
  for b in boxes:
    crop = tf.ones([3, 3])

    mask = tf.pad(crop, [[b[0], shape[0] - b[2]], [b[1] , shape[1] - b[3]]])
    extracted.append(tf.boolean_mask(image, mask))

  return extracted

Answer 2

谢谢你非常好的功能@edkevekeh。我不得不稍微修改它以使它做我想要的。一，我无法迭代作为Tensor对象的盒子。此外，裁剪尺寸由方框决定，并不总是3x3。此外，tf.boolean_mask返回裁剪，但我想保留裁剪，但在裁剪外部替换为0.所以我用乘法替换了tf.boolean_mask。

对于我的用例，num_boxes可能很大，所以我想知道是否有比for循环更有效的方法，猜测不是。我修改过的@ edkevekeh解决方案，如果有人需要的话。

def extract_with_padding(image, boxes):
    """
    boxes: tensor of shape [num_boxes, 4]. 
          boxes are the coordinates of the extracted part
          box is an array [y1, x1, y2, x2] 
          where [y1, x1] (respectively [y2, x2]) are the coordinates 
          of the top left (respectively bottom right ) part of the image
    image: tensor containing the initial image
    """
    extracted = []
    shape = tf.shape(image)
    for i in range(boxes.shape[0]):
        b = boxes[i]
        crop = tf.ones([b[2] - b[0], b[3] - b[1]])
        mask = tf.pad(crop, [[b[0], shape[0] - b[2]], [b[1] , shape[1] - b[3]]])
        extracted.append(image*mask)
    return extracted