从 detectorron2 全景分割中提取信息并使用它来掩盖图像中的特定对象

Question

理论上，我想模仿本博客开头所做的事情：https://www.sicara.fr/blog-technique/dataset- Generation-fine-tune-stable-diffusion-inpainting

我需要使用 detectorron2 来完成此操作，以便使用它的全景分割功能。

但是，在通过全景检查点处理图像后，我很难正确提取信息（边界坐标、类标签等）。

如果我要像博客中那样定位要屏蔽的特定对象，我需要这些信息。

import cv2
import numpy as np
from detectron2 import model_zoo
from detectron2.config import get_cfg
from detectron2.engine import DefaultPredictor
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog

width, height = 512, 512

# Load the Panoptic Segmentation model
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-PanopticSegmentation/panoptic_fpn_R_50_3x.yaml"))
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-PanopticSegmentation/panoptic_fpn_R_50_3x.yaml")
predictor = DefaultPredictor(cfg)

# Load the input image
image = cv2.imread("dog.jpg")

# Preprocess the image (resize, convert to RGB)
image = cv2.resize(image, (width, height))
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Perform panoptic segmentation
panoptic_output = predictor(image)

# Get the panoptic segmentation instances
instances = panoptic_output["instances"]

# Visualize the predictions
v = Visualizer(image[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
v = v.draw_panoptic_seg_predictions(panoptic_output["panoptic_seg"], instances)
output_image = v.get_image()[:, :, ::-1]

# Create a mask image of the same size as the input image
mask_image = np.zeros(image.shape[:2], dtype=np.uint8)

# Get the panoptic segmentation mask
panoptic_mask = instances.pred_masks.numpy()

# Iterate over the panoptic segmentation mask and extract the mask of dogs
for idx, mask in enumerate(panoptic_mask):
    if instances.pred_classes[idx] == 16:  # Assuming class ID 16 represents dogs
        mask_image += mask.astype(np.uint8) * 255

# Perform translation on the mask image
mask_image += np.roll(mask_image, 10, axis=0)  # Translate the mask 10 pixels to the left
mask_image += np.roll(mask_image, -10, axis=0)  # Translate the mask 10 pixels to the right
mask_image += np.roll(mask_image, 10, axis=1)  # Translate the mask 10 pixels to the bottom
mask_image += np.roll(mask_image, -10, axis=1)  # Translate the mask 10 pixels to the top

# Set non-black pixels to white pixels
mask_image = np.clip(mask_image, 0, 255)

# Save the mask image
cv2.imwrite("mask_image.jpg", mask_image)

我到处都遇到错误。这是一个例子： return _VF.meshgrid(tensors, **kwargs) # 类型：忽略[attr-defined] 回溯（最近一次调用最后一次）： 文件“C:\Users\Daniel Kolb\Moreland-Connect\SD-Housing-Proj\Diffusers est.py”，第 32 行，位于 v = v.draw_panoptic_seg_predictions(panoptic_output["panoptic_seg"], 实例) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 文件“c：\ users \ daniel kolb \ moreland-connect \ sd-housing-proj \ diffusers \ detectorron2 \ detectorron2 \ utils isualizer.py”，第488行，在draw_panoptic_seg中 pred = _PanopticPrediction（panoptic_seg，segments_info，self.metadata） ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^ 文件“c：\ users \ daniel kolb \ moreland-connect \ sd-housing-proj \ diffusers \ detectorron2 \ detectorron2 \ utils isualizer.py”，第186行，在init中 self._sinfo = {s["id"]: s for s in paragraphs_info} # seg id -> seg info ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 文件“c:\users\daniel kolb\moreland-connect\sd-housing-proj\diffusers\Detectron2\Detectron2\structs\instances.py”，第 151 行，位于 iter raise NotImplementedError("

Instances

对象不可迭代！") NotImplementedError：

Instances

对象不可迭代！

Answer 1

detectron2

已更新，以便预测器中的

panoptic_seg

键输出全景分割图像和分段信息。因此，您应该将其更新为：

# Get the panoptic segmentation instances
panoptic_seg, segments_info = panoptic_output["panoptic_seg"]

# Visualize the predictions
v = Visualizer(image[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
v = v.draw_panoptic_seg_predictions(panoptic_seg, segments_info)