当图像中的矩形具有不同的颜色、轮廓并且有时非常接近背景颜色时，如何识别它们

Question

我正在尝试从图像中提取矩形。这些是数字记事本上的数字便签。它们可以是任何用户可配置的颜色，包括带边框的透明颜色。我希望能够输入 jpg/png 文件并获取每个矩形的列表、它们的坐标和矩形的颜色。

OpenCV 与 Python 是我想使用的途径。下面是示例图像，目的是仅检测所有矩形并检索上述信息。

我已经做了很多阅读，并一直在使用查找轮廓方法来尝试实现我的目标，但是我没有得到想要的结果。

import cv2

# reading image
img = cv2.imread('images/example_shapes.jpg')

# converting image into grayscale image
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# setting threshold of gray image
_, threshold = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY)

# using a findContours() function
contours, _ = cv2.findContours(
    threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

i = 0

# list for storing names of shapes
for contour in contours:

    # here we are ignoring first counter because
    # findcontour function detects whole image as shape
    if i == 0:
        i = 1
        continue

    # cv2.approxPloyDP() function to approximate the shape
    approx = cv2.approxPolyDP(
        contour, 0.01 * cv2.arcLength(contour, True), True)

    if len(approx) == 4:
        cv2.drawContours(img, [contour], 0, (0, 0, 255), 5)

# displaying the image after drawing contours
# img = cv2.resize(img, (500, 500))
cv2.imshow('shapes', img)

cv2.waitKey(0)
cv2.destroyAllWindows()

这只会检测到中间的 2 个矩形，并给出以下结果：

然后我尝试将阈值调整为自适应阈值：

threshold = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 13, 7)

产生以下结果

这两种方法似乎都无法检测到靠近的矩形以及与背景颜色接近的矩形，并且都无法检测到带有笔划的矩形。自适应阈值还返回许多不相关的项目。

非常欢迎任何有关如何处理的建议！

Answer 1

此代码首先将图像转换为灰度，然后使用 Canny 边缘检测来查找边缘。然后，它会找到图像中的轮廓，并按形状对其进行过滤以识别矩形。对于每个矩形，它计算其坐标、宽度、高度和颜色。最后，它打印结果并在图像上绘制矩形。

import cv2
import numpy as np

def find_rectangles(image):
  """
  Finds rectangles in an image and returns their coordinates and color.

  Args:
    image: The image to be processed.

  Returns:
    A list of dictionaries, where each dictionary contains the following keys:
      - x: The x-coordinate of the top-left corner of the rectangle.
      - y: The y-coordinate of the top-left corner of the rectangle.
      - w: The width of the rectangle.
      - h: The height of the rectangle.
      - color: The color of the rectangle as a tuple of (B, G, R) values.
  """

  # Convert the image to grayscale
  gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  

  # Find edges using Canny edge detection
  edges = cv2.Canny(gray, 50, 60)

  # Find contours in the image
  contours, _ = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

  # Filter contours by shape
  rectangles = []
  for contour in contours:
    approx = cv2.approxPolyDP(contour, 0.02 * cv2.arcLength(contour, True), True)
    if len(approx) == 4:
      # Get the bounding box of the contour
      x, y, w, h = cv2.boundingRect(approx)
      rectangles.append({
          "x": x,
          "y": y,
          "w": w,
          "h": h,
          "color": cv2.mean(image[y:y+h, x:x+w])
      })

  return rectangles

# Read the image
image = cv2.imread("image_path.jpg")

# Find rectangles
rectangles = find_rectangles(image)

# Print the results
for rectangle in rectangles:
  print(f"Rectangle: x={rectangle['x']}, y={rectangle['y']}, w={rectangle['w']}, h={rectangle['h']}, color={rectangle['color']}")

# Draw the rectangles on the image
for rectangle in rectangles:
  cv2.rectangle(image, (rectangle['x'], rectangle['y']), (rectangle['x'] + rectangle['w'], rectangle['y'] + rectangle['h']), (0, 255, 0), 2)

# Display the image with rectangles
cv2.imshow("Image with Rectangles", cv2.resize(image, None , fx=0.5, fy=0.5))
cv2.waitKey(0)
cv2.destroyAllWindows()