oak-d 相机上的深度和彩色图像对齐

Question

我正在尝试创建 RGB-D 数据集，但在对齐 Oak-D 相机的颜色和深度图像时遇到一些麻烦。深度图像与立体装备的右相机对齐，目标是将其与彩色图像对齐，相机位于立体装备的中心。为此，我想使用校准参数。我有两个相机的内在矩阵（K1，K2）以及从一个相机引用转换为另一个相机引用的矩阵（D）。 D 是一个 4x4 齐次矩阵，包含两个相机之间的旋转和平移矩阵。我已经仅使用旋转矩阵执行了校准，使用公式：

H = K2 * R_2_1 * inv(K1)

。 尽管我在考虑平移矩阵时也遇到了问题，但我发现这在我的情况下非常必要。谁能给我提供最终变换矩阵的公式，或其他方法，以便我可以在考虑两个相机之间的旋转和平移矩阵的情况下执行颜色和深度图像之间的对齐？

这是我用来执行对齐的代码：

import cv2
import numpy as np
from google.colab.patches import cv2_imshow

# Intrinsic parameters of RGB camera
K2 = np.array([[553.0378462082966, 0, 355.449626693058],
              [0, 555.6255112161116, 197.16318449210289],
              [0, 0, 1]])

# Intrinsic parameters of depth camera
K1 = np.array([[426.6990467952769, 0, 319.5763569688426],
              [0, 428.2569124831735, 203.66385041327385],
              [0, 0, 1]])

# Distortion coefficients of RGB camera
distortion_coeffs = np.array([-0.01015686833772626, 0.018782266649409993, 0.0015606228293145892, -0.00019459459264923275])

#-0.07501121652962993/2
# Homogeneous transformation matrix
D = np.array([[0.9999971496369501, 0.00012642174287522421, -0.0023842683365914922, -0.07501121652962993/2],
            [-0.00012887613805504113, 0.9999994619796904, -0.0010292867693778803, -4.5329260960026137e-05],
            [0.002384136929579421, 0.0010295911108322208, 0.9999966279109286, -0.0005594943499687595],
            [0, 0, 0, 1]])

# Load depth and color images
depth = cv2.imread("/content/drive/MyDrive/RGBD_images/img_00001_dep.png", cv2.IMREAD_GRAYSCALE)
color = cv2.imread("/content/drive/MyDrive/RGBD_images/img_00001_rgb.jpg")

# Undistort the color image
height, width = color.shape[:2]
newcameramtx, roi = cv2.getOptimalNewCameraMatrix(K2, distortion_coeffs, (width, height), 0, (width, height))
color_undistorted = cv2.undistort(color, K2, distortion_coeffs, None, newcameramtx)

# Resize color image to the same size as depth image
resized_depth = cv2.resize(depth, (color_undistorted.shape[1], color_undistorted.shape[0]))

# Compute the transformation matrix H
R = D[:3, :3]
t = D[:3, 3].reshape(-1, 1)
H = np.dot(K2, np.dot(R, np.linalg.inv(K1)))

# Warp the depth image to align with the color image
aligned_depth = cv2.warpPerspective(resized_depth, H, (color.shape[1], color.shape[0]))

# Step 5: Project the transformed depth image onto the color image
depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(aligned_depth, alpha=0.03), cv2.COLORMAP_JET)
aligned_image = cv2.addWeighted(color_undistorted, 0.5, depth_colormap, 0.5, 0)


# Display the aligned depth and color images
cv2_imshow(color_undistorted)
cv2_imshow(resized_depth)
cv2_imshow(aligned_image)

# Save the image
cv2.imwrite("/content/drive/MyDrive/selected_RGBD_images/aligned_depth_0000.jpg", aligned_image)

Answer 1

解决了吗？请分享你的方法。