reprojectImageto3d后左右图像点重投影不一致

Question

大家。我试图在一个涉及两个摄像头的设置中对位于飞机上的一些点（密集重建）进行三角测量。 [参考图片]：https://imgur.com/gOps4vP和[其他图片]：https://imgur.com/VIiH9Rv

首先，我使用5pts算法解决相对姿势问题对于基本矩阵估计的未失真点，我恢复了姿势。我正在使用RANSAC。

然后，我以通常的方式纠正立体声对。

R1, R2, Pn1, Pn2, Q, _, _ = cv2.stereoRectify(K1, dcoeffs1, K2, dcoeffs2, 
                                                  img1.shape[::-1], R, t, 
                                                  flags=cv2.CALIB_ZERO_DISPARITY, 
                                                  alpha=-1)

    # Compute the rigid transform that OpenCV apply to world points (USEFUL LATER)
    # in order for the rectified reference camera to be K_new[I|0]
    tn_1 = np.zeros((3,1)) # Cameras are never translated in the rectification


    G1_rect = np.block([[R1, tn_1], [np.zeros((1,3)), 1.0]])
    maps1   =   cv2.initUndistortRectifyMap(K1, dcoeffs1, R1, Pn1, (1920,1080), cv2.CV_32FC1)
    maps2   =   cv2.initUndistortRectifyMap(K2, dcoeffs2, R2, Pn2, (1920,1080), cv2.CV_32FC1)
    img1_remap = cv2.remap(img1, maps1[0], maps1[1], cv2.INTER_LANCZOS4)
    img2_remap = cv2.remap(img2, maps2[0], maps2[1], cv2.INTER_LANCZOS4)

整改的结果：[整顿参考图像] https://drive.google.com/open?id=10VfgXrXFO3_lYqtO9qJXr17Dc6F1PuXU [另一个整顿] https://drive.google.com/open?id=13ZkeMiF5xEovGmX13LSQVaJ237hoJLX0

现在我调用一个识别图像（目标）中已知对象的函数。

#Now call a function that recognize a known object in the images (target)
# Find target
target_corners, _ = dt.detectTarget(img_scene1, img_target, 0.5) # return 4 corners of the detected polygon
target_corners = target_corners[:,0,:]
# Compute mask for the target cutout:
target_mask = mp.maskPolygon(target_corners, img_scene1.shape[::-1]) # Output: mask of same dimension of the image

找到目标（请注意突出显示的角落）：[找到目标] https://imgur.com/QjYV8tp

然后我使用StereoSGBM计算视差图。我只对目标视差的计算感兴趣（我将掩盖所有其他点）。利用获得的Disparity图并使用stereoRectify给出的4x4投影矩阵Q，我执行视差图的3d重投影。

    # Compute disparity map
    # https://docs.opencv.org/3.3.1/d2/d85/classcv_1_1StereoSGBM.html
    window_size = 5
    min_disp = 16
    max_disp = 1024
    num_disp = max_disp-min_disp # Deve essere divisibile per 16!

    stereo = cv2.StereoSGBM_create(minDisparity = min_disp,
        numDisparities = num_disp,
        blockSize = window_size,
        P1 = 8*3*window_size**2,
        P2 = 32*3*window_size**2,
        disp12MaxDiff = 1,
        uniquenessRatio = 10,
        speckleWindowSize = 150,
        speckleRange = 2
    )

    print('Calcolo SGBM della disparità...')
    disp = stereo.compute(img_scene1, img_scene2).astype(np.float32) / 16.0

    target_disparity = target_mask*disp

    points = cv2.reprojectImageTo3D(target_disparity, Q)


    # DEBUG:
    cv2.namedWindow('scene1', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('scene1', 800,450)
    cv2.imshow('scene1', img_scene1)
    cv2.namedWindow('disparity', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('disparity', 800,450)
    cv2.imshow('disparity', (disp-min_disp)/num_disp)
    cv2.namedWindow('target_disparity', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('target_disparity', 800,450)
    cv2.imshow('target_disparity', target_mask*(disp-min_disp)/num_disp)
    cv2.waitKey()
    cv2.destroyAllWindows()


    # Obtain matrix of the target 3D points starting from disparity image obtained from reprojectImageTo3D()
    mask_disp = disp > disp.min()
    mask_inf = ~(np.isinf(points[:,:,0]) | np.isinf(points[:,:,1]) | np.isinf(points[:,:,2]))
    mask_nan = ~(np.isnan(points[:,:,0]) | np.isnan(points[:,:,1]) | np.isnan(points[:,:,2]))

    mask = mask_disp & mask_inf & mask_nan

    pts3D = points[mask]

现在，我有3d重建了与目标对应的图像区域。我注意到OpenCv在相机校正期间对世界点应用了刚性变换，使得参考原始相机和新（整流）参考相机具有相同的外在（R = eye（3）和t = [0,0,0] ]'）。事实上，在整流过程中，两个摄像机都必须旋转，我认为OpenCV只是将新摄像机带回一个新的参考，使得参考整流摄像机具有与原始摄像机相同的外部特征。但这意味着重建的三维点将以世界参考的形式表示，而这个参考不是原始相机的世界参考！

因此，将逆刚性变换应用于pts3D，我们在原始参考相机帧中获得重建。 （见代码）。

target3Dpts_hom = cv2.convertPointsToHomogeneous(target3Dpts)[:,0,:].T
    target3Dpts_hom = G.T @ target3Dpts_hom
    new_target3Dpts = cv2.convertPointsFromHomogeneous(target3Dpts_hom.T[:,np.newaxis,:])[:,0,:]

请注意，如果我不执行此操作，则通过投影矩阵在原始相机上重新投影的pt3D将与目标点不对应！

通过重投影检查重建;现在，我可以重新投影new_target3Dpts：让我介绍一下我调用的投影函数：

    def proj_dist(P, dcoeffs, M):

    import numpy as np
    import cv2

    K, R, t,_,_,_,_ = cv2.decomposeProjectionMatrix(P)
    rotv, _ = cv2.Rodrigues(R)

    # Projection. Returns a (N,2) shaped array
    m,_ = cv2.projectPoints(M,rotv,t[0:-1],K,dcoeffs)
    m = m.squeeze()

    return m

最后，重新注入：

    #P_kin = K_kin[eye(3),0] # Originals MPPs of two cameras
    #P_rpi = K_rpi[R,t]

    m0 = proj.proj_dist(P_kin,dcoeffs_kin,new_points).astype('int32')

    for (x, y) in m0:

        x = int(x)
        y= int(y)

        cv2.circle(img_kin, (x, y), 2, (255, 255, 0), 4)

    cv2.namedWindow('frame1', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('frame1', 800,450)
    cv2.imshow('frame1',img_kin)
    cv2.waitKey(0)

    m1 = proj.proj_dist(P_rpi,dcoeffs_rpi,new_points).astype('int32')
    img_rpi1 = img_rpi.copy()
    for (x, y) in m1:
        x = int(x)
        y = int(y)

    cv2.circle(img_rpi1, (x, y), 2, (255, 255, 0), 4)

    cv2.namedWindow('frame2', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('frame2', 800,450)
    cv2.imshow('frame2',img_rpi1)
    cv2.waitKey(0)

但是，虽然原始参考摄像机上的重新投影点是正确的，但对于第二个摄像机来说这是不正确的......这些点只是简单翻译，但我无法解释原因。

结果：[第一帧repj] https://imgur.com/S4lo9Wz [第2帧repj。错误] https://imgur.com/y4igaEI

有任何想法吗？我现在将包含所有代码。谢谢。

SM

Answer 1

我解决了这个与reprojectImageto3D无关的问题 - 这很好用 - 但是我用这段代码编写并且我曾经将这些点重新投影到原始帧上：

def proj_dist(P, dcoeffs, M):

import numpy as np
import cv2

K, R, t,_,_,_,_ = cv2.decomposeProjectionMatrix(P)
rotv, _ = cv2.Rodrigues(R)

# Projection. Returns a (N,2) shaped array
m,_ = cv2.projectPoints(M,rotv,t[0:-1],K,dcoeffs)
m = m.squeeze()

return m

我已经为点投影编写了自己的函数：

def proj(P, M, hom=0):
# proj(): Esegue la proiezione prospettica dei punti 3D M secondo la MPP P,
# sul piano immagine 2D di una camera pinhole.

import numpy as np

n = M.shape[1]
M = np.concatenate((M, np.ones((1,n))))

# Proiezione
m = P @ M

m = m/m[2,:]

if hom !=1 :
    # Passo a cartesiane
    m = m[0:2,:]

return m

问题解决了！我的功能没有考虑到镜头失真。我将进一步研究与projectPoints（）OpenCV函数相关的问题。