根据网格和相机参数渲染深度

Question

我想从网格文件和相机参数渲染深度，为此我尝试使用此

网格文件

从open3d RaycastingScene，如下所示：

#!/usr/bin/env python3

import numpy as np
import open3d as o3d
import matplotlib.pyplot as plt


def render_depth(
    intrins:o3d.core.Tensor,
    width:int,
    height:int,
    extrins:o3d.core.Tensor,
    tmesh:o3d.t.geometry.TriangleMesh
)->np.ndarray:
    """
    Render depth from mesh file

    Parameters
    ----------
    intrins : o3d.core.Tensor
        Camera Intrinsics matrix K: 3x3
    width : int
        image width
    height : int
        image height
    extrins : o3d.core.Tensor
        camera extrinsics matrix 4x4
    tmesh : o3d.t.geometry.TriangleMesh
        TriangleMesh

    Returns
    -------
    np.ndarray
        Rendred depth image
    """
    scene = o3d.t.geometry.RaycastingScene()
    scene.add_triangles(tmesh)
    
    rays = scene.create_rays_pinhole(
        intrinsic_matrix=intrins,
        extrinsic_matrix=extrins,
        width_px=width, height_px=height
    )
    
    ans = scene.cast_rays(rays)
    t_hit = ans["t_hit"].numpy() / 1000.0

    return t_hit


if __name__=="__main__":
    import os
    mesh_path = f"{os.getenv('HOME')}/bbq_sauce.ply"
    mesh = o3d.t.io.read_triangle_mesh(mesh_path)
    mesh.compute_vertex_normals()
    
    # camera_info[k].reshape(3, 3)
    intrins_ = np.array([
            [606.9275512695312, 0.0, 321.9704895019531],
            [0.0, 606.3505859375, 243.5377197265625],
            [0.0, 0.0, 1.0]
            ])
    width_  = 640 # camera_info.width
    height_ = 480 # camera_info.height
    # root2cam 4x4
    extrens_ = np.eye(4)
    #
    intrins_t = o3d.core.Tensor(intrins_)
    extrins_t = o3d.core.Tensor(extrens_)
    
    rendered_depth = render_depth(
        intrins=intrins_t, 
        width=width_, 
        height=height_, 
        extrins = extrins_t, 
        tmesh=mesh
    )
    
    plt.imshow(rendered_depth)
    plt.show()

但我得到的深度图像似乎不正确！

你能告诉我如何解决这个问题吗？谢谢。

Answer 1

光线投射场景工作正常。然而，您的外部矩阵将相机设置在网格内部，因此您可以看到网格从内部看起来如何。

我在 meshlab 中粗略测量了网格

bbq_sauce

，它的宽度约为 53 个单位。您可以使用以下任一方法将相机移离网格：

extrens_[2, 3] = 200

或者不移动对象，只需移动相机姿势。

rays = o3d.t.geometry.RaycastingScene.create_rays_pinhole(
        fov_deg=90,  # Not computed from intrinsic parameters.
        center=[0, 0, 0],  # Look towards
        eye=[100, 100, 0],  # Look from
        up=[0, 1, 0],  # This helps in orienting the camera so object is not inverted.
        width_px=640,
        height_px=480,
    )