Empty Volume from a RGBD image

[phongnhhn92]

Hi guys, I have been trying to create a Volume from a RGBD image through the Pointcloud. I have been able to render the image given a PerspectiveCameras but when I try to use the function add_pointclouds_to_volumes, the obtain volume is completely zeros. I have included the code and data in the zip file above.
Volume.zip

import torch
import torchvision.transforms as T
import numpy as np
from torchvision.utils import make_grid, save_image
from pytorch3d.utils import pulsar_from_opencv_projection,cameras_from_opencv_projection
from pytorch3d.structures import Meshes,Pointclouds,Volumes
from pytorch3d.renderer import (
    NDCGridRaysampler,
    NDCMultinomialRaysampler,
    ray_bundle_to_ray_points,
    EmissionAbsorptionRaymarcher,
    VolumeRenderer,
    PointsRenderer,
    PointsRasterizer,
    PointsRasterizationSettings,
    AlphaCompositor,
    PulsarPointsRenderer
)
from pytorch3d.ops import add_pointclouds_to_volumes
from einops import rearrange, repeat
torch.backends.cudnn.benchmark = True

data = np.load('data.npy', allow_pickle=True).item()
W,H = 640,512
size_ = torch.tensor([H, W])
# adding batch size
size_ = repeat(size_, 'c -> b c', b=1)
near_far = data['near_far'][0].cpu().numpy()
print('start')
source_imgs = data['imgs'][:,1:].cuda()
intrinsic_up = data['intrinsic'].clone()
intrinsic_up[:,:2] *= 4
N = 1
for i in range(N):
    img = source_imgs[:,i]
    depth = data['input_depths'][:,i].unsqueeze(1)
    c2w = data['input_c2w'][:,i]
    R = c2w[:, :3, :3]
    T = c2w[:, :3, 3]

    cameras_PT = cameras_from_opencv_projection(R,T,intrinsic_up,size_).cuda()

    # convert the depth maps to point clouds using the grid ray sampler
    pts_3d = ray_bundle_to_ray_points(
        NDCGridRaysampler(
            image_width=W,
            image_height=H,
            n_pts_per_ray=1,
            min_depth=0.1,
            max_depth=near_far[1],
        )(cameras_PT)._replace(lengths=depth[:, 0, ..., None].cuda())
    )
    pts_3d = pts_3d.view(1,-1,3)
    features = img.view(1,3,-1).permute(0,2,1)
    
    # Sanity check
    point_cloud = Pointclouds(points=pts_3d, features=features)


    raster_settings = PointsRasterizationSettings(
        image_size=(H,W), 
        radius = 0.001,
        points_per_pixel = 5
    )
    # PointsRasterizer
    rasterizer = PointsRasterizer(cameras=cameras_PT, raster_settings=raster_settings)
    renderer = PointsRenderer(
        rasterizer=rasterizer,
        compositor=AlphaCompositor(background_color=(0,0,0))
    ).cuda()
    images_PC = renderer(point_cloud)
    images_PC = images_PC.permute(0,3,1,2)[:,:3]
    save_image(images_PC,'render_img_PC_{}.png'.format(i))

    initial_volumes = Volumes(
        features = torch.zeros(1, 3, 64, H, W), 
        densities = torch.zeros(1, 1, 64, H, W), voxel_size = near_far[1] / W).cuda()
    updated_volumes = add_pointclouds_to_volumes(
        pointclouds=point_cloud, 
        initial_volumes=initial_volumes, mode="trilinear",)
    
    print(torch.count_nonzero(updated_volumes._features))
print('done!')

[phongnhhn92]

This is the rendered image using PointsRenderer. When I tried to use VolumeRenderer, the rendered image is completely black.

[bottler]

Can you look at the ranges of coordinates of the points and verify that they are within the world coordinates of the volume?

[phongnhhn92]

Hi @bottler, thanks to your comment! I was able to translate the volume so that the world coordinates of the volume fall cover the range of Pointcloud. I also add a sanity check to render the Volume using the same camera but the rendered image is bad. This is the output of the modified script:

volume X range: 2.244824171066284, -2.244824171066284
volume Y range: 2.244824171066284, -2.244824171066284
volume Z range: 5.56982421875, 1.0801758766174316
pts X range: 0.7498435974121094, -1.4710500240325928
pts Y range: 1.0931239128112793, -2.0141003131866455
pts Z range: 5.490264415740967, 1.236769437789917
tensor(9867, device='cuda:0')
tensor(50948., device='cuda:0')
tensor(0., device='cuda:0')
/home/phong/Code/pytorch3d/pytorch3d/renderer/implicit/raymarching.py:187: UserWarning: One or more elements of rays_densities are outside of validrange (0.0, 1.0)
  warnings.warn(
done!

This is the modified script:

import torch
import torchvision.transforms as T
import numpy as np
from torchvision.utils import make_grid, save_image
from pytorch3d.utils import pulsar_from_opencv_projection,cameras_from_opencv_projection
from pytorch3d.structures import Meshes,Pointclouds,Volumes
from pytorch3d.renderer import (
    NDCGridRaysampler,
    NDCMultinomialRaysampler,
    ray_bundle_to_ray_points,
    EmissionAbsorptionRaymarcher,
    VolumeRenderer,
    PointsRenderer,
    PointsRasterizer,
    PointsRasterizationSettings,
    AlphaCompositor,
    PulsarPointsRenderer
)
from pytorch3d.ops import add_pointclouds_to_volumes
from einops import rearrange, repeat
torch.backends.cudnn.benchmark = True

data = np.load('data.npy', allow_pickle=True).item()
W,H = 640,512
size_ = torch.tensor([H, W])
# adding batch size
size_ = repeat(size_, 'c -> b c', b=1)
near_far = data['near_far'][0].cpu().numpy()
print('start')
source_imgs = data['imgs'][:,1:].cuda()
intrinsic_up = data['intrinsic'].clone()
intrinsic_up[:,:2] *= 4
N = 1
for i in range(N):
    img = source_imgs[:,i]
    depth = data['input_depths'][:,i].unsqueeze(1)
    c2w = data['input_c2w'][:,i]
    R = c2w[:, :3, :3]
    T = c2w[:, :3, 3]

    cameras_PT = cameras_from_opencv_projection(R,T,intrinsic_up,size_).cuda()

    # convert the depth maps to point clouds using the grid ray sampler
    pts_3d = ray_bundle_to_ray_points(
        NDCMultinomialRaysampler(
            image_width=W,
            image_height=H,
            n_pts_per_ray=1,
            min_depth=0.1,
            max_depth=near_far[1],
        )(cameras_PT)._replace(lengths=depth[:, 0, ..., None].cuda())
    )
    pts_3d = pts_3d.view(1,-1,3)
    features = img.view(1,3,-1).permute(0,2,1)
    
    # Sanity check
    point_cloud = Pointclouds(points=pts_3d, features=features)


    raster_settings = PointsRasterizationSettings(
        image_size=(H,W), 
        radius = 0.001,
        points_per_pixel = 5
    )
    # PointsRasterizer
    rasterizer = PointsRasterizer(cameras=cameras_PT, raster_settings=raster_settings)
    renderer = PointsRenderer(
        rasterizer=rasterizer,
        compositor=AlphaCompositor(background_color=(0,0,0))
    ).cuda()
    images_PC = renderer(point_cloud)
    images_PC = images_PC.permute(0,3,1,2)[:,:3]
    save_image(images_PC,'render_img_PC_{}.png'.format(i))
    mid = (near_far[0] + near_far[1]) / 2
    initial_volumes = Volumes(
        features = torch.zeros(1, 3, 128, 128, 128), 
        densities = torch.zeros(1, 1, 128, 128, 128),
        voxel_size=near_far[1]/128,
        volume_translation=(0,0,-mid)).cuda()
    coords = initial_volumes.get_coord_grid().view(1,-1,3)
    print('volume X range: {}, {}'.format(coords[:,:,0].max(),coords[:,:,0].min()))
    print('volume Y range: {}, {}'.format(coords[:,:,1].max(),coords[:,:,1].min()))
    print('volume Z range: {}, {}'.format(coords[:,:,2].max(),coords[:,:,2].min()))

    print('pts X range: {}, {}'.format(pts_3d[:,:,0].max(),pts_3d[:,:,0].min()))
    print('pts Y range: {}, {}'.format(pts_3d[:,:,1].max(),pts_3d[:,:,1].min()))
    print('pts Z range: {}, {}'.format(pts_3d[:,:,2].max(),pts_3d[:,:,2].min()))

    updated_volumes = add_pointclouds_to_volumes(
        pointclouds=point_cloud, 
        initial_volumes=initial_volumes, mode="trilinear",)
    
    print(torch.count_nonzero(updated_volumes._features))
    print(updated_volumes._densities.max())
    print(updated_volumes._densities.min())
    raysampler = NDCMultinomialRaysampler(
        image_width=W,
        image_height=H,
        n_pts_per_ray=150,
        min_depth=0.1,
        max_depth=near_far[1],
    )
    raymarcher = EmissionAbsorptionRaymarcher()
    renderer = VolumeRenderer(
        raysampler=raysampler, raymarcher=raymarcher,
    )
    img = renderer(cameras=cameras_PT, volumes=updated_volumes)[0]
    rendered_images, rendered_silhouettes = img.split([3, 1], dim=-1)
    rendered_images = rendered_images.permute(0,3,1,2)
    save_image(rendered_images,'vol.png')
    rendered_silhouettes = rendered_silhouettes.permute(0,3,1,2)
    save_image(rendered_silhouettes,'mask.png')
print('done!')

This is the rendered image using the VolumeRenderer:

Can u check why the ray density is so large on the updated volume ?

[bibbygoodwin]

For those who might still be coming to this issue, the commit referenced above doesn't explicitly solve the large density issue that @phongnhhn92 mentioned in their last comment (which in turn leads to images like the one shared above). As is mentioned in a comment added in that commit

pytorch3d/pytorch3d/ops/points_to_volumes.py

Lines 255 to 257 in 78fd5af

	added points. If True, they are averaged. In both cases,
	output densities are just summed without rescaling, so
	you may need to rescale them afterwards.

, the densities after using add_pointclouds_to_volumes are not scaled, and so rescaling is necessary in order to get sensible rendering.

As a running example, I'm using a volume representing a moped. Without rescaling, the max density in the volume after add_pointclouds_to_volumes is 587.2265. Rendering leads to images like this:

To get something to work a bit better, scale the densities by this max density value:

volumes = Volumes(
            densities = updated_volumes.densities() / updated_volumes.densities().max(),
            features = updated_volumes.features(),
            voxel_size=volume_extent_world / volume_size,
        ).to(device)

ims = renderer(cameras=target_cameras.to(device), volumes=volumes)[0]
rgb, sil = ims.split([3, 1], dim=-1)

This gives a render like the following

However, it's still unclear how to optimally scale these densities -- in general, using the global max leads to mostly low densities in places which (by most point cloud metrics) would not be low density. Heuristically tuning, you can get better looking results, e.g. here where I scale by 1/5th of the maximum density: in generally, the image displays truer colours and less alpha. However, you can see some pixels are already being affected by high density (clipping to black/white values):

Hope this helps someone. I suspect this behaviour is not an "issue", but a fundamental challenge with inferring voxel grid densities from point clouds? It might still be nice to hear from the team on this.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.