pytorch3d.renderer.implicit.renderer

renderer

class pytorch3d.renderer.implicit.renderer.ImplicitRenderer(raysampler: Callable, raymarcher: Callable)

Bases: Module

A class for rendering a batch of implicit surfaces. The class should be initialized with a raysampler and raymarcher class which both have to be a Callable.

VOLUMETRIC_FUNCTION

The forward function of the renderer accepts as input the rendering cameras as well as the volumetric_function Callable, which defines a field of opacity and feature vectors over the 3D domain of the scene.

A standard volumetric_function has the following signature:

def volumetric_function(
    ray_bundle: Union[RayBundle, HeterogeneousRayBundle],
    **kwargs,
) -> Tuple[torch.Tensor, torch.Tensor]

With the following arguments:

ray_bundle: A RayBundle or HeterogeneousRayBundle object

containing the following variables:

origins: A tensor of shape (minibatch, …, 3) denoting

the origins of the rendering rays.

directions: A tensor of shape (minibatch, …, 3)

containing the direction vectors of rendering rays.

lengths: A tensor of shape

`(minibatch, …, num_points_per_ray)`containing the lengths at which the ray points are sampled.

xys: A tensor of shape

(minibatch, …, 2) containing the xy locations of each ray’s pixel in the screen space.

Calling volumetric_function then returns the following:

rays_densities: A tensor of shape

(minibatch, …, num_points_per_ray, opacity_dim) containing the an opacity vector for each ray point.

rays_features: A tensor of shape

(minibatch, …, num_points_per_ray, feature_dim) containing the an feature vector for each ray point.

Note that, in order to increase flexibility of the API, we allow multiple other arguments to enter the volumetric function via additional (optional) keyword arguments **kwargs. A typical use-case is passing a CamerasBase object as an additional keyword argument, which can allow the volumetric function to adjust its outputs based on the directions of the projection rays.

Example

A simple volumetric function of a 0-centered RGB sphere with a unit diameter is defined as follows:

def volumetric_function(
    ray_bundle: Union[RayBundle, HeterogeneousRayBundle],
    **kwargs,
) -> Tuple[torch.Tensor, torch.Tensor]:

    # first convert the ray origins, directions and lengths
    # to 3D ray point locations in world coords
    rays_points_world = ray_bundle_to_ray_points(ray_bundle)

    # set the densities as an inverse sigmoid of the
    # ray point distance from the sphere centroid
    rays_densities = torch.sigmoid(
        -100.0 * rays_points_world.norm(dim=-1, keepdim=True)
    )

    # set the ray features to RGB colors proportional
    # to the 3D location of the projection of ray points
    # on the sphere surface
    rays_features = torch.nn.functional.normalize(
        rays_points_world, dim=-1
    ) * 0.5 + 0.5

    return rays_densities, rays_features

__init__(raysampler: Callable, raymarcher: Callable) → None

Parameters:

• raysampler – A Callable that takes as input scene cameras (an instance of CamerasBase) and returns a RayBundle or HeterogeneousRayBundle, that describes the rays emitted from the cameras.

• raymarcher – A Callable that receives the response of the volumetric_function (an input to self.forward) evaluated along the sampled rays, and renders the rays with a ray-marching algorithm.

forward(cameras: CamerasBase, volumetric_function: Callable, **kwargs) → Tuple[Tensor, RayBundle | HeterogeneousRayBundle]

Render a batch of images using a volumetric function represented as a callable (e.g. a Pytorch module).

Parameters:

• cameras – A batch of cameras that render the scene. A self.raysampler takes the cameras as input and samples rays that pass through the domain of the volumetric function.

• volumetric_function – A Callable that accepts the parametrizations of the rendering rays and returns the densities and features at the respective 3D of the rendering rays. Please refer to the main class documentation for details.

Returns:

images –

A tensor of shape (minibatch, …, feature_dim + opacity_dim)

containing the result of the rendering.

ray_bundle: A Union[RayBundle, HeterogeneousRayBundle] containing

the parametrizations of the sampled rendering rays.

class pytorch3d.renderer.implicit.renderer.VolumeRenderer(raysampler: Callable, raymarcher: Callable, sample_mode: str = 'bilinear')

Bases: Module

A class for rendering a batch of Volumes. The class should be initialized with a raysampler and a raymarcher class which both have to be a Callable.

__init__(raysampler: Callable, raymarcher: Callable, sample_mode: str = 'bilinear') → None

Parameters:

• raysampler – A Callable that takes as input scene cameras (an instance of CamerasBase) and returns a Union[RayBundle, HeterogeneousRayBundle], that describes the rays emitted from the cameras.

• raymarcher – A Callable that receives the volumes (an instance of Volumes input to self.forward) sampled at the ray-points, and renders the rays with a ray-marching algorithm.

• sample_mode – Defines the algorithm used to sample the volumetric voxel grid. Can be either “bilinear” or “nearest”.

forward(cameras: CamerasBase, volumes: Volumes, **kwargs) → Tuple[Tensor, RayBundle | HeterogeneousRayBundle]

Render a batch of images using raymarching over rays cast through input Volumes.

Parameters:

• cameras – A batch of cameras that render the scene. A self.raysampler takes the cameras as input and samples rays that pass through the domain of the volumetric function.

• volumes – An instance of the Volumes class representing a batch of volumes that are being rendered.

Returns:

images –

A tensor of shape (minibatch, …, (feature_dim + opacity_dim)

containing the result of the rendering.

ray_bundle: A RayBundle or HeterogeneousRayBundle containing the

parametrizations of the sampled rendering rays.

class pytorch3d.renderer.implicit.renderer.VolumeSampler(volumes: Volumes, sample_mode: str = 'bilinear', padding_mode: str = 'zeros')

Bases: Module

A module to sample a batch of volumes Volumes at 3D points sampled along projection rays.

__init__(volumes: Volumes, sample_mode: str = 'bilinear', padding_mode: str = 'zeros') → None

Parameters:

• volumes – An instance of the Volumes class representing a batch of volumes that are being rendered.

• sample_mode – Defines the algorithm used to sample the volumetric voxel grid. Can be either “bilinear” or “nearest”.

• padding_mode – How to handle values outside of the volume. One of: zeros, border, reflection See torch.nn.functional.grid_sample for more information.

forward(ray_bundle: RayBundle | HeterogeneousRayBundle, **kwargs) → Tuple[Tensor, Tensor]

Given an input ray parametrization, the forward function samples self._volumes at the respective 3D ray-points. Can also accept ImplicitronRayBundle as argument for ray_bundle.

Parameters:

ray_bundle –

A RayBundle or HeterogeneousRayBundle object with the following fields: rays_origins_world: A tensor of shape (minibatch, …, 3) denoting the

origins of the sampling rays in world coords.

rays_directions_world: A tensor of shape (minibatch, …, 3)

containing the direction vectors of sampling rays in world coords.

rays_lengths: A tensor of shape (minibatch, …, num_points_per_ray)

containing the lengths at which the rays are sampled.

Returns:

rays_densities –

A tensor of shape

(minibatch, …, num_points_per_ray, opacity_dim) containing the density vectors sampled from the volume at the locations of the ray points.

rays_features: A tensor of shape

(minibatch, …, num_points_per_ray, feature_dim) containing the feature vectors sampled from the volume at the locations of the ray points.