pulsar renderer PyTorch integration.

Proper Python support for pytorch requires creating a torch.autograd.function (independent of whether this is being done within the C++ module). This is done here and a torch.nn.Module is exposed for the use in more complex models.

class pytorch3d.renderer.points.pulsar.renderer.Renderer(width: int, height: int, max_num_balls: int, orthogonal_projection: bool = False, right_handed_system: bool = False, background_normalized_depth: float = <Mock name='pytorch3d._C.EPS' id='140638901539344'>, n_channels: int = 3, n_track: int = 5)[source]

Bases: Module

Differentiable rendering module for the Pulsar renderer.

Set the maximum number of balls to a reasonable value. It is used to determine several buffer sizes. It is no problem to render less balls than this number, but never more.

When optimizing for sphere positions, sphere radiuses or camera parameters you have to use higher gamma values (closer to one) and larger sphere sizes: spheres can only ‘move’ to areas that they cover, and only with higher gamma values exists a gradient w.r.t. their color depending on their position.

  • width (*) – result image width in pixels.

  • height (*) – result image height in pixels.

  • max_num_balls (*) – the maximum number of balls this renderer will handle.

  • orthogonal_projection (*) – use an orthogonal instead of perspective projection. Default: False.

  • right_handed_system (*) – use a right-handed instead of a left-handed coordinate system. This is relevant for compatibility with other drawing or scanning systems. Pulsar by default assumes a left-handed world and camera coordinate system as known from mathematics with x-axis to the right, y axis up and z axis for increasing depth along the optical axis. In the image coordinate system, only the y axis is pointing down, leading still to a left-handed system. If you set this to True, it is assuming a right-handed world and camera coordinate system with x axis to the right, y axis to the top and z axis decreasing along the optical axis. Again, the image coordinate system has a flipped y axis, remaining a right-handed system. Default: False.

  • background_normalized_depth (*) – the normalized depth the background is placed at. This is on a scale from 0. to 1. between the specified min and max depth (see the forward function). The value 0. is the most furthest depth whereas 1. is the closest. Be careful when setting the background too far front - it may hide elements in your scene. Default: EPS.

  • n_channels (*) – the number of image content channels to use. This is usually three for regular color representations, but can be a higher or lower number. Default: 3.

  • n_track (*) – the number of spheres to track for gradient calculation per pixel. Only the closest n_track spheres will receive gradients. Default: 5.

static sphere_ids_from_result_info_nograd(result_info: Tensor) Tensor[source]

Get the sphere IDs from a result info tensor.

static depth_map_from_result_info_nograd(result_info: Tensor) Tensor[source]

Get the depth map from a result info tensor.

This returns a map of the same size as the image with just one channel containing the closest intersection value at that position. Gradients are not available for this tensor, but do note that you can use sphere_ids_from_result_info_nograd to get the IDs of the spheres at each position and directly create a loss on their depth if required.

The depth map contains -1. at positions where no intersection has been detected.

forward(vert_pos: ~torch.Tensor, vert_col: ~torch.Tensor, vert_rad: ~torch.Tensor, cam_params: ~torch.Tensor, gamma: float, max_depth: float, min_depth: float = 0.0, bg_col: ~torch.Tensor | None = None, opacity: ~torch.Tensor | None = None, percent_allowed_difference: float = 0.01, max_n_hits: int = <Mock name='pytorch3d._C.MAX_UINT' id='140638901539280'>, mode: int = 0, return_forward_info: bool = False, first_R_then_T: bool = False) Tensor | Tuple[Tensor, Tensor | None][source]

Rendering pass to create an image from the provided spheres and camera parameters.

  • vert_pos (*) – vertex positions. [Bx]Nx3 tensor of positions in 3D space.

  • vert_col (*) – vertex colors. [Bx]NxK tensor of channels.

  • vert_rad (*) – vertex radii. [Bx]N tensor of radiuses, >0.

  • cam_params (*) –

    camera parameter(s). [Bx]8 tensor, consisting of: - 3 components for camera position, - 3 components for camera rotation (axis angle representation) or

    6 components as described in “On the Continuity of Rotation Representations in Neural Networks” (Zhou et al.),

    • focal length,

    • the sensor width in world coordinates,

    • [optional] an offset for the principal point in x, y (no gradients).

  • gamma (*) – sphere transparency in [1.,1E-5], with 1 being mostly transparent. [Bx]1.

  • max_depth (*) – maximum depth for spheres to render. Set this as tightly as possible to have good numerical accuracy for gradients. float > min_depth + eps.

  • min_depth (*) – a float with the minimum depth a sphere must have to be rendered. Must be 0. or > max(focal_length) + eps.

  • bg_col (*) – K tensor with a background color to use or None (uses all ones).

  • opacity (*) – [Bx]N tensor of opacity values in [0., 1.] or None (uses all ones).

  • percent_allowed_difference (*) – a float in [0., 1.[ with the maximum allowed difference in color space. This is used to speed up the computation. Default: 0.01.

  • max_n_hits (*) – a hard limit on the number of hits per ray. Default: max int.

  • mode (*) – render mode in {0, 1}. 0: render an image; 1: render the hit map.

  • return_forward_info (*) – whether to return a second map. This second map contains 13 channels: first channel contains sm_m (the maximum exponent factor observed), the second sm_d (the normalization denominator, the sum of all coefficients), the third the maximum closest possible intersection for a hit. The following channels alternate with the float encoded integer index of a sphere and its weight. They are the five spheres with the highest color contribution to this pixel color, ordered descending. Default: False.

  • first_R_then_T (*) – bool, whether to first apply rotation to the camera, then translation (PyTorch3D convention). Default: False.


** image* – [Bx]HxWx3 float tensor with the resulting image. * forw_info: [Bx]HxWx13 float forward information as described above, if


extra_repr() str[source]

Extra information to print in pytorch graphs.