pytorch3d.renderer.implicit.raysampling

raysampling

class pytorch3d.renderer.implicit.raysampling.MultinomialRaysampler(*, min_x: float, max_x: float, min_y: float, max_y: float, image_width: int, image_height: int, n_pts_per_ray: int, min_depth: float, max_depth: float, n_rays_per_image: int | None = None, n_rays_total: int | None = None, unit_directions: bool = False, stratified_sampling: bool = False)[source]

Bases: Module

Samples a fixed number of points along rays which are regularly distributed in a batch of rectangular image grids. Points along each ray have uniformly-spaced z-coordinates between a predefined minimum and maximum depth.

The raysampler first generates a 3D coordinate grid of the following form:

   / min_x, min_y, max_depth -------------- / max_x, min_y, max_depth
  /                                        /|
 /                                        / |     ^
/ min_depth                    min_depth /  |     |
min_x ----------------------------- max_x   |     | image
min_y                               min_y   |     | height
|                                       |   |     |
|                                       |   |     v
|                                       |   |
|                                       |   / max_x, max_y,     ^
|                                       |  /  max_depth        /
min_x                               max_y /                   / n_pts_per_ray
max_y ----------------------------- max_x/ min_depth         v
        < --- image_width --- >

In order to generate ray points, MultinomialRaysampler takes each 3D point of the grid (with coordinates [x, y, depth]) and unprojects it with cameras.unproject_points([x, y, depth]), where cameras are an additional input to the forward function.

Note that this is a generic implementation that can support any image grid coordinate convention. For a raysampler which follows the PyTorch3D coordinate conventions please refer to NDCMultinomialRaysampler. As such, NDCMultinomialRaysampler is a special case of MultinomialRaysampler.

min_x: The leftmost x-coordinate of each ray’s source pixel’s center.

max_x: The rightmost x-coordinate of each ray’s source pixel’s center.

min_y: The topmost y-coordinate of each ray’s source pixel’s center.

max_y: The bottommost y-coordinate of each ray’s source pixel’s center.

__init__(*, min_x: float, max_x: float, min_y: float, max_y: float, image_width: int, image_height: int, n_pts_per_ray: int, min_depth: float, max_depth: float, n_rays_per_image: int | None = None, n_rays_total: int | None = None, unit_directions: bool = False, stratified_sampling: bool = False) → None[source]

Parameters:

min_x – The leftmost x-coordinate of each ray’s source pixel’s center.
max_x – The rightmost x-coordinate of each ray’s source pixel’s center.
min_y – The topmost y-coordinate of each ray’s source pixel’s center.
max_y – The bottommost y-coordinate of each ray’s source pixel’s center.
image_width – The horizontal size of the image grid.
image_height – The vertical size of the image grid.
n_pts_per_ray – The number of points sampled along each ray.
min_depth – The minimum depth of a ray-point.
max_depth – The maximum depth of a ray-point.
n_rays_per_image – If given, this amount of rays are sampled from the grid. n_rays_per_image and n_rays_total cannot both be defined.
n_rays_total – How many rays in total to sample from the cameras provided. The result is as if n_rays_total_training cameras were sampled with replacement from the cameras provided and for every camera one ray was sampled. If set returns the HeterogeneousRayBundle with batch_size=n_rays_total. n_rays_per_image and n_rays_total cannot both be defined.
unit_directions – whether to normalize direction vectors in ray bundle.
stratified_sampling – if True, performs stratified random sampling along the ray; otherwise takes ray points at deterministic offsets.

Parameters:

cameras – A batch of batch_size cameras from which the rays are emitted.
mask – if given, the rays are sampled from the mask. Should be of size (batch_size, image_height, image_width).
min_depth – The minimum depth of a ray-point.
max_depth – The maximum depth of a ray-point.
n_rays_per_image – If given, this amount of rays are sampled from the grid. n_rays_per_image and n_rays_total cannot both be defined.
n_pts_per_ray – The number of points sampled along each ray.
stratified_sampling – if set, overrides stratified_sampling provided in __init__.
n_rays_total – How many rays in total to sample from the cameras provided. The result is as if n_rays_total_training cameras were sampled with replacement from the cameras provided and for every camera one ray was sampled. If set returns the HeterogeneousRayBundle with batch_size=n_rays_total. n_rays_per_image and n_rays_total cannot both be defined.

Returns:

A named tuple RayBundle or dataclass HeterogeneousRayBundle with the following fields:

origins: A tensor of shape: (batch_size, s1, s2, 3) denoting the locations of ray origins in the world coordinates.
directions: A tensor of shape: (batch_size, s1, s2, 3) denoting the directions of each ray in the world coordinates.
lengths: A tensor of shape: (batch_size, s1, s2, n_pts_per_ray) containing the z-coordinate (=depth) of each ray in world units.
xys: A tensor of shape: (batch_size, s1, s2, 2) containing the 2D image coordinates of each ray or, if mask is given, (batch_size, n, 1, 2)

Here s1, s2 refer to spatial dimensions. (s1, s2) refer to (highest priority first):

(1, 1) if n_rays_total is provided, (batch_size=n_rays_total)

(n_rays_per_image, 1) if `n_rays_per_image if provided,

(n, 1) where n is the minimum cardinality of the mask
in the batch if mask is provided

(image_height, image_width) if nothing from above is satisfied

HeterogeneousRayBundle has additional members:

camera_ids: tensor of shape (M,), where M is the number of unique sampled
cameras. It represents unique ids of sampled cameras.
camera_counts: tensor of shape (M,), where M is the number of unique sampled
cameras. Represents how many times each camera from camera_ids was sampled

HeterogeneousRayBundle is returned if n_rays_total is provided else RayBundle is returned.

class pytorch3d.renderer.implicit.raysampling.NDCMultinomialRaysampler(*, image_width: int, image_height: int, n_pts_per_ray: int, min_depth: float, max_depth: float, n_rays_per_image: int | None = None, n_rays_total: int | None = None, unit_directions: bool = False, stratified_sampling: bool = False)[source]

Bases: MultinomialRaysampler

Samples a fixed number of points along rays which are regularly distributed in a batch of rectangular image grids. Points along each ray have uniformly-spaced z-coordinates between a predefined minimum and maximum depth.

NDCMultinomialRaysampler follows the screen conventions of the Meshes and Pointclouds renderers. I.e. the pixel coordinates are in [-1, 1]x[-u, u] or [-u, u]x[-1, 1] where u > 1 is the aspect ratio of the image.

For the description of arguments, see the documentation to MultinomialRaysampler.

class pytorch3d.renderer.implicit.raysampling.MonteCarloRaysampler(min_x: float, max_x: float, min_y: float, max_y: float, n_rays_per_image: int, n_pts_per_ray: int, min_depth: float, max_depth: float, *, n_rays_total: int | None = None, unit_directions: bool = False, stratified_sampling: bool = False)[source]

Bases: Module

Samples a fixed number of pixels within denoted xy bounds uniformly at random. For each pixel, a fixed number of points is sampled along its ray at uniformly-spaced z-coordinates such that the z-coordinates range between a predefined minimum and maximum depth.

For practical purposes, this is similar to MultinomialRaysampler without a mask, however sampling at real-valued locations bypassing replacement checks may be faster.

__init__(min_x: float, max_x: float, min_y: float, max_y: float, n_rays_per_image: int, n_pts_per_ray: int, min_depth: float, max_depth: float, *, n_rays_total: int | None = None, unit_directions: bool = False, stratified_sampling: bool = False) → None[source]

Parameters:

min_x – The smallest x-coordinate of each ray’s source pixel.
max_x – The largest x-coordinate of each ray’s source pixel.
min_y – The smallest y-coordinate of each ray’s source pixel.
max_y – The largest y-coordinate of each ray’s source pixel.
n_rays_per_image – The number of rays randomly sampled in each camera. n_rays_per_image and n_rays_total cannot both be defined.
n_pts_per_ray – The number of points sampled along each ray.
min_depth – The minimum depth of each ray-point.
max_depth – The maximum depth of each ray-point.
n_rays_total – How many rays in total to sample from the cameras provided. The result is as if n_rays_total_training cameras were sampled with replacement from the cameras provided and for every camera one ray was sampled. If set returns the HeterogeneousRayBundle with batch_size=n_rays_total. n_rays_per_image and n_rays_total cannot both be defined.
unit_directions – whether to normalize direction vectors in ray bundle.
stratified_sampling – if True, performs stratified sampling in n_pts_per_ray bins for each ray; otherwise takes n_pts_per_ray deterministic points on each ray with uniform offsets.

forward(cameras: CamerasBase, *, stratified_sampling: bool | None = None, **kwargs) → RayBundle | HeterogeneousRayBundle[source]

Parameters:

cameras – A batch of batch_size cameras from which the rays are emitted.
stratified_sampling – if set, overrides stratified_sampling provided in __init__.

Returns:

A named tuple RayBundle or dataclass HeterogeneousRayBundle with the following fields:

origins: A tensor of shape: (batch_size, n_rays_per_image, 3) denoting the locations of ray origins in the world coordinates.
directions: A tensor of shape: (batch_size, n_rays_per_image, 3) denoting the directions of each ray in the world coordinates.
lengths: A tensor of shape: (batch_size, n_rays_per_image, n_pts_per_ray) containing the z-coordinate (=depth) of each ray in world units.
xys: A tensor of shape: (batch_size, n_rays_per_image, 2) containing the 2D image coordinates of each ray.

If n_rays_total is provided batch_size=n_rays_total`and `n_rays_per_image=1 and HeterogeneousRayBundle is returned else RayBundle is returned.

HeterogeneousRayBundle has additional members:

camera_ids: tensor of shape (M,), where M is the number of unique sampled
cameras. It represents unique ids of sampled cameras.
camera_counts: tensor of shape (M,), where M is the number of unique sampled
cameras. Represents how many times each camera from camera_ids was sampled

pytorch3d.renderer.implicit.raysampling.GridRaysampler(min_x: float, max_x: float, min_y: float, max_y: float, image_width: int, image_height: int, n_pts_per_ray: int, min_depth: float, max_depth: float) → MultinomialRaysampler[source]: GridRaysampler has been DEPRECATED. Use MultinomialRaysampler instead. Preserving GridRaysampler for backward compatibility.

pytorch3d.renderer.implicit.raysampling.NDCGridRaysampler(image_width: int, image_height: int, n_pts_per_ray: int, min_depth: float, max_depth: float) → NDCMultinomialRaysampler[source]: NDCGridRaysampler has been DEPRECATED. Use NDCMultinomialRaysampler instead. Preserving NDCGridRaysampler for backward compatibility.