pytorch3d.implicitron.models.renderer.ray_sampler

ray_sampler

class pytorch3d.implicitron.models.renderer.ray_sampler.RaySamplerBase(*args, **kwargs)

Bases: ReplaceableBase

Base class for ray samplers.

forward(cameras: CamerasBase, evaluation_mode: EvaluationMode, mask: Tensor | None = None) → ImplicitronRayBundle

Parameters:

• cameras – A batch of batch_size cameras from which the rays are emitted.

• evaluation_mode – one of EvaluationMode.TRAINING or EvaluationMode.EVALUATION which determines the sampling mode that is used.

• mask – Active for the RenderSamplingMode.MASK_SAMPLE sampling mode. Defines a non-negative mask of shape (batch_size, image_height, image_width) where each per-pixel value is proportional to the probability of sampling the corresponding pixel’s ray.

Returns:

ray_bundle –

A ImplicitronRayBundle object containing the parametrizations of the

sampled rendering rays.

class pytorch3d.implicitron.models.renderer.ray_sampler.AbstractMaskRaySampler(*args, **kwargs)

Bases: RaySamplerBase, Module

Samples a fixed number of points along rays which are in turn sampled for each camera in a batch.

This class utilizes NDCMultinomialRaysampler which allows to either randomly sample rays from an input foreground saliency mask (RenderSamplingMode.MASK_SAMPLE), or on a rectangular image grid (RenderSamplingMode.FULL_GRID). The sampling mode can be set separately for training and evaluation by setting self.sampling_mode_training and self.sampling_mode_training accordingly.

The class allows to adjust the sampling points along rays by overwriting the AbstractMaskRaySampler._get_min_max_depth_bounds function which returns the near/far planes (min_depth/max_depth) NDCMultinomialRaysampler.

Settings:

image_width: The horizontal size of the image grid. image_height: The vertical size of the image grid. sampling_mode_training: The ray sampling mode for training. This should be a str

option from the RenderSamplingMode Enum

sampling_mode_evaluation: Same as above but for evaluation. n_pts_per_ray_training: The number of points sampled along each ray during training. n_pts_per_ray_evaluation: The number of points sampled along each ray during evaluation. n_rays_per_image_sampled_from_mask: The amount of rays to be sampled from the image

grid. Given a batch of image grids, this many is sampled from each. n_rays_per_image_sampled_from_mask and n_rays_total_training cannot both be defined.

n_rays_total_training: (optional) How many rays in total to sample from the entire

batch of provided image grid. The result is as if n_rays_total_training cameras/image grids were sampled with replacement from the cameras / image grids provided and for every camera one ray was sampled. n_rays_per_image_sampled_from_mask and n_rays_total_training cannot both be defined, to use you have to set n_rays_per_image to None. Used only for EvaluationMode.TRAINING.

stratified_point_sampling_training: if set, performs stratified random sampling

along the ray; otherwise takes ray points at deterministic offsets.

stratified_point_sampling_evaluation: Same as above but for evaluation. cast_ray_bundle_as_cone: If True, the sampling will generate the bins and radii

attribute of ImplicitronRayBundle. The bins contain the z-coordinate (=depth) of each ray in world units and are of shape (batch_size, n_rays_per_image, n_pts_per_ray_training/evaluation + 1) while lengths is equal to the midpoint of the bins: (0.5 * (bins[…, 1:] + bins[…, :-1]). If False, bins is None, radii is None and lengths contains the z-coordinate (=depth) of each ray in world units and are of shape (batch_size, n_rays_per_image, n_pts_per_ray_training/evaluation)

Raises:

TypeError – if cast_ray_bundle_as_cone is set to True and n_rays_total_training is not None will result in an error. HeterogeneousRayBundle is not supported for conical frustum computation yet.

image_width: int = 400

image_height: int = 400

sampling_mode_training: str = 'mask_sample'

sampling_mode_evaluation: str = 'full_grid'

n_pts_per_ray_training: int = 64

n_pts_per_ray_evaluation: int = 64

n_rays_per_image_sampled_from_mask: int | None = 1024

n_rays_total_training: int | None = None

stratified_point_sampling_training: bool = True

stratified_point_sampling_evaluation: bool = False

cast_ray_bundle_as_cone: bool = False

forward(cameras: CamerasBase, evaluation_mode: EvaluationMode, mask: Tensor | None = None) → ImplicitronRayBundle

Parameters:

• cameras – A batch of batch_size cameras from which the rays are emitted.

• evaluation_mode – one of EvaluationMode.TRAINING or EvaluationMode.EVALUATION which determines the sampling mode that is used.

• mask – Active for the RenderSamplingMode.MASK_SAMPLE sampling mode. Defines a non-negative mask of shape (batch_size, image_height, image_width) where each per-pixel value is proportional to the probability of sampling the corresponding pixel’s ray.

Returns:

ray_bundle –

A ImplicitronRayBundle object containing the parametrizations of the

sampled rendering rays.

class pytorch3d.implicitron.models.renderer.ray_sampler.AdaptiveRaySampler(*args, **kwargs)

Bases: AbstractMaskRaySampler

Adaptively samples points on each ray between near and far planes whose depths are determined based on the distance from the camera center to a predefined scene center.

More specifically, `min_depth = max(

(self.scene_center-camera_center).norm() - self.scene_extent, eps

)` and max_depth = (self.scene_center-camera_center).norm() + self.scene_extent.

This sampling is ideal for object-centric scenes whose contents are centered around a known self.scene_center and fit into a bounding sphere with a radius of self.scene_extent.

Parameters:

• scene_center – The xyz coordinates of the center of the scene used along with scene_extent to compute the min and max depth planes for sampling ray-points.

• scene_extent – The radius of the scene bounding box centered at scene_center.

scene_extent: float = 8.0

scene_center: Tuple[float, float, float] = (0.0, 0.0, 0.0)

class pytorch3d.implicitron.models.renderer.ray_sampler.NearFarRaySampler(*args, **kwargs)

Bases: AbstractMaskRaySampler

Samples a fixed number of points between fixed near and far z-planes. Specifically, samples points along each ray with approximately uniform spacing of z-coordinates between the minimum depth self.min_depth and the maximum depth self.max_depth. This sampling is useful for rendering scenes where the camera is in a constant distance from the focal point of the scene.

Parameters:

• min_depth – The minimum depth of a ray-point.

• max_depth – The maximum depth of a ray-point.

min_depth: float = 0.1

max_depth: float = 8.0

pytorch3d.implicitron.models.renderer.ray_sampler.compute_radii(cameras: CamerasBase, xy_grid: Tensor, pixel_hw_ndc: Tuple[float, float]) → Tensor

Compute radii of conical frustums in world coordinates.

Parameters:

• cameras – cameras object representing a batch of cameras.

• xy_grid – torch.tensor grid of image xy coords.

• pixel_hw_ndc – pixel height and width in NDC

Returns:

radii – A tensor of shape (…, 1) radii of a cone.