pytorch3d.implicitron.models.renderer.ray_sampler
ray_sampler
- class pytorch3d.implicitron.models.renderer.ray_sampler.RaySamplerBase(*args, **kwargs)[source]
Bases:
ReplaceableBase
Base class for ray samplers.
- forward(cameras: CamerasBase, evaluation_mode: EvaluationMode, mask: Tensor | None = None) ImplicitronRayBundle [source]
- Parameters:
cameras – A batch of batch_size cameras from which the rays are emitted.
evaluation_mode – one of EvaluationMode.TRAINING or EvaluationMode.EVALUATION which determines the sampling mode that is used.
mask – Active for the RenderSamplingMode.MASK_SAMPLE sampling mode. Defines a non-negative mask of shape (batch_size, image_height, image_width) where each per-pixel value is proportional to the probability of sampling the corresponding pixel’s ray.
- Returns:
ray_bundle –
- A ImplicitronRayBundle object containing the parametrizations of the
sampled rendering rays.
- class pytorch3d.implicitron.models.renderer.ray_sampler.AbstractMaskRaySampler(*args, **kwargs)[source]
Bases:
RaySamplerBase
,Module
Samples a fixed number of points along rays which are in turn sampled for each camera in a batch.
This class utilizes NDCMultinomialRaysampler which allows to either randomly sample rays from an input foreground saliency mask (RenderSamplingMode.MASK_SAMPLE), or on a rectangular image grid (RenderSamplingMode.FULL_GRID). The sampling mode can be set separately for training and evaluation by setting self.sampling_mode_training and self.sampling_mode_training accordingly.
The class allows to adjust the sampling points along rays by overwriting the AbstractMaskRaySampler._get_min_max_depth_bounds function which returns the near/far planes (min_depth/max_depth) NDCMultinomialRaysampler.
- Settings:
image_width: The horizontal size of the image grid. image_height: The vertical size of the image grid. sampling_mode_training: The ray sampling mode for training. This should be a str
option from the RenderSamplingMode Enum
sampling_mode_evaluation: Same as above but for evaluation. n_pts_per_ray_training: The number of points sampled along each ray during training. n_pts_per_ray_evaluation: The number of points sampled along each ray during evaluation. n_rays_per_image_sampled_from_mask: The amount of rays to be sampled from the image
grid. Given a batch of image grids, this many is sampled from each. n_rays_per_image_sampled_from_mask and n_rays_total_training cannot both be defined.
- n_rays_total_training: (optional) How many rays in total to sample from the entire
batch of provided image grid. The result is as if n_rays_total_training cameras/image grids were sampled with replacement from the cameras / image grids provided and for every camera one ray was sampled. n_rays_per_image_sampled_from_mask and n_rays_total_training cannot both be defined, to use you have to set n_rays_per_image to None. Used only for EvaluationMode.TRAINING.
- stratified_point_sampling_training: if set, performs stratified random sampling
along the ray; otherwise takes ray points at deterministic offsets.
stratified_point_sampling_evaluation: Same as above but for evaluation. cast_ray_bundle_as_cone: If True, the sampling will generate the bins and radii
attribute of ImplicitronRayBundle. The bins contain the z-coordinate (=depth) of each ray in world units and are of shape (batch_size, n_rays_per_image, n_pts_per_ray_training/evaluation + 1) while lengths is equal to the midpoint of the bins: (0.5 * (bins[…, 1:] + bins[…, :-1]). If False, bins is None, radii is None and lengths contains the z-coordinate (=depth) of each ray in world units and are of shape (batch_size, n_rays_per_image, n_pts_per_ray_training/evaluation)
- Raises:
TypeError – if cast_ray_bundle_as_cone is set to True and n_rays_total_training is not None will result in an error. HeterogeneousRayBundle is not supported for conical frustum computation yet.
- image_width: int = 400
- image_height: int = 400
- sampling_mode_training: str = 'mask_sample'
- sampling_mode_evaluation: str = 'full_grid'
- n_pts_per_ray_training: int = 64
- n_pts_per_ray_evaluation: int = 64
- n_rays_per_image_sampled_from_mask: int | None = 1024
- n_rays_total_training: int | None = None
- stratified_point_sampling_training: bool = True
- stratified_point_sampling_evaluation: bool = False
- cast_ray_bundle_as_cone: bool = False
- forward(cameras: CamerasBase, evaluation_mode: EvaluationMode, mask: Tensor | None = None) ImplicitronRayBundle [source]
- Parameters:
cameras – A batch of batch_size cameras from which the rays are emitted.
evaluation_mode – one of EvaluationMode.TRAINING or EvaluationMode.EVALUATION which determines the sampling mode that is used.
mask – Active for the RenderSamplingMode.MASK_SAMPLE sampling mode. Defines a non-negative mask of shape (batch_size, image_height, image_width) where each per-pixel value is proportional to the probability of sampling the corresponding pixel’s ray.
- Returns:
ray_bundle –
- A ImplicitronRayBundle object containing the parametrizations of the
sampled rendering rays.
- class pytorch3d.implicitron.models.renderer.ray_sampler.AdaptiveRaySampler(*args, **kwargs)[source]
Bases:
AbstractMaskRaySampler
Adaptively samples points on each ray between near and far planes whose depths are determined based on the distance from the camera center to a predefined scene center.
More specifically, `min_depth = max(
(self.scene_center-camera_center).norm() - self.scene_extent, eps
)` and max_depth = (self.scene_center-camera_center).norm() + self.scene_extent.
This sampling is ideal for object-centric scenes whose contents are centered around a known self.scene_center and fit into a bounding sphere with a radius of self.scene_extent.
- Parameters:
scene_center – The xyz coordinates of the center of the scene used along with scene_extent to compute the min and max depth planes for sampling ray-points.
scene_extent – The radius of the scene bounding box centered at scene_center.
- scene_extent: float = 8.0
- scene_center: Tuple[float, float, float] = (0.0, 0.0, 0.0)
- class pytorch3d.implicitron.models.renderer.ray_sampler.NearFarRaySampler(*args, **kwargs)[source]
Bases:
AbstractMaskRaySampler
Samples a fixed number of points between fixed near and far z-planes. Specifically, samples points along each ray with approximately uniform spacing of z-coordinates between the minimum depth self.min_depth and the maximum depth self.max_depth. This sampling is useful for rendering scenes where the camera is in a constant distance from the focal point of the scene.
- Parameters:
min_depth – The minimum depth of a ray-point.
max_depth – The maximum depth of a ray-point.
- min_depth: float = 0.1
- max_depth: float = 8.0
- pytorch3d.implicitron.models.renderer.ray_sampler.compute_radii(cameras: CamerasBase, xy_grid: Tensor, pixel_hw_ndc: Tuple[float, float]) Tensor [source]
Compute radii of conical frustums in world coordinates.
- Parameters:
cameras – cameras object representing a batch of cameras.
xy_grid – torch.tensor grid of image xy coords.
pixel_hw_ndc – pixel height and width in NDC
- Returns:
radii – A tensor of shape (…, 1) radii of a cone.