pytorch3d.implicitron.models.renderer.multipass_ea
multipass_ea
- class pytorch3d.implicitron.models.renderer.multipass_ea.MultiPassEmissionAbsorptionRenderer(*args, **kwargs)[source]
Bases:
BaseRenderer
,Module
Implements the multi-pass rendering function, in particular, with emission-absorption ray marching used in NeRF [1]. First, it evaluates opacity-based ray-point weights and then optionally (in case more implicit functions are given) resamples points using importance sampling and evaluates new weights.
During each ray marching pass, features, depth map, and masks are integrated: Let o_i be the opacity estimated by the implicit function, and d_i be the offset between points i and i+1 along the respective ray. Ray marching is performed using the following equations:
ray_opacity_n = cap_fn(sum_i=1^n cap_fn(d_i * o_i)), weight_n = weight_fn(cap_fn(d_i * o_i), 1 - ray_opacity_{n-1}),
and the final rendered quantities are computed by a dot-product of ray values with the weights, e.g. features = sum_n(weight_n * ray_features_n).
- By default, for the EA raymarcher from [1] (
activated with self.raymarcher_class_type=”EmissionAbsorptionRaymarcher”
):
cap_fn(x) = 1 - exp(-x), weight_fn(x) = w * x.
Note that the latter can altered by changing self.raymarcher_class_type, e.g. to “CumsumRaymarcher” which implements the cumulative-sum raymarcher from NeuralVolumes [2].
- Settings:
- n_pts_per_ray_fine_training: The number of points sampled per ray for the
fine rendering pass during training.
- n_pts_per_ray_fine_evaluation: The number of points sampled per ray for the
fine rendering pass during evaluation.
- stratified_sampling_coarse_training: Enable/disable stratified sampling in the
refiner during training. Only matters if there are multiple implicit functions (i.e. in GenericModel if num_passes>1).
- stratified_sampling_coarse_evaluation: Enable/disable stratified sampling in
the refiner during evaluation. Only matters if there are multiple implicit functions (i.e. in GenericModel if num_passes>1).
- append_coarse_samples_to_fine: Add the fine ray points to the coarse points
after sampling.
- density_noise_std_train: Standard deviation of the noise added to the
opacity field.
- return_weights: Enables returning the rendering weights of the EA raymarcher.
Setting to True can lead to a prohibitivelly large memory consumption.
- blurpool_weights: Use blurpool defined in [3], on the input weights of
each implicit_function except the first (implicit_functions[0]).
sample_pdf_eps: Padding applied to the weights (alpha in equation 18 of [3]). raymarcher_class_type: The type of self.raymarcher corresponding to
a child of RaymarcherBase in the registry.
- raymarcher: The raymarcher object used to convert per-point features
and opacities to a feature render.
References
- [1] Mildenhall, Ben, et al. “Nerf: Representing Scenes as Neural Radiance
Fields for View Synthesis.” ECCV 2020.
- [2] Lombardi, Stephen, et al. “Neural Volumes: Learning Dynamic Renderable
Volumes from Images.” SIGGRAPH 2019.
- [3] Jonathan T. Barron, et al. “Mip-NeRF: A Multiscale Representation
for Anti-Aliasing Neural Radiance Fields.” ICCV 2021.
- raymarcher_class_type: str = 'EmissionAbsorptionRaymarcher'
- raymarcher: RaymarcherBase
- n_pts_per_ray_fine_training: int = 64
- n_pts_per_ray_fine_evaluation: int = 64
- stratified_sampling_coarse_training: bool = True
- stratified_sampling_coarse_evaluation: bool = False
- append_coarse_samples_to_fine: bool = True
- density_noise_std_train: float = 0.0
- return_weights: bool = False
- blurpool_weights: bool = False
- sample_pdf_eps: float = 1e-05
- forward(ray_bundle: ImplicitronRayBundle, implicit_functions: List[ImplicitFunctionWrapper], evaluation_mode: EvaluationMode = EvaluationMode.EVALUATION, **kwargs) RendererOutput [source]
- Parameters:
ray_bundle – A ImplicitronRayBundle object containing the parametrizations of the sampled rendering rays.
implicit_functions – List of ImplicitFunctionWrappers which define the implicit functions to be used sequentially in the raymarching step. The output of raymarching with implicit_functions[n-1] is refined, and then used as input for raymarching with implicit_functions[n].
evaluation_mode – one of EvaluationMode.TRAINING or EvaluationMode.EVALUATION which determines the settings used for rendering
- Returns:
instance of RendererOutput