pytorch3d.renderer.mesh.rasterize_meshes

rasterize_meshes

pytorch3d.renderer.mesh.rasterize_meshes.rasterize_meshes(meshes, image_size: int | List[int] | Tuple[int, int] = 256, blur_radius: float = 0.0, faces_per_pixel: int = 8, bin_size: int | None = None, max_faces_per_bin: int | None = None, perspective_correct: bool = False, clip_barycentric_coords: bool = False, cull_backfaces: bool = False, z_clip_value: float | None = None, cull_to_frustum: bool = False)

Rasterize a batch of meshes given the shape of the desired output image. Each mesh is rasterized onto a separate image of shape (H, W) if image_size is a tuple or (image_size, image_size) if it is an int.

If the desired image size is non square (i.e. a tuple of (H, W) where H != W) the aspect ratio needs special consideration. There are two aspect ratios to be aware of:

• the aspect ratio of each pixel

• the aspect ratio of the output image

The camera can be used to set the pixel aspect ratio. In the rasterizer, we assume square pixels, but variable image aspect ratio (i.e rectangle images).

In most cases you will want to set the camera aspect ratio to 1.0 (i.e. square pixels) and only vary the image_size (i.e. the output image dimensions in pixels).

Parameters:

• meshes – A Meshes object representing a batch of meshes, batch size N.

• image_size – Size in pixels of the output image to be rasterized. Can optionally be a tuple of (H, W) in the case of non square images.

• blur_radius – Float distance in the range [0, 2] used to expand the face bounding boxes for rasterization. Setting blur radius results in blurred edges around the shape instead of a hard boundary. Set to 0 for no blur.

• faces_per_pixel (Optional) – Number of faces to save per pixel, returning the nearest faces_per_pixel points along the z-axis.

• bin_size – Size of bins to use for coarse-to-fine rasterization. Setting bin_size=0 uses naive rasterization; setting bin_size=None attempts to set it heuristically based on the shape of the input. This should not affect the output, but can affect the speed of the forward pass.

• max_faces_per_bin – Only applicable when using coarse-to-fine rasterization (bin_size > 0); this is the maximum number of faces allowed within each bin. This should not affect the output values, but can affect the memory usage in the forward pass.

• perspective_correct – Bool, Whether to apply perspective correction when computing barycentric coordinates for pixels. This should be set to True if a perspective camera is used.

• clip_barycentric_coords – Whether, after any perspective correction is applied but before the depth is calculated (e.g. for z clipping), to “correct” a location outside the face (i.e. with a negative barycentric coordinate) to a position on the edge of the face.

• cull_backfaces – Bool, Whether to only rasterize mesh faces which are visible to the camera. This assumes that vertices of front-facing triangles are ordered in an anti-clockwise fashion, and triangles that face away from the camera are in a clockwise order relative to the current view direction. NOTE: This will only work if the mesh faces are consistently defined with counter-clockwise ordering when viewed from the outside.

• z_clip_value – if not None, then triangles will be clipped (and possibly subdivided into smaller triangles) such that z >= z_clip_value. This avoids camera projections that go to infinity as z->0. Default is None as clipping affects rasterization speed and should only be turned on if explicitly needed. See clip.py for all the extra computation that is required.

• cull_to_frustum – if True, triangles outside the view frustum will be culled. Culling involves removing all faces which fall outside view frustum. Default is False so that it is turned on only when needed.

Returns:

4-element tuple containing

• pix_to_face: LongTensor of shape (N, image_size, image_size, faces_per_pixel) giving the indices of the nearest faces at each pixel, sorted in ascending z-order. Concretely pix_to_face[n, y, x, k] = f means that faces_verts[f] is the kth closest face (in the z-direction) to pixel (y, x). Pixels that are hit by fewer than faces_per_pixel are padded with -1.

• zbuf: FloatTensor of shape (N, image_size, image_size, faces_per_pixel) giving the NDC z-coordinates of the nearest faces at each pixel, sorted in ascending z-order. Concretely, if pix_to_face[n, y, x, k] = f then zbuf[n, y, x, k] = face_verts[f, 2]. Pixels hit by fewer than faces_per_pixel are padded with -1.

• barycentric: FloatTensor of shape (N, image_size, image_size, faces_per_pixel, 3) giving the barycentric coordinates in NDC units of the nearest faces at each pixel, sorted in ascending z-order. Concretely, if pix_to_face[n, y, x, k] = f then [w0, w1, w2] = barycentric[n, y, x, k] gives the barycentric coords for pixel (y, x) relative to the face defined by face_verts[f]. Pixels hit by fewer than faces_per_pixel are padded with -1.

• pix_dists: FloatTensor of shape (N, image_size, image_size, faces_per_pixel) giving the signed Euclidean distance (in NDC units) in the x/y plane of each point closest to the pixel. Concretely if pix_to_face[n, y, x, k] = f then pix_dists[n, y, x, k] is the squared distance between the pixel (y, x) and the face given by vertices face_verts[f]. Pixels hit with fewer than faces_per_pixel are padded with -1.

In the case that image_size is a tuple of (H, W) then the outputs will be of shape (N, H, W, …).

pytorch3d.renderer.mesh.rasterize_meshes.non_square_ndc_range(S1, S2)

In the case of non square images, we scale the NDC range to maintain the aspect ratio. The smaller dimension has NDC range of 2.0.

Parameters:

• S1 – dimension along with the NDC range is needed

• S2 – the other image dimension

Returns:

ndc_range – NDC range for dimension S1

pytorch3d.renderer.mesh.rasterize_meshes.pix_to_non_square_ndc(i, S1, S2)

The default value of the NDC range is [-1, 1]. However in the case of non square images, we scale the NDC range to maintain the aspect ratio. The smaller dimension has NDC range from [-1, 1] and the other dimension is scaled by the ratio of H:W. e.g. for image size (H, W) = (64, 128)

Height NDC range: [-1, 1] Width NDC range: [-2, 2]

Parameters:

• i – pixel position on axes S1

• S1 – dimension along with i is given

• S2 – the other image dimension

Returns:

pixel – NDC coordinate of point i for dimension S1

pytorch3d.renderer.mesh.rasterize_meshes.rasterize_meshes_python(meshes, image_size: int | Tuple[int, int] = 256, blur_radius: float = 0.0, faces_per_pixel: int = 8, perspective_correct: bool = False, clip_barycentric_coords: bool = False, cull_backfaces: bool = False, z_clip_value: float | None = None, cull_to_frustum: bool = True, clipped_faces_neighbor_idx: Tensor | None = None)

Naive PyTorch implementation of mesh rasterization with the same inputs and outputs as the rasterize_meshes function.

This function is not optimized and is implemented as a comparison for the C++/CUDA implementations.

pytorch3d.renderer.mesh.rasterize_meshes.edge_function(p, v0, v1)

Determines whether a point p is on the right side of a 2D line segment given by the end points v0, v1.

Parameters:

• p – (x, y) Coordinates of a point.

• v0 – (x, y) Coordinates of the end points of the edge.

• v1 – (x, y) Coordinates of the end points of the edge.

Returns:

area –

The signed area of the parallelogram given by the vectors

B = p - v0
A = v1 - v0

      v1 ________
        /\      /
    A  /  \    /
      /    \  /
  v0 /______\/
        B    p

The area can also be interpreted as the cross product A x B. If the sign of the area is positive, the point p is on the right side of the edge. Negative area indicates the point is on the left side of the edge. i.e. for an edge v1 - v0

         v1
        /
       /
-     /    +
     /
    /
  v0

pytorch3d.renderer.mesh.rasterize_meshes.barycentric_coordinates_clip(bary)

Clip negative barycentric coordinates to 0.0 and renormalize so the barycentric coordinates for a point sum to 1. When the blur_radius is greater than 0, a face will still be recorded as overlapping a pixel if the pixel is outside the face. In this case at least one of the barycentric coordinates for the pixel relative to the face will be negative. Clipping will ensure that the texture and z buffer are interpolated correctly.

Parameters:

bary – tuple of barycentric coordinates

Returns

bary_clip: (w0, w1, w2) barycentric coordinates with no negative values.

pytorch3d.renderer.mesh.rasterize_meshes.barycentric_coordinates(p, v0, v1, v2)

Compute the barycentric coordinates of a point relative to a triangle.

Parameters:

• p – Coordinates of a point.

• v0 – Coordinates of the triangle vertices.

• v1 – Coordinates of the triangle vertices.

• v2 – Coordinates of the triangle vertices.

Returns

bary: (w0, w1, w2) barycentric coordinates in the range [0, 1].

pytorch3d.renderer.mesh.rasterize_meshes.point_line_distance(p, v0, v1)

Return minimum distance between line segment (v1 - v0) and point p.

Parameters:

• p – Coordinates of a point.

• v0 – Coordinates of the end points of the line segment.

• v1 – Coordinates of the end points of the line segment.

Returns:

non-square distance to the boundary of the triangle.

Consider the line extending the segment - this can be parameterized as v0 + t (v1 - v0).

First find the projection of point p onto the line. It falls where t = [(p - v0) . (v1 - v0)] / |v1 - v0|^2 where . is the dot product.

The parameter t is clamped from [0, 1] to handle points outside the segment (v1 - v0).

Once the projection of the point on the segment is known, the distance from p to the projection gives the minimum distance to the segment.

pytorch3d.renderer.mesh.rasterize_meshes.point_triangle_distance(p, v0, v1, v2)

Return shortest distance between a point and a triangle.

Parameters:

• p – Coordinates of a point.

• v0 – Coordinates of the three triangle vertices.

• v1 – Coordinates of the three triangle vertices.

• v2 – Coordinates of the three triangle vertices.

Returns:

shortest absolute distance from the point to the triangle.