Projective Sampling Integrators

[njroussel]

Short Description

This changes Mitsuba's way of handling visibilty-induced discontinuities for differentiable rendering. It removes all the existing code relating reparameterizations and introduces a new architecture for path-space/projective-sampling styled integrators.

Overview

The overall architecture is defined around the ability to generate boundary segments and estimate contributions of paths that have such a boundary segment. Briefly, a boundary segment is a point on a shape with a tangential direction to the surface of the shape at that point.
Refer to the reference API for a full description of the new interface. This text below is meant as gentle introduction to the new architecture.

1. Boundary segments:

We first have a new struct/record to hold boundary segments. This is introduced with a mi.SilhouetteSample3f object which is used for the following:

• Encapsulates a boundary segment (point on surface, direction, etc.)
• The main primitive that is used in integrator to defined bounary paths and passed between methods/functions

2. Shapes:

Every shape has 4 new methods:

• Shape.sample_silhouette():
  • Maps a sample in boundary space (unit cube) to a boundary segement (mi.SilhouetteSample3f)
  • Can handle different types of discontinuities (boundary or interior)
• Shape.invert_silhouette_sample():
  • Maps a boundary segment(mi.SilhouetteSample3f) to a point in boundary space (unit cube)
• Shape.precompute_silhouette():
  • Precomputes a data structure for sampling the silhouette of the shape for a given viewpoint.
  • This is meant to be used for primarily visible discontinuities where the viewpoint is the sensor's pinhole.
• Shape.sample_precomputed_silhouette():
  • Uses the data structure computed in the previous method to sample a mi.SilhouetteSample3f.

The first two methods create a bijective mapping between boundary segments and boundary sample space. Assuming we have some guiding structure in boundary sample space, these two methods will allows us to fill the guiding structure (take a boundary segment and convert it to boundary sample space) or sample from it (take a boundary sample space point and get its corresponding boundary segment).

The last two methods are a specialization of the silhouette sampling problem. For the primary visibility discontinuities, it is more efficient to have specialized methods as the viewpoint for the silhouette is already know (it's the sensor's pinhole).

3. Scene:

The Scene now also has the same two first methods as the shapes (sample_silhouette and invert_silhouette_sample). It will dispatch sampling or inverse sampling requests to all the shapes that are being differentiated in the scene. (We can weigh each shape: it does not have to be a uniform sampling).

4. Integrator:

With this in place we define a new PSIntegrator interface (PS stands for path-space and projective sampling - very convenient). It still has the typical:

• sample()
• render()
• render_forward()
• render_backward()

methods.

But in addition we add the following:

• render_ad:
  This method is more of a intermediary/utiliy method for render_forward and render_backward. Instead of just calling the sample method it will also sample boundary paths. In short, this method will accumulate 3 different types of paths: continuous, directly visible discontinuities, indirectly visible discontinuities. The continuous paths are just normal calls to sample, whereas the two types of discontinuous paths are paths that contain exactly one boundary segment. This method is defined directly in the PSIntegrator interface.
• sample_radiance_difference:
  We need the ability to construct a full light path from a boundary segment. This method samples a path from the boundary segment to an emitter and measures the radiance difference depending on whether or not hat we had hit the boundary segment's shape or if we barely grazed it.
• sample_importance:
  This is the counterpart to the previous method, it samples a path from the boundary segment to an emitter.

The last two methods are not defined in the base class and must be overridden by integrators.

Much of the details of sampling boundary segments, and using projective sampling is held in a separate new file projective.py.

Two new integrators are shipped in this PR: direct_projective and prb_projective.

5. Guiding:

A new file guiding.py was added in this PR. It holds different guiding data structures that should be used by new PSIntegrators.

Important notes/breaking changes

• Reparameterizations were completely removed from the codebase. This has the following "unexpected" consequences:

  • Sensor position's optimizations are no longer possible.
  • The sdfgrid and instance plugins are no longer differentiable
  • Discontinuities introduced by anything else than visibility changes (like a checkerboard texture, a nearest interpolation scheme on a texture, or even non-smooth shape surface normals) are not handled.

  We plan on re-adding these features to the new integrators in a future release.

• The batch sensor cannot be used with the new integrator when primarily visible discontinuities are needed. This will also be fixed in a future release.

• The new interfaces, especially for guiding structures, still have some rough edges. We expect these to still change quite a bit in the near future, especially as we add-in the lost features mentioned above.

[Speierers]

Minor stuff