Very Noisy Gradients with Respect to Latent Variables?

[coult099]

Hello,

I am having an issue with forward inverse rendering: when differentiating with respect to latent variables in the scene, the gradient image appears much noisier than otherwise. I have included an example with images below, based on the Cornell box scene. Here, the small ball is red and all walls are white.

Is there a reason why the latent variable gradients (size, position) appear so much noisier than the non-latent variable gradient (color)? Is the gradient with respect to these parameters just so small that noise dominates? Or, is there an issue with my implementation?

A confounding factor: I have repeated the experiment with much higher SPP and while the gradient does get cleaner with more samples (although not as clean as the non-latent gradient), some artifacts/bright pixels start to appear in the gradient image, as seen below. These bright pixels/artifacts also tend to appear in the same locations on successive runs, sometimes even with different random seeds. Thus, while I would otherwise try to average out the gradient over successive runs to decrease the noise, I can't do that here since the artifacts sometimes constructively interfere. Any help is appreciated!

System:

• macOS Sonoma 14.5 (Apple Silicon)
• Python 3.10.8 (running in a virtual environment)
• Mitsuba 3.5.2
• Drjit 0.4.6
• PRB integrator with max_depth=-1 (infinite bounces).
• Sampler: independent

import drjit as dr
import mitsuba as mi

mi.set_variant('llvm_ad_rgb')
scene = mi.load_file('./scenes/cbox/cbox.xml')
params = mi.traverse(scene)

# From Cornell box .xml
spp_val = 1024
ball_size = mi.Float(0.25)
ball_x = mi.Float(0.5)
ball_y = mi.Float(-0.75)
ball_z = mi.Float(-0.2)
pos_add = mi.Float(0)
gradient_param = "size" # "size", "color", or "position"

if gradient_param == "color":
    key = 'red.reflectance.value'
    dr.enable_grad(params[key])
    params.update();
elif gradient_param == "size":
    key = "redsphere.to_world"
    dr.enable_grad(ball_size)
    params[key] = mi.Transform4f(mi.Matrix4f([[ball_size, 0, 0, ball_x + pos_add], [0, ball_size, 0, ball_y + pos_add], [0, 0, ball_size, ball_z + pos_add], [0, 0, 0, 1]]))
    params.update();
elif gradient_param == "position":
    key = 'redsphere.to_world'
    dr.enable_grad(pos_add)
    params[key] = mi.Transform4f(mi.Matrix4f([[ball_size, 0, 0, ball_x + pos_add], [0, ball_size, 0, ball_y + pos_add], [0, 0, ball_size, ball_z + pos_add], [0, 0, 0, 1]]))
    params.update();

if gradient_param == "size":
    dr.forward(ball_size, dr.ADFlag.ClearEdges)
elif gradient_param == "position":
    dr.forward(pos_add, dr.ADFlag.ClearEdges)

# Render image
image = mi.render(scene, params, spp=spp_val)

if gradient_param == "color":
    dr.forward(params[key])
    grad_image = dr.grad(image)
else:
    dr.forward_to(image)
    grad_image = 2 * dr.grad(image)

mi.util.write_bitmap('cbox_original.png', image)
mi.util.write_bitmap('cbox_'+gradient_param+'_grad.png', grad_image)

SPP = 1024:

Original Image	Red Ball Color Gradient	Red Ball Size Gradient	Red Ball Position Gradient

SPP = 16384:

Original Image	Color Gradient	Size Gradient	Position Gradient

[njroussel]

Hi @coult099

Your code looks fine.

I do indeed think that the noise-level difference is also related to the magnitude of the actual derivative. Indirect diffuse light will barely see any change if you scale/move the red ball.

However, I do thing that your gradients for scale and position are wrong. Are you using the prb_projective integator? Unless you're using a *_projective integrator, you won't be accounting for the gradients that arise from changing visibility. (There's a small introduction on this topic in this tutorial.) You can double-check your forward gradients by comparing the to a "brute-force" finite-differences computation, similarly to what is done in this tutorial.

[njroussel]

Aha, that is the issue I believe.

[coult099]

Hello @njroussel ,

I conducted the test using the roughplastic BSDF in green for the larger sphere, and got more similar results, as well as a faster runtime. Original image

Gradients (SPP = 4096):

Auto-diff Gradient
Finite-Difference Gradient
	Red Ball Position	Red Ball Size ( log(abs(image)) )

The automatic differentiation gradient for both position and size contains more noise (and more spurious pixel groups, as in my very first post) than the finite-difference gradient. Do you know why this would be? I generated the auto-diff gradients with 4096 SPP, and the FD gradients with 32 rounds of 4096 SPP each.

[coult099]

I re-ran these experiments with higher SPP for the AD gradients, and got a similar output. In the higher SPP renders, there are spurious pixel groups/artifacts in the AD images, which I would expect not to be the case (indeed, this is the initial issue I had). Because of these artifacts, the FD gradient looks less noisy than the AD gradient, something I would not at all expect to be the case. Do you know what might be causing this behavior? I am using a delta size of 1e-3 for the FD gradients.

Gradients:

AD: 4096 SPP
AD: 16384 SPP
FD: 32 x 4096 SPP
	Color Gradient	Size Gradient	Position Gradient

[coult099]

I've repeated the experiment with different scenes, and always seem to get the same result: color gradients are very clean, but position and size gradients are always noisy with spurious groups of pixels/artifacts as seen above. It is very surprising and counter-intuitive that using finite differences can give a cleaner gradient image than automatic differentiation! I hope I am doing something incorrectly.

[njroussel]

Hi @coult099

color gradients are very clean, but position and size gradients are always noisy

This is expected. Measuring gradients induced by visibility discontinuities is a difficult problem, these only appear in the latter case when scene geometry is changing.

It is very surprising and counter-intuitive that using finite differences can give a cleaner gradient image than automatic differentiation!

Not really. We're not just running AD, these are full on reverse Monte Carlo simulations, so you can expect the same noise characteristics as traditional forward rendering. In addition, with visibility changes, the space of paths that produce forward gradients is extremely sparse.

I obviously can't guarantee that your code is bug-free, but from the results you've shown I'd say that the results are as expected. Keep in mind that in order to run an optimization you don't need perfectly clean gradients, low gradient spp renderings work fine in most cases.