How to obtain the gradient loss with respect to parameters?

[monlabiss]

Hello,

I’d like to obtain the gradient loss with respect to parameters(params[PARAMETERS]).

I obtained the dL/dI value with the below code.

dr.enable_grad(params[PARAMETERS])
params.update()
img = mi.render(scene, params=params, seed=NUM_SEED, spp=NUM)
# loss function
loss = dr.sum(dr.sqr(img - refImg))
# dL/dI
dr.set_grad(loss, 1) # Initialize gradient
dr.backward_to(img)
dL_dI = dr.grad(img)

And I would like to obtain the dL/dI ⊙ Σ_i dI/dθ_i value.

When I tried like this, it results dL/dI ⋅ dI/dθ value.

# dL/dI * dI/dθ
dr.set_grad(img, dL_dI) # Modify gradient of the image
dr.backward_to(params[PARAMETERS]) # Propagate from the image to parameters  
dI_dθ = dr.grad(img)

How would I go about doing this value(dL/dI ⊙ Σ_i dI/dθ_i) ?
The resolution of value is same with dL/dI and image I.

Thanks!

[njroussel]

Hi @monlabiss

I think you could compute Σ_i dI/dθ_i as the forward-mode derivative w.r.t to all parameters. We have a tutorial on this: https://mitsuba.readthedocs.io/en/latest/src/inverse_rendering/forward_inverse_rendering.html (in the tutorial we only do it w.r.t. to a single parameter IIRC).

Or am I misunderstanding?

[monlabiss]

Thanks for your friendly descriptions! You understood correctly, but I think that $\sum_i \frac{\partial I}{\partial θ_i}$ could not be obtained by the forward-mode method for optimization.

[ziyi-zhang]

Hi,

$dL/dI$ measures the sensitivity between the loss and each pixel.
$\sum dI/d\theta$ measures the sensitivity between a pixel and scene parameters. The use of summation is like "total derivative" -- $dL=\partial_{\theta1}L+\partial_{\theta2}L+\partial_{\theta3}L+\cdots$.
So the multiplication of these two quantities, by the chain rule, would be the sensitivity between the loss and scene parameters. Why is the "resolution of the value the same with dL/dI and image I"?

[njroussel]

"resolution of the value the same with dL/dI and image I"?

I think what they meant by this is that they have the same shape: the value is a tensor of shape width x height x channels

[monlabiss]

To optimize the loss with $\frac{\partial L}{\partial I} ⊙ \sum_i \frac{\partial I}{\partial θ_i}$, not with $\frac{\partial L}{\partial \theta}$, I tried like this.

# Optimization
for it in range(iteration_count):
    dr.enable_grad(params[PARAMETERS])
    params.update()
    img = mi.render(scene, params=params, seed=NUM_SEED, spp=NUM)
    loss = dr.sum(dr.sqr(img - refImg))

    # dL/dI
    dr.set_grad(loss, 1) 
    dr.backward_to(img) 
    dL_dI = dr.grad(img)

    # dL/dI ⊙ Σ_i dI/dθ_i
    dr.set_grad(img, dL_dI) 
    dr.forward(params[PARAMETERS]) 

    opt.step()
    opt[PARAMETERS] = dr.clamp(opt[PARAMETERS], 0.0, 1.0)
    params.update(opt)
    err_ref = dr.mean(dr.sqr(opt[PARAMETERS] - param_ref))
    print(f"Iteration {it:02d}: parameter error = {err_ref[0]:.6f}", end="\r")

The parameter error rate more decreases better than when I use with $\frac{\partial L}{\partial \theta}$ (https://mitsuba.readthedocs.io/en/latest/src/inverse_rendering/gradient_based_opt.html).

But when I visualize the final images, It's almost unchanged from the initial image.
I'm not sure if I implemented the code poorly or if the forward() function is not meant to be an optimization.

[ziyi-zhang]

Hi,

dr.forward(params[PARAMETERS]) calls forward_from() which sets the grad of params[PARAMETERS] and starts a forward propagation that overrides the stored dL_dI. In opt.step(), the optimizer fetches the grad of params, which is one here due to forward initialization, for the optimization. This seems to be a wrong use of the ad system.

[monlabiss]

Thanks for your help! I thought that was what you were describing, but now that you've explained it in more detail, it's clearer.

So, go back to the beginning, is there any method to measures the sensitivity between a pixel and scene parameters($\sum_i \frac{\partial I}{\partial θ_i}$) for optimization? I cannot find the way except forward() function.