Does a mi.TensorXf need to be registered to the scene to become a target for differentiation?

[mark-at-hypothetic]

Trying to optimize a dr.jit tensor

base_color_val = mi.TensorXf(np.random.rand(1,1,3))

That I apply to a part of the BaseColor texture using a mask:

params[keys.tex_base_color] = (1-mask) * params[keys.tex_base_color] + mask * base_color_val

So base_color_val is not part of the scene, just used as an intermediary tensor to modify the texture (that is part of the scene).
When doing a dr.backward(loss) on the render, I get the error:

TypeError: backward_from(): the argument does not depend on the input variable(s) being differentiated.

Apparently my custom operation was not registered to the graph? What extra step do I need to do add it to the scene to add this to the graph? Maybe have to write a plugin for it?
Also open to alternative methods to do the same thing. Thanks

EDIT:
I thought that adding a tensor to the optimizer makes it differentiable, and it was confirmed I thought by the 'isDiff' == True property of my tensor. But setting dr.enable_grad(base_color_val) actually resolves the error. However, now, python exists without error at opt.step()

[mark-at-hypothetic]

I've successfully optimized textures (that are part of the scene) before with my code.

[njroussel]

Hi @mark-at-hypothetic

Are you using a built-in optimizer? If not, did you enable gradients with dr.enable_grad(base_color_val) ?

In short, IsDiff is a check to manage if the type is differentiable. However, by default it does not mean that it will be differentiated. The variable must manually be set to have gradient tracking enabled with dr.enable_grad(). Any variable resulting of an operation on variables with gradient tracking enabled will then in turn also have gradient tracking enabled. This is what should create the chain/graph from your tensor to the loss.

If you're still stuck, a longer reproducer would help us pinpoint the issue.

[mark-at-hypothetic]

Hi, thanks for getting back! I just updated my initial question with the dr.enable_grad(), which indeed did add it to the graph. However, now with the new error it'd help to know if any tensor that needs to get optimized needs to be a part of the scene, so within the parameters from params = mi.traverse(scene)

Indeed using the built-in optimizer:

opt = mi.ad.Adam(lr=0.05) # 0.05
opt[keys.val_base_color] = base_color_val
params.update(opt)

[njroussel]

No, that is not a requirement. In the tutorials, we refer to such cases as "latent variables" (relevant tutorial section).

[mark-at-hypothetic]

Thanks, that's very useful! I'll try see if I can resolve it. If I can't find an oversight I'll come back here with some more context