How did you setup the loss for the unet in training?

[patakk]

Hello, I'm trying to train a latent diffusion model using huggingface's pipeline, but the model doesn't converge. I've set it up like a vanilla ddpm and replaced image inputs with latent inputs and I think I'm missing something.
Any tips?

Thanks