After the training total loss stabilized, the training total loss decreased again after the model was resumed

[xl4533]

I migrated the trained COCO model to a new data set for training, which was normal at the beginning. I set up 3000 epochs, but after hundreds of epochs total loss was almost unchanged. I stopped training at this time, but when I used the trained model again, total loss dropped again. I want to know why? Will the learning rate drop to a very low level after a certain epoch?

[glenn-jocher]

@xl4533 there is a learning rate scheduler updating the LR depending on the epoch and the hyperparameters you set in train.py. If you change the number of training epochs after starting training this may affect your results.

[glenn-jocher]

See #238

[xl4533]

Thank you very much. I'm trying to adjust these parameters to see performance.

[glenn-jocher]

@xl4533 you're welcome! Feel free to experiment with different hyperparameters and let us know if you have any more questions. Good luck with your adjustments! 🚀