How much cuda memory does dotav15 need?

[kouyuanbo]

I try to train rhino with the config 'rhino-4scale_r50_2xb2-12e_dotav15.py' on RTX 4090(24 G)×2,batch size is 2.But the following error message has appeared:

RuntimeError: CUDA out of memory. Tried to allocate 1.03 GiB (GPU 0; 23.65 GiB total capacity; 19.63 GiB already allocated; 735.00 MiB free; 20.85 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[nijkah]

@kouyuanbo I am sorry for the late response. Can you recheck your settings?
In my case, the same setting 'rhino-4scale_r50_2xb2-12e_dotav15.py' with batch_size=2 requires 8300~1570 MB.
Note that 'the denoising training' from DINO can result in higher memory usage when there are a lot of GT objects.