If you have direct control of the terminal on the server,

nvidia-smi

might tell you something interesting. namely the amount of vram being used and which process is using