RunTimeError: cuDNN version incompatibility: PyTorch was compiled against (8, 3, 2) but found runtime version (8, 2, 0)

Question

I'm working in GoogleCloud and I have create a virtual machine with the following specs:

• Machine: a2-highgpu-1g
• CPU platform: Intel Cascade Lake
• GPU: 1 x NVIDIA A100 40GB

I use this machine to train and test different RNN models and it was working fine till last friday and today suddenly this error shows:

RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (8, 3, 2) but found runtime version (8, 2, 0). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN.Looks like your LD_LIBRARY_PATH contains incompatible version of cudnnPlease either remove it from the path or install cudnn (8, 3, 2)

I have tried many things:

1. changing the LD_LIBRARY_PATH as shown here: https://againstentropy.github.io/blog/2023/cuDNN-incompatibility/
2. changing pytorch version but the GPU can not work with a Pytorch version that supports cuDNN (8.2.0)
3. tried, without success, to reinstall cuDNN

I have finished all the ideas. Can someone help me in someway? Thanks