Coqui TTS (using tortoise v2) does not use GPU despite Cuda Torch enabled

Question

I cannot get to use the GPU when enabling the GPU for Coqui TTS. When checking if torch cuda is compiled, I run:

torch.cuda.is_available()

It returns

True

Here is what I tried

class TTS_custom:
    def __init__(self, tortoise=False) -> None:
        self.tortoise = tortoise
        if tortoise is True:
            cuda_available = torch.cuda.is_available()
            self.tts = TTS("tts_models/en/multi-dataset/tortoise-v2")
            self.tts.to('cuda')

def test():
    tt = TTS_custom(True)
    print(torch.cuda.is_available())
    tt.read("I'll be executed for this, but I don't care. I need to warn people.")


test()

I installed and compiled PyTorch with

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121