Do Google TPU VMs get automatically deleted?

Question

I'm currently using a TPU VM that I set up to use my TPU quotas for the TPU Research Cloud program. I tried SSHing into the VM earlier and encountered an unknown error. Upon refreshing the page the VM was deleted.

The TPU that I was using was a v4 TPU, and it's stated that v4 TPU VMs cannot be stopped and can only be deleted. My assumption is that when the resource was reclaimed or something by Google it may have gotten deleted due to this but I'm not 100% sure.

Answer 1

Google Cloud TPU VMs are not automatically deleted unless under specific circumstances. These might be the reasons why your TPU VM was deleted:

• It might have an inactive TPU quota. The TPU Cloud program might have quotas tied to TPU usage. If your VM wasn't used for a significant amount of time, Google might have reclaimed the resources to provide them to active users.
• Check the logs of your project as it's also possible that someone with the necessary permissions accidentally deleted the VM.

To avoid this in the future, check the TPU Cloud program documentation for specific policies on VM inactivity and quota usage. You might also want to contact Google Cloud support for further investigation into the cause of your VM deletion.

Answer 2

You might be using On-demand quota type on your TPU VMs which is different with Reserved quotas. It means that even if you could request and be approved on the increase on your quota limit for Cloud TPU resources, the availability of resources is not guaranteed when you do not have reserved resources.