Argo workflow pvcs not deleted after workflow completion

Question

I have the following volume claim configuration in the workflow template, but still volume claim is not happening after job completion even if one of the phases fails.

  volumeClaimGC:
    strategy: OnWorkflowCompletion
  volumeClaimTemplates: # persistent volumes share data between steps; created and deleted per each run
  - metadata:
      name: root
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Mi

Please suggest if I am missing anything

Answer 1

By looking at the source code for this feature you can see this (line in 1388 this file):

case wfv1.VolumeClaimGCOnSuccess:
        if woc.wf.Status.Phase == wfv1.NodeError || woc.wf.Status.Phase == wfv1.NodeFailed {
            // Skip deleting PVCs to reuse them for retried failed/error workflows.
            // PVCs are automatically deleted when corresponded owner workflows get deleted.
            return nil
        }

When you're workflow ends successfully but with failed pods it skips the deletion, as the comment mentions this was added to allow a retry if needed.

A workaround you can do is set the ARGO_REMOVE_PVC_PROTECTION_FINALIZER value to false, this will cause the behavior you are expecting.