Consul (KV) is wiping off all data when restarting Kubernetes cluster

Question

I am currently working on Spring micro-service(Eureka Implementation) project. To manage the distributed configuration we are using Consul KV. We are deploying services on Kubernetes cluster.

The issue I am facing that, whenever I restart the cluster for Consul it deletes all the data of KV. I am creating Kubernetes cluster on local with docker image by having Deployment.yaml file. Please refer the below Deployment.yaml file for consul.

apiVersion: v1
kind: Service
metadata:
  name: consul
  labels:
    app: consul
spec:
  clusterIP: None
  ports:
    - port: 8500
      name: consul
  selector:
    app: consul
    
---
    
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
spec:
  serviceName: consul
  replicas: 1
  selector:
    matchLabels:
      app: consul
  template:
    metadata:
      labels:
        app: consul
    spec:
      containers:
      - name: consul
        image: hashicorp/consul:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8500
---

apiVersion: v1
kind: Service
metadata:
  name: consul-lb
  labels:
    app: consul
spec:
  selector:
    app: consul
  type: NodePort
  ports:
  - port: 80
    targetPort: 8500

After some research I found that we can specify the -data-dir location in config, so I have modified StatefulSet kind yaml as below:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
spec:
  serviceName: consul
  replicas: 1
  selector:
    matchLabels:
      app: consul
  template:
    metadata:
      labels:
        app: consul
    spec:
      containers:
      - name: consul
        image: hashicorp/consul:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8500
        args:
            - "agent"
            - "-server"
            - "-data-dir=/home/consul/data"

But after this Consul UI is not getting started, so wanted some help to resolve so it stores the data even after I delete Consul cluster. PS: I tried deploying cluster with helm, and it was persisting the data but I did not know how to make that cluster StatefulSet so I can refer it in other services with static url. Thanks!

Answer 1

Please note that k8s pods are by default ephemeral even if you deploy them as StatefulSet.

StatefulSet is providing you option for pod with define name eg. consul-0 rather that standard consul-<<random string>>. It also keeps track of where to deploy pod in case you have different zones and you need to deploy pod in the same zone as storage.

What is missing in your manifest is volumeMounts and volumeClaimTemplates sections . If you set your data directory to /home/consul/data your manifest should looks similar to this:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
spec:
  serviceName: consul
  replicas: 1
  selector:
    matchLabels:
      app: consul
  template:
    metadata:
      labels:
        app: consul
    spec:
      containers:
      - name: consul
        image: hashicorp/consul:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8500
        args:
          - "agent"
          - "-server"
          - "-data-dir=/home/consul/data"
        volumeMounts:
        - name: consul-data
          mountPath: /home/consul/data
  volumeClaimTemplates:                       # volume claim template will create volume for you so you don't need to define PVC
  - metadata:
      name: consul-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "my-storage-class"    # you can get this with kubectl get sc
      resources:
        requests:
          storage: 1Gi

Regarding you second problem with consul UI I would not help much since I never use consul but I can advise to deploy helm chart once again and check how arguments are passed there.