AKS (version 1.25.5) Nginx Ingress Controller : External DNS is not working

Question

I got the following setup:

Azure Kubernetes -> Ingress-Nginx-Controller (uses Azure Load-Balancer) -> External DNS

I am exposing the Ingress-Nginx-Controller via an Ingress, backed by the Azure Load Balancer Controller using public IP.

I have used the following terraform code to setup the Azure Kubernetes

resource "azurerm_virtual_network" "test" {
  name                = var.virtual_network_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  address_space       = [var.virtual_network_address_prefix]

  subnet {
    name           = var.aks_subnet_name
    address_prefix = var.aks_subnet_address_prefix
  }

  tags = var.tags
}

data "azurerm_subnet" "kubesubnet" {
  name                 = var.aks_subnet_name
  virtual_network_name = azurerm_virtual_network.test.name
  resource_group_name  = azurerm_resource_group.rg.name
  depends_on           = [azurerm_virtual_network.test]
}

resource "azurerm_user_assigned_identity" "aks_external_dns_managed_id" {
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location

  name = "aks-external-dns-managed-id"

  depends_on = [
    azurerm_resource_group.rg
  ]
}

resource "azurerm_kubernetes_cluster" "k8s" {
  name       = var.aks_name
  location   = azurerm_resource_group.rg.location
  dns_prefix = var.aks_dns_prefix

  resource_group_name = azurerm_resource_group.rg.name

  http_application_routing_enabled = false

  linux_profile {
    admin_username = var.vm_user_name

    ssh_key {
      key_data = file(var.public_ssh_key_path)
    }
  }

  default_node_pool {
    name            = "agentpool"
    node_count      = var.aks_agent_count
    vm_size         = var.aks_agent_vm_size
    os_disk_size_gb = var.aks_agent_os_disk_size
    vnet_subnet_id  = data.azurerm_subnet.kubesubnet.id
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin     = "azure"
    dns_service_ip     = var.aks_dns_service_ip
    docker_bridge_cidr = var.aks_docker_bridge_cidr
    service_cidr       = var.aks_service_cidr
  }

  # Enabled the cluster configuration to the Azure kubernets with RBAC
  azure_active_directory_role_based_access_control { 
    managed                     = var.azure_active_directory_role_based_access_control_managed
    admin_group_object_ids      = var.active_directory_role_based_access_control_admin_group_object_ids
    azure_rbac_enabled          = var.azure_rbac_enabled
  }

  oms_agent {
    log_analytics_workspace_id  = module.log_analytics_workspace[0].id
  }

  timeouts {
    create = "20m"
    delete = "20m"
  }  

  depends_on = [data.azurerm_subnet.kubesubnet,module.log_analytics_workspace]
  tags       = var.tags
}

resource "azurerm_role_assignment" "ra5" {
  scope                = data.azurerm_subnet.kubesubnet.id
  role_definition_name = "Network Contributor"
  principal_id         = azurerm_user_assigned_identity.aks_external_dns_managed_id.principal_id
  depends_on           = [azurerm_kubernetes_cluster.k8s,data.azurerm_subnet.kubesubnet]
}

resource "azurerm_role_assignment" "ra6" {
  scope                = module.container_registry.0.id
  role_definition_name = "AcrPush"
  principal_id         = azurerm_user_assigned_identity.aks_external_dns_managed_id.principal_id
  depends_on           = [azurerm_kubernetes_cluster.k8s,module.container_registry]
}

resource "azurerm_role_assignment" "ra7" {
  scope                = azurerm_resource_group.rg.id
  role_definition_name = "Reader"
  principal_id         = azurerm_user_assigned_identity.aks_external_dns_managed_id.principal_id
  depends_on           = [azurerm_kubernetes_cluster.k8s]
}

resource "azurerm_role_assignment" "ra8" {
  scope                = azurerm_dns_zone.dns_zone.id
  role_definition_name = "Contributor"
  principal_id         = azurerm_user_assigned_identity.aks_external_dns_managed_id.principal_id
  depends_on           = [azurerm_kubernetes_cluster.k8s,]
}

and following details are displayed

output "subscriptionId" {
  value = "${data.azurerm_client_config.current.subscription_id}"
}

output "resource_group_name" {
  value = azurerm_resource_group.rg.name
}    

output "aks_external_dns_managed_id" {
  value = azurerm_user_assigned_identity.aks_external_dns_managed_id.client_id
}

and added the user managed identity as a user assigned identity in the AKS VMSS

and executed the following commands

Install Nginx Ingress Controller

kubectl create namespace ingress-basic
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx  --namespace ingress-basic 

Update the details in the azure.json

{
  "tenantId": "tenant ID GUID",
  "subscriptionId": "subscription ID GUID",
  "resourceGroup": "rg-blessed-baboon", 
  "useManagedIdentityExtension": true,
  "userAssignedIdentityID": "1a8e6a4b-2d84-447d-8be5-efd1e95e2dda"
}

and created the secret

kubectl create secret generic azure-config-file --from-file=azure.json

and applied the

kubectl apply -f external-dns.yml

external-dns.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods", "nodes"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"] 
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: k8s.gcr.io/external-dns/external-dns:v0.13.2
        args:
        - --source=service
        - --source=ingress
        #- --domain-filter=example.com # (optional) limit to only example.com domains; change to match the zone created above.
        - --provider=azure
        #- --azure-resource-group=externaldns # (optional) use the DNS zones from the specific resource group
        volumeMounts:
        - name: azure-config-file
          mountPath: /etc/kubernetes
          readOnly: true
      volumes:
      - name: azure-config-file
        secret:
          secretName: azure-config-file

and Deployed the application

kubectl apply -f .\deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1-nginx-deployment
  labels:
    app: app1-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app1-nginx
  template:
    metadata:
      labels:
        app: app1-nginx
    spec:
      containers:
        - name: app1-nginx
          image: stacksimplify/kube-nginxapp1:1.0.0
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: app1-nginx-clusterip-service
  labels:
    app: app1-nginx
spec:
  type: ClusterIP
  selector:
    app: app1-nginx
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginxapp1-ingress-service
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: eapp1.eat-eggs.ca
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app1-nginx-clusterip-service
                port: 
                  number: 80
---

It got deployed and I can access the application using cluster IP

DNS did not get updated even after 20 mins after the deployment.

External DNS throws the following error

kubectl logs -f external-dns-765645d59d-d2bsp

Logs:

time="2023-02-11T02:28:30Z" level=error msg="azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions//resourceGroups/rg-blessed-baboon/providers/Microsoft.Network/dnsZones?api-version=2018-05-01: StatusCode=404 -- Original Error: adal: Refresh request failed. Status Code = '404'. Response body: clientID in request: 1a8e##### REDACTED #####2dda, getting assigned identities for pod default/external-dns-765645d59d-d2bsp in CREATED state failed after 16 attempts, retry duration [5]s, error: . Check MIC pod logs for identity assignment errors\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=1a8e6a4b-2d84-447d-8be5-efd1e95e2dda&resource=https%3A%2F%2Fmanagement.core.windows.net%2F"

What am I missing?

Update#1

I have replaced the external DNS file with the below

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
  - apiGroups: [""]
    resources: ["services","endpoints","pods", "nodes"]
    verbs: ["get","watch","list"]
  - apiGroups: ["extensions","networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get","watch","list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
  - kind: ServiceAccount
    name: external-dns
    namespace: ingress-basic
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
        - name: external-dns
          image: registry.k8s.io/external-dns/external-dns:v0.13.2
          args:
            - --source=service
            - --source=ingress
            #- --domain-filter=example.com # (optional) limit to only example.com domains; change to match the zone created above.
            - --provider=azure
            #- --azure-resource-group=MyDnsResourceGroup # (optional) use the DNS zones from the tutorial's resource group
            #- --txt-prefix=externaldns-
          volumeMounts:
            - name: azure-config-file
              mountPath: /etc/kubernetes
              readOnly: true
      volumes:
        - name: azure-config-file
          secret:
            secretName: azure-config-file

and tried with both Client ID & Object (principal) ID. Issue is still persist.

Answer 1

you didnt attach managed identity created for external dns to aks nodes. normally you have aad pod identity controller handling that (and security around that), but you can simply go to the aks managed group click on the vmss and assign managed identity for external dns to the nodepool