Accessing output value from a data source in a remote state file

Question

Background - I have been asked to take 5 Azure SQL servers off the public network and place them onto private endpoints. The SQL Servers are used by an application in a container within Kubernetes.

I am confident the network is in place as intended. The problem is that the application is still connecting to the old FQDNs not the FQDNs of the private endpoints. My understanding is that the hostname used by the application is generated by the Kubernetes secret resource below and this comes from an output in another project via a remote state file.

I am creating the following DNS records in one state file and have set and output like so:

variable "dns_records" {
default = {
"btarget"  = "172.16.102.4"
"public-0" = "172.16.102.5"
"public-1" = "172.16.102.6"
"public-2" = "172.16.102.7"
  }
}

resource "azurerm_private_dns_a_record" "dns-records" {
for_each = var.dns_records
name = each.key
records = [each.value]
resource_group_name = "${var.application_name}-${var.environment_tag}-rg"
ttl = 3600
zone_name = azurerm_private_dns_zone.dns-zone.name
}

output "sql_admin_fqdns" {
description = "FQDNs of all other SQL Servers"
sensitive = false
value = { for entry in azurerm_private_dns_a_record.dns-records : entry.name => entry.fqdn }
}

I am then calling the FQDN value from another project with a different state file like this:

data "terraform_remote_state" "staged_azure" {
  backend = "azurerm"
  config = {
    subscription_id      = var.tf_state_subscription_id
    resource_group_name  = var.tf_state_rg_name
    storage_account_name = var.tf_state_sa_name
    container_name       = var.tf_state_container_name
    key                  = var.tf_state_staged_azure_key
    access_key           = var.tf_state_access_key
  }
}

resource "kubernetes_secret" "cms_database" {
metadata {
name = join("-", [var.application_name, "database", each.key])
namespace = kubernetes_namespace.magnolia.id
  }
data = {
username = each.value
password = data.terraform_remote_state.staged_azure.outputs.sql_admin_passwords[each.key]
hostname = data.terraform_remote_state.staged_azure.outputs.sql_admin_fqdns[each.key].fqdn
schema = data.terraform_remote_state.staged_azure.outputs.sql_admin_schemas[each.key]
port = 1433
resource-group = data.terraform_remote_state.staged_azure.outputs.sql_admin_rg
  }
for_each = data.terraform_remote_state.staged_azure.outputs.sql_admin_usernames
}

When running the plan on the second project it is not getting the correct value from the output in the other project:

Error: Unsupported attribute

│

on tomcat.tf line 59, in resource "kubernetes_secret" "bca_cms_database":

59: hostname = data.terraform_remote_***.staged_azure.outputs.sql_admin_fqdns[each.key].fqdn

│ ├────────────────

data.terraform_remote_***.staged_azure.outputs.sql_admin_fqdns is object with 4 attributes

each.key is "public-2"

│

Can't access attributes on a primitive-typed value (string).

I've tried so many different approaches and the data source never pulls through the correct value even though I can see the correct value is being output from the source project

I am expecting the Kubernetes Secret resource to grab the FQDNS of the private DNS records from the output in the remote state project and inject them into the container.

I posted on the Reddit Terraform sub and they said that hashicorp recommends avoiding referencing remote states and that I should access the data from Azure directly using an Azure Data Source. I've not been able to figure out how to do this yet.

The Kubernetes secret resource was working fine when it was pulling the FQDNs from the SQL servers on the public network.

Answer 1

I tried to Accessing output value from a source in a remote state file and I was able to provision the requirement successfully.

You got an error because you tried to access .fqdn on a string value, instead of an object with a fqdn attribute. This is because your output sql_admin_fqdns is a map of FQDNs, and when you access a map element, you get the FQDN string, not an object.

The error can be resolved by changing the way you refer to the FQDNs in your kubernetes_secret resource. You can omit the .fqdn attribute when accessing the values of the sql_admin_fqdns map, as they are already strings.

Although using remote state references is common, it can lead to tight coupling between Terraform projects. As suggested, using Azure data sources directly in your Terraform configuration to fetch the necessary information about the SQL servers can be a more decoupled approach.

resource "kubernetes_secret" "cms_database" {
 ....
    hostname        = data.terraform_remote_state.staged_azure.outputs.sql_admin_fqdns[each.key] 
 ....

My terraform configuration:

provider "azurerm" {
  features {}
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

variable "application_name" {
  default = "demoappvk"
}

variable "environment_tag" {
  default = "dev"
}

resource "azurerm_resource_group" "example" {
  name     = "${var.application_name}-${var.environment_tag}-rg"
  location = "West Europe"
}

resource "azurerm_virtual_network" "example_vnet" {
  name                = "${var.application_name}-${var.environment_tag}-vnet"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_subnet" "example_subnet" {
  name                 = "${var.application_name}-${var.environment_tag}-subnet"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example_vnet.name
  address_prefixes     = ["10.0.1.0/24"]
}


resource "azurerm_mssql_server" "example" {
  name                         = "${var.application_name}-${var.environment_tag}-sqlserver"
  resource_group_name          = azurerm_resource_group.example.name
  location                     = azurerm_resource_group.example.location
  version                      = "12.0"
  administrator_login          = "sqladmin"
  administrator_login_password = "H@Sh1CoR3!"
  minimum_tls_version          = "1.2"
}

resource "azurerm_private_dns_zone" "example" {
  name                = "privatelink.database.windows.net"
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_private_endpoint" "example" {
  name                = "${var.application_name}-${var.environment_tag}-sqlendpoint"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  subnet_id           = azurerm_subnet.example_subnet.id

  private_service_connection {
    name                           = "${var.application_name}-${var.environment_tag}-sqlserver-privateserviceconnection"
    private_connection_resource_id = azurerm_mssql_server.example.id
    is_manual_connection           = false
    subresource_names              = ["sqlServer"]
  }

  private_dns_zone_group {
    name                 = "demovk-dns-zone-group"
    private_dns_zone_ids = [azurerm_private_dns_zone.example.id]
  }
}


# Adjusted to reference the DNS zone linked with the private endpoint
resource "azurerm_private_dns_a_record" "example" {
  for_each = toset(["sql-${var.application_name}-${var.environment_tag}"])

  name                = each.key
  zone_name           = azurerm_private_dns_zone.example.name
  resource_group_name = azurerm_resource_group.example.name
  ttl                 = 3600
  records             = [azurerm_private_endpoint.example.private_service_connection[0].private_ip_address]
}

output "sql_server_fqdn" {
  value = azurerm_mssql_server.example.fully_qualified_domain_name
}

resource "kubernetes_secret" "sql_server" {
  metadata {
    name      = "sql-server-secret"
    namespace = "default"
  }

  data = {
    username = "sqladmin"
    password = "H@Sh1CoR3!"
    hostname = base64encode(azurerm_mssql_server.example.fully_qualified_domain_name)
    port     = base64encode("1433")
  }
}

output "sql_server_fqdn_from_k8s" {
  value = base64decode(kubernetes_secret.sql_server.data["hostname"])
  sensitive = true
}

Output: