Unset environment variable when pushing container to Dagster cloud

Question

I'm attempting to push a container to Dagster Cloud with the following agent configuration:

locations:
  - location_name: unit_ingestion
    code_source:
      package_name: unit_ingestion
    build:
      directory: ./orchestration
      registry: "****"
    container_context:
      ecs:
        env_vars:
          - ENV
          - TASK_ROLE_ARN
        server_resources:
          cpu: "256"
          memory: "512"
        run_resources:
          cpu: "256"
          memory: "1024"
        task_role_arn:
          env: TASK_ROLE_ARN

However, this deployment fails with the following message:

Error loading unit_ingestion: {'__typename': 'PythonError', 'message': 'dagster._core.errors.DagsterInvalidConfigError: Errors while parsing ECS container context
    Error 1: Post processing at path root:task_role_arn of original value {\'env\': \'TASK_ROLE_ARN\'} failed:
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "TASK_ROLE_ARN" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/dagster/dagster/_config/post_process.py", line 79, in _post_process
    new_value = context.config_type.post_process(config_value)
  File "/dagster/dagster/_config/source.py", line 40, in post_process
    return str(_ensure_env_variable(cfg))
  File "/dagster/dagster/_config/source.py", line 16, in _ensure_env_variable
    raise PostProcessingError(

', 'stack': ['  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1278, in _reconcile
    new_dagster_servers[to_update_key] = self._start_new_dagster_server(
', '  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1573, in _start_new_dagster_server
    return self._start_new_server_spinup(
', '  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 330, in _start_new_server_spinup
    ).merge(EcsContainerContext.create_from_config(metadata.container_context))
', '  File "/dagster-aws/dagster_aws/ecs/container_context.py", line 380, in create_from_config
    raise DagsterInvalidConfigError(
']}

From this I gather that Dagster Cloud is assuming that the TASK_ROLE_ARN hasn't been defined. However, I have done so:

As far as I can tell, this configuration should be acceptable to Dagster so I'm not sure what's causing this error. Any help would be appreciated.

Answer 1

I discussed this issue with the Dagster team, and they informed me that:

Unfortunately using Dagster Cloud environment variables in that way won’t work, because setting those environment variables in the UI only ensures that the environment variable is set in the task that is launched that runs the code, but that ARN needs to be determined in the agent before the task starts up. Therefore, the task role ARN either needs to be hard-coded to a string, or the environment variable value needs to be set on the agent task in some other way.

There are two possible workarounds for this issue.

Environment-specific YAML Files

The first, and most obvious, solution here is to create a dagster_cloud YAML file for each environment that includes the values specific to that environment. In our case, this meant dagster_cloud_dev.yaml, dagster_cloud_stage.yaml and dagster_cloud_prod.yaml. Each of these files will look like this:

locations:
  - location_name: unit_ingestion
    code_source:
      package_name: unit_ingestion
    build:
      directory: ./orchestration
      registry: "****"
    container_context:
      ecs:
        env_vars:
          - ENV
          - TASK_ROLE_ARN
        server_resources:
          cpu: "256"
          memory: "512"
        run_resources:
          cpu: "256"
          memory: "1024"
        task_role_arn: arn:aws:iam::{account_id}:role/{role_name}

where you replace account_id and role_name with the AWS account ID and role name for the task role ARNs defined in each of your environment workload accounts.

Use a Single Task Role ARN

Fortunately for me, our AWS organization is setup with a shared_dagster account that references resources in our workload accounts. Therefore, I can create a single role that has permissions to assume any of my workload roles. The result here is that I only needed one task role ARN (defined in the shared_dagster account) and therefore only needed one file. In this case, the YAML file looks exactly the same except the ARN references a role that exists within the account where the task is meant to run.