v1.86.0

Add `--process-templates` flag to `atmos describe stacks` and `atmos describe component` commands. Update docs @aknysh (#669)

## what

• Add logging to the template functions atmos.Component and atmos.GomplateDatasource
• Add --process-templates flag to atmos describe stacks and atmos describe component commands
• Update docs

why

• When the environment variable ATMOS_LOGS_LEVEL is set to Trace, the template functions atmos.Component and atmos.GomplateDatasource will log the execution flow and the results of template evaluation - useful for debugging

   ATMOS_LOGS_LEVEL=Trace atmos terraform plan <component> -s <stack>

• Enable/disable processing of Go templates in Atmos stacks manifests when executing the commands

• For atmos describe component <component> -s <stack> command, use the --process-templates flag to see the component configuration before and after the templates are processed. If the flag is not provided, it's set to true by default

# Process `Go` templates in stack manifests and show the final values
atmos describe component <component> -s <stack>

# Process `Go` templates in stack manifests and show the final values
atmos describe component <component> -s <stack> --process-templates=true

# Do not process `Go` templates in stack manifests and show the template tokens in the output
atmos describe component <component> -s <stack> --process-templates=false

• For atmos describe stacks command, use the --process-templates flag to see the stack configurations before and after the templates are processed. If the flag is not provided, it's set to true by default

  # Process `Go` templates in stack manifests and show the final values
  atmos describe stacks
  
  # Process `Go` templates in stack manifests and show the final values
  atmos describe stacks --process-templates=true
  
  # Do not process `Go` templates in stack manifests and show the template tokens in the output
  atmos describe stacks --process-templates=false

  The command atmos describe stacks --process-templates=false can also be used in Atmos custom commands that just list Atmos stacks does not require template processing. This will significantly speed up the custom command execution. For example, the custom command atmos list stacks just outputs the top-level stack names and might not require template processing. It will execute much faster if implemented like this (using the --process-templates=false flag with the atmos describe stacks command :

    - name: list
      commands:
        - name: stacks
          description: |
            List all Atmos stacks.
          steps:
            - >
              atmos describe stacks --process-templates=false --sections none | grep -e "^\S" | sed s/://g

fix: Atmos Affected GitHub Action Documentation @milldr (#661)

## what - Update affected-stacks job outputs and matrix integration

why

• The affected step was missed when the plan example was updated

references

• closes https://github.com/orgs/cloudposse/discussions/18

Updated Documentation for GHA Versions @milldr (#657)

## what - Update documentation for Atmos GitHub Action version management

why

• New major releases for both actions

references

1. https://github.com/cloudposse/github-action-atmos-terraform-plan/releases/tag/v3.0.0
2. https://github.com/cloudposse/github-action-atmos-terraform-drift-detection/releases/tag/v2.0.0

v1.85.0

Update `atmos describe affected` and `atmos terraform` commands @aknysh (#654)

## what

• Update atmos describe affected command
• Update atmos terraform command
• Allow Gomplate, Sprig and Atmos template functions in imports in Atmos stack manifests

why

• The atmos describe affected command had an issue with calculating the included_in_dependents field for all combination of the affected components with their dependencies. Now it's correctly calculated for all affected

• In atmos describe affected command, if the Git config core.untrackedCache is enabled, it breaks the command execution. We disable this option if it is set

• The atmos terraform command now respects the TF_WORKSPACE environment variable. If the environment variable is set by the caller, Atmos will not calculate and set a Terraform workspace for the component in the stack, but instead will let Terraform use the workspace provided in the TF_WORKSPACE environment variable

• Allow Gomplate, Sprig and Atmos template functions in imports in Atmos stack manifests. All functions are allowed now in Atmos stacks manifests and in the import templates

v1.84.0

Add Atmos Pro integration to `atmos.yaml`. Add caching to `atmos.Component` template function. Implement `atmos.GomplateDatasource` template function @aknysh (#647)

## what

• Add Atmos Pro integration to atmos.yaml
• Add caching to atmos.Component template function
• Implement atmos.GomplateDatasource template function
• Update docs
  • atmos.Component
  • atmos.GomplateDatasource

why

• Add Atmos Pro integration to atmos.yaml. This is in addition to the functionality added in Add --upload flag to atmos describe affected command. If the Atmos Pro configuration is present in the integrations.pro section in atmos.yaml, it will be added in the config section when executing the atmos describe affected --upload=true command for further processing on the server

  {
       "base_sha": "6746ba4df9e87690c33297fe740011e5ccefc1f9",
       "head_sha": "5360d911d9bac669095eee1ca1888c3ef5291084",
       "owner": "cloudposse",
       "repo": "atmos",
       "config": {
         "timeout": 3,
          "events": [
             "pull_request": [
                {
                  "on": ["open", "synchronize", "reopen"],
                  "workflow": "atmos-plan.yml",
                  "dispatch_only_top_level_stacks": true
                },               
                {
                  "on": ["merged"],
                  "workflow": "atmos-apply.yaml",
                },               
              ],
              "release": [
              ]
           ]
       }
       "stacks": [
          {
            "component": "vpc",
            "component_type": "terraform",
            "component_path": "components/terraform/vpc",
            "stack": "plat-ue2-dev",
            "stack_slug": "plat-ue2-dev-vpc",
            "affected": "stack.vars",
            "included_in_dependents": false,
            "dependents": []
          }
      ]
   }

• Add caching to atmos.Component template function

  Atmos caches (in memory) the results of atmos.Component template function execution. If you call the function for the same component in a stack more than once, the first call will produce the result and cache it, and all the consecutive calls will just use the cached data. This is useful when you use the atmos.Component function for the same component in a stack in multiple places in Atmos stack manifests. It will speed up the function execution and stack processing.

  For example:

  components:
    terraform:
      test2:
        vars:
          tags:
            test: '{{ (atmos.Component "test" .stack).outputs.id }}'
            test2: '{{ (atmos.Component "test" .stack).outputs.id }}'
            test3: '{{ (atmos.Component "test" .stack).outputs.id }}'

  In the example, the test2 Atmos component uses the outputs (remote state) of the test Atmos component from the same stack. The template function {{ atmos.Component "test" .stack }} is executed three times (once for each tag).

  After the first execution, Atmos caches the result in memory (all the component sections, including the outputs), and reuses it in the next two calls to the function. The caching makes the stack processing about three times faster in this particular example. In a production environment where many components are used, the speedup can be even more significant.

• Implement atmos.GomplateDatasource template function

  The atmos.GomplateDatasource template function wraps the Gomplate Datasources and caches the results, allowing executing the same datasource many times without calling the external endpoint multiple times. It speeds up the datasource execution and stack processing, and can eliminate other issues with calling an external endpoint, e.g. timeouts and rate limiting.

  Usage

    {{ (atmos.GomplateDatasource "<alias>").<attribute> }}

  Caching the result of atmos.GomplateDatasource function

  Atmos caches (in memory) the results of atmos.GomplateDatasource template function execution. If you execute the function for the same datasource alias more than once, the first execution will call the external endpoint, produce the result and cache it. All the consecutive calls will just use the cached data. This is useful when you use the atmos.GomplateDatasource function for the same datasource alias in multiple places in Atmos stack manifests. It will speed up the function execution and stack processing.

  For example:

  settings:
    templates:
      settings:
        gomplate:
          timeout: 5
          datasources:
            ip:
              url: "https://api.ipify.org?format=json"
              headers:
                accept:
                  - "application/json"
  components:
    terraform:
      test:
        vars:
          tags:
            test1: '{{ (datasource "ip").ip }}'
            test2: '{{ (atmos.GomplateDatasource "ip").ip }}'
            test3: '{{ (atmos.GomplateDatasource "ip").ip }}'
            test4: '{{ (atmos.GomplateDatasource "ip").ip }}'

  In the example, we define a gomplate datasource ip and specify an external endpoint in the url parameter.

  We use the Gomplate datasource function in the tag test1, and the atmos.GomplateDatasource wrapper for the same datasource alias ip in the other tags. The atmos.GomplateDatasource wrapper will call the same external endpoint, but will cache the result and reuse it between the datasource invocations.

  When processing the component test from the above example, Atmos does the following:

  • Executes the {{ (datasource "ip").ip }} template. It calls the external endpoint using the HTTP protocol and assign the ip attribute from the result to the tag test1

  • Executes the {{ (atmos.GomplateDatasource "ip").ip }} template. It calls the external endpoint again, caches the result in memory, and assigns the ip attribute from the result to the tag test2

  • Executes the {{ (atmos.GomplateDatasource "ip").ip }} two more times for the tags test3 and test4. It detects that the result for the same datasource alias ip is already presend in the memory cache and reuses it without calling the external endpoint two more times

  The datasource result caching makes the stack processing much faster and significantly reduces the load on external endpoints, preventing such issues as timeouts and rate limiting.

v1.83.1

Auto completion for zsh devcontainer @osterman (#639)

## what - Add autocompletion while you type

why

• Better DX (less typing)

demo

Add docker image @osterman (#627)

## what - Add a docker image for Atmos - Bundle typcal dependencies - Multi-architecture build for ARM64 and AMD64

why

• Make it easier to get up and running with Atmos

Introduce License Check @osterman (#638)

## what

Check for approved licenses

why

Avoid accidentally introducing code that is non-permissively licensed

Fix Codespace url @osterman (#637)

## what

• Update to Codespace URL for main branch

why

• It was pointed to an older reorg branch

Reorganize Documentation For a Better Learning Journey @osterman (#612)

## what - Rename top menu items to "Learn" and "Reference" - Move community to the left, remove discussions and add contributing - Introduce sidebar sections, so that content is further left-justified - Consolidate Terraform content into one section to tell a better story about how to use Terraform with Atmos.

why

• Reorganize Atmos Docs to better help developers on their learning journey

Note

• Dependency Review action is failing. Might be something new with the action.
  actions/dependency-review-action#786

🚀 Enhancements

Don't copy unix sockets in `atmos describe affected` command @aknysh (#640)

## what

• Don't copy unix sockets when executing atmos describe affected command
• Fix some links (left over after renaming examples/quick-start to examples/quick-start-advanced)

why

• Sockets are not regular files, and if someone uses tools like git-fsmonito and executes atmos describe affected command, the following error will be thrown:

open .git/fsmonitor--daemon.ipc: operation not supported on socket

v1.83.0

Update `atmos describe affected` command @aknysh (#635)

## what

• Update atmos describe affected command
• If --upload=true flag is passed, include dependents for all dependents (even an empty list), and include the settings section for all the dependent components

why

• Make the API schema consistent on the server that processes the result of atmos describe affected --upload=true command

v1.82.0

Add `--upload` flag to `atmos describe affected` command @aknysh (#631)

## what

• Add --upload flag to atmos describe affected command
• Update docs
  • https://atmos.tools/cli/commands/describe/affected/

why

If the --upload=true command-line flag is passed, Atmos will upload the affected components and stacks to a specified HTTP endpoint.

The endpoint can process the affected components and their dependencies in a CI/CD pipeline (e.g. execute terraform apply on all the affected components in the stacks and all the dependencies).

Atmos will perform an HTTP POST request to the URL ${ATMOS_PRO_BASE_URL}/${ATMOS_PRO_ENDPOINT}, where the base URL is defined by the ATMOS_PRO_BASE_URL environment variable, and the URL path is defined by the ATMOS_PRO_ENDPOINT environment variable.

An Authorization header Authorization: Bearer $ATMOS_PRO_TOKEN will be added to the HTTP request (if the ATMOS_PRO_TOKEN environment variable is set) to provide credentials to authenticate with the server.

NOTE: If the --upload=true command-line flag is passed, the --include-dependencies and --include-settings flags are automatically set to true, so the affected components will be uploaded with their dependencies and settings (if they are configured in Atmos stack manifests).

The payload of the HTTP POST request will be a JSON object with the following schema:

  {
     "base_sha": "6746ba4df9e87690c33297fe740011e5ccefc1f9",
     "head_sha": "5360d911d9bac669095eee1ca1888c3ef5291084",
     "repo_url": "https://github.com/cloudposse/atmos",
     "repo_host": "github.com",
     "repo_name": "atmos",
     "repo_owner": "cloudposse",
     "stacks": [
        {
          "component": "vpc",
          "component_type": "terraform",
          "component_path": "examples/quick-start/components/terraform/vpc",
          "stack": "plat-ue2-dev",
          "stack_slug": "plat-ue2-dev-vpc",
          "affected": "stack.vars",
          "included_in_dependents": false,
          "dependents": [],
          "settings": {}
        }
    ]
 }

where:

• base_sha - the Git commit SHA of the base branch against which the changes in the current commit are compared

• head_sha - the SHA of the current Git commit

• repo_url - the URL of the current repository

• repo_host - the host of the current repository

• repo_name - the name of the current repository

• repo_owner - the owner of the current repository

• stacks - a list of affected components and stacks with their dependencies

v1.81.0

Introduce Atmos `Go` template functions. Add `atmos.Component` function to read values from other Atmos components including outputs (remote state) @aknysh (#628)

## what

• Introduce Atmos Go template functions
• Add atmos.Component template function to read values from other Atmos components including outputs (remote state)
• Update docs
  • Atmos Template Functions
  • atmos.Component function

why

• Atmos now supports custom Go template functions similar to Sprig and Gomplate

  In Go templates in Atmos stack manifests, you can use the following functions and datasources:

  • Go text/template functions
  • Sprig Functions
  • Gomplate Functions
  • Gomplate Datasources
  • Atmos Template Functions

description

The atmos.Component template function allows you to read any Atmos section or any attribute from a section for an Atmos component in a stack, and use it in Go templates in Atmos component configurations.

Usage

  {{ (atmos.Component "<component>" "<stack>").<section>.<attribute> }}

Arguments

• component - Atmos component name

• stack - Atmos stack name

• section - Atmos section name. Any section returned by the CLI command atmos describe component can be used. A special outputs section is also supported to get the outputs (remote state) of Terraform/OpenTofu components.

  NOTE: Using the outputs section in the atmos.Component command is an alternative way to read the outputs (remote state) of a component in a stack directly in Atmos stack manifests instead of using the remote-state module and configuring Terraform/OpenTofu components to use the remote-state module as described in Component Remote State

• attribute - attribute name (field) from the section. attribute is optional, you can use the section itself if it's a simple type (e.g. string). Any number of attributes can be chained using the dot (.) notation. For example, if the first two attributes are maps, you can chain them and get a field from the last map:

    {{ (atmos.Component "<component>" "<stack>").<section>.<attribute1>.<attribute2>.<field1> }}

Specifying Atmos stack

stack is the second argument of the atmos.Component function, and it can be specified in a few different ways:

• Hardcoded stack name. Use it if you want to get an output from a component from a different (well-known and static) stack. For example, you have a tgw component in a stack plat-ue2-dev that requires the vpc_id output from the vpc component from the stack plat-ue2-prod:

    components:
      terraform:
        tgw:
          vars:
            vpc_id: '{{ (atmos.Component "vpc" "plat-ue2-prod").outputs.vpc_id }}'

• Use the .stack (or .atmos_stack) template identifier to specify the same stack as the current component (for which the atmos.Component function is executed):

    {{ (atmos.Component "<component>" .stack).<section>.<attribute> }}
    {{ (atmos.Component "<component>" .atmos_stack).<section>.<attribute> }}

  For example, you have a tgw component that requires the vpc_id output from the vpc component in the same stack:

    components:
      terraform:
        tgw:
          vars:
            vpc_id: '{{ (atmos.Component "vpc" .stack).outputs.vpc_id }}'

• Use the printf template function to construct stack names using static strings and dynamic identifiers:

    {{ (atmos.Component "<component>" (printf "%s-%s-%s" .vars.tenant .vars.environment .vars.stage)).<section>.<attribute> }}
  
    {{ (atmos.Component "<component>" (printf "plat-%s-prod" .vars.environment)).<section>.<attribute> }}
  
    {{ (atmos.Component "<component>" (printf "%s-%s-%s" .settings.context.tenant .settings.context.region .settings.context.account)).<section>.<attribute> }}

  For example, you have a tgw component deployed in the stack plat-ue2-dev. The tgw component requires the vpc_id output from the vpc component from the same environment (ue2) and same stage (dev), but from a different tenant net (instead of plat):

    components:
      terraform:
        tgw:
          vars:
            vpc_id: '{{ (atmos.Component "vpc" (printf "net-%s-%s" .vars.environment .vars.stage)).outputs.vpc_id }}'

  
  

  NOTE: By using the printf "%s-%s-%s" function, you are constructing stack names using the stack context variables/identifiers. For more information on Atmos stack names and how to define them, refer to stacks.name_pattern and stacks.name_template sections in atmos.yaml CLI config file

Examples

The following configurations show different ways of using the atmos.Component template function to read values from different Atmos sections directly in Atmos stack manifests, including the outputs of other (already provisioned) components.

# Global `settings` section
# It will be added and deep-merged to the `settings` section of all components
settings:
  test: true

components:
  terraform:
    test:
      metadata:
        # Point to the Terraform/OpenTofu component
        component: "test"
      vars:
        name: "test"

    test1:
      metadata:
        # Point to the Terraform/OpenTofu component
        component: "test1"
      vars:
        name: "test1"
    
    test2:
      metadata:
        # Point to the Terraform/OpenTofu component
        component: "test2"
      vars:
        name: "test2"
        # Use the `atmos.Component` function to get the outputs of the Atmos component `test1`
        # The `test1` component must be already provisioned and its outputs stored in the Terraform/OpenTofu state
        # Atmos will execute `terraform output` on the `test1` component in the same stack to read its outputs
        test1_id: '{{ (atmos.Component "test1" .stack).outputs.test1_id }}'
        tags:
          # Get the `settings.test` field from the `test` component in the same stack
          test: '{{ (atmos.Component "test" .stack).settings.test }}'
          # Get the `metadata.component` field from the `test` component in the same stack
          test_terraform_component: '{{ (atmos.Component "test" .stack).metadata.component }}'
          # Get the `vars.name` field from the `test1` component in the same stack
          test1_name: '{{ (atmos.Component "test1" .stack).vars.name }}'

[docs] Integration GHA fix version compatibility table @goruha (#626)

## what * [docs] Integration GHA fix version compatibility table

why

• Table in tip box looks ugly

v1.80.0

Add `--include-settings` flag to `atmos describe affected` command @aknysh (#624)

## what

• Add --include-settings flag to atmos describe affected command
• Update docs

why

• If the --include-settings=true flag is passed, atmos describe affected will include the settings section for each affected component in the stack. The settings sections is a free-form map used to pass configuration information to Atmos Integrations. Having the settings section in the output will allow the integrations to parse it and detect settings for the corresponding integration

Fix goreleaser @goruha (#623)

## what * Added `atmos` specific goreleaser

why

• Set atmos version on building

v1.79.0

Fix an issue with the `component_info` output from the `atmos describe component` command. Add `assume_role` property to Atmos JSON Schema S3 backend @aknysh (#621)

## what

• Fix an issue with the component_info output from the atmos describe component command
• Add assume_role property to Atmos JSON Schema S3 backend

why

• The issue with the component_info output from the atmos describe component command was introduced in the previous PRs (different order of execution when evaluation Go templates in Atmos stack manifests)

• Support the recommended assume_role property in S3 backends. Assuming an IAM Role can be configured in two ways. The preferred way is to use the argument assume_role, the other, which is deprecated, is with arguments at the top level (e.g. role_arn)

references

• https://developer.hashicorp.com/terraform/language/settings/backends/s3#assume-role-configuration

Update github actions documentation @goruha (#606)

## what * Update github actions documentation

why

• Document the latest gitops

references

• DEV-491: Update Atmos.tools documentation for GitHub Actions to use atmos.yaml

Fix tests @goruha (#619)

## what * Fix tests * Fix documentation

why

• Because branch master renamed to main

Go auto release workflows @goruha (#586)

# What * Use go auto-release workflow `cloudposse/.github/.github/workflows/shared-go-auto-release.yml@main` * Remove `.goreleaser.yml`. Now will use https://github.com/cloudposse/.github/blob/main/.github/goreleaser.yml * Drop `auto-release.yaml`. Now will use https://github.com/cloudposse/.github/blob/main/.github/auto-release.yml and https://github.com/cloudposse/.github/blob/main/.github/auto-release-hotfix.yml

Why

• Consolidate go releases workflow pattern
• Closes #579

v1.78.0

Update `atmos validate stacks` command @aknysh (#611)

what

• Update atmos vendor pull command
• Update atmos validate stacks command
• Add --include-dependents flag to atmos describe affected command
• Update docs
  • https://atmos.tools/cli/commands/describe/affected/

why

• When executing atmos vendor pull, Atmos creates a temp directory to clone the remote repo into.
  Atmos uses go-getter to download the sources into the temp directory. When cloning from the root of a repo w/o using modules (sub-paths), go-getter does the following:

  • If the destination directory does not exist, it creates it and runs git init
  • If the destination directory exists, it should be an already initialized Git repository (otherwise an error will be thrown)

  For more details, refer to

  • hashicorp/go-getter#114
  • https://github.com/hashicorp/go-getter?tab=readme-ov-file#subdirectories

• Don't check for duplicate abstract components in the same stack from different stack manifests. Abstract components are never provisioned and serve as blueprints for real components. This is an update (follow up) to the previous PRs:

  • #608
  • #611

• The --include-dependents flag allows including dependencies for the affected components

If the command-line flag --include-dependents=true is passed to the atmos describe affected command, and there are other components that depend on the affected components in the stack, the command will include a dependents property (list) for each affected component. The dependents property is hierarchical - each component in the list will also contain a dependents property if that component has dependent components as well.

For example, suppose that we have the following configuration for the Atmos components component-1, component-2 and component-3 in the stack plat-ue2-dev:

  components:
    terraform:
      component-1:
        metadata:
          component: "terraform-component-1"
        vars: {}

      component-2:
        metadata:
          component: "terraform-component-2"
        vars: {}
        settings:
          depends_on:
            1:
              component: "component-1"

      component-3:
        metadata:
          component: "terraform-component-3"
        vars: {}
        settings:
          depends_on:
            1:
              component: "component-2"

In the above configuration, component-3 depends on component-2, whereas component-2 depends on component-1.

If all the components are affected (modified) in the current working branch, the atmos describe affected --include-dependents=true command will produce the following result:

 [
   {
     "component": "component-1",
     "stack": "plat-ue2-dev",
     "stack_slug": "plat-ue2-dev-component-1",
     "included_in_dependents": false,
     "dependents": [
       {
         "component": "component-2",
         "stack": "plat-ue2-dev",
         "stack_slug": "plat-ue2-dev-component-2",
         "dependents": [
           {
             "component": "component-3",
             "stack": "plat-ue2-dev",
             "stack_slug": "plat-ue2-dev-component-3"
           }
         ]
       }
     ]
   },
   {
     "component": "component-2",
     "stack": "plat-ue2-dev",
     "stack_slug": "plat-ue2-dev-component-2",
     "included_in_dependents": true,
     "dependents": [
       {
         "component": "component-3",
         "stack": "plat-ue2-dev",
         "stack_slug": "plat-ue2-dev-component-3"
       }
     ]
   },
   {
     "component": "component-3",
     "stack": "plat-ue2-dev",
     "stack_slug": "plat-ue2-dev-component-3",
     "included_in_dependents": true
   }
 ]

The component-1 component does not depend on any other component, and therefore it has the included_in_dependents attribute set to false. The component-2 and component-3 components depend on other components and are included in the dependents property of the other components, and hence the included_in_dependents attribute is set to true.

When processing the above output, you might decide to not plan/apply the component-2 and component-3 components since they are in the dependents property of the component-1 component. Instead, you might just trigger component-1 and then component-2 and component-3 in the order of dependencies.