Testing Automation

Test your automation as you develop it, then run the same checks in CI. Check shared functions, command behavior, and deployed services so changes to your build and release process have tests of their own.

A type: test step runs checks and reports which passed and which failed. Each check can be a script, an HTTP request, a shell command, or another supported step. Use test steps in workflows, custom commands, and lifecycle hooks. Atmos collects the results, shows the output of failed checks, and returns a failing exit code when a suite fails.

In a script written in the Atmos Automation Language, use a condition and fail() to report an unmet expectation. A script that finishes successfully passes.

Create a test workflow

With Atmos installed and on your PATH, save this as atmos.yaml. The first example uses only built-in functions and needs no external tools:

atmos.yaml

base_path: .
workflows:
  base_path: workflows

Save this complete workflow as workflows/checks.yaml. It produces a structured result, then checks both that result and a reusable calculation:

workflows/checks.yaml

workflows:
  check:
    steps:
      - name: calculate
        type: script
        interpreter: starlark
        script: |
          output = {"replicas": 3, "workers": 12}
      - name: checks
        type: test
        title: Capacity checks
        steps:
          - name: result
            type: script
            interpreter: starlark
            env:
              RESULT: '{{ .steps.calculate.value }}'
            script: |
              actual = json.decode(env["RESULT"])
              if actual["workers"] != actual["replicas"] * 4:
                  fail("worker count must equal replicas times four")
          - name: calculation
            type: script
            interpreter: starlark
            script: |
              def capacity(replicas):
                  return replicas * 4
              for replicas, expected in [(1, 4), (2, 8), (3, 12)]:
                  actual = capacity(replicas)
                  if actual != expected:
                      fail("expected {}, got {}".format(expected, actual))

Run from the directory containing atmos.yaml:

atmos workflow check -f checks

Both checks pass and the command exits with code zero. The output value from the first step is JSON-encoded; the test receives it through its environment and decodes it into a dictionary. Shared logic can live in a .star module loaded by both your automation and its checks.

Change the produced worker count from 12 to 11 and rerun. The result check fails with worker count must equal replicas times four, the independent calculation check still passes, and the workflow exits nonzero.

Test a custom command's behavior

Test the interface your team uses as well as the logic behind it. For the custom command guide's atmos capacity, add this child under the test step's steps list in the same project:

- name: rejects-zero
  type: script
  interpreter: starlark
  script: |
    result = exec.run(
        ["atmos", "capacity", "--replicas", "0"],
        check=False,
        output="capture",
    )
    if result.exit_code == 0:
        fail("capacity accepted zero replicas")
    if "replicas must be positive" not in result.stdout + result.stderr:
        fail("capacity failed for an unexpected reason")

This runs the real custom command. check=False returns an unsuccessful process result so the script can check both the exit code and the diagnostic. Launch or cancellation errors still fail the step. output="capture" keeps the expected error out of the live output while making it available to the check.

The same approach tests standalone scripts and other tools. Use fixed inputs for deterministic checks; subprocess tests still need the tools they invoke.

Read results and use them in CI

Successful logs stay hidden by default. Failed checks display their buffered output and errors; set output: all on the test step to include successful logs. Interactive terminals show a live tree, while CI and redirected output use static results.

The default failure policy lets independent checks finish, then fails the group if unhandled failures remain. Nest parallel or matrix groups when checks can run concurrently or need several input combinations. Keep the default failing policy for suites intended to gate CI.

Test steps also work inside custom commands and lifecycle hooks. See the test step reference for failure policies, supported children, counts, and hook configuration. The language overview describes output values and process execution; the test example shows a larger suite.