Agentic test automation from the command line (CLI)

This tutorial shows how to create automated test cases from plain-text specifications with the Step AI Agent, from within your IDE or your coding assistant. The agent runs from the CLI, where your code is: it explores the application, designs and implements the automation, runs it, and repairs what fails before handing back a Step Automation Package. What comes back is a plain Automation Package: its descriptor and fragments in YAML, and the keyword implementations in code (TypeScript here), which you review, commit, and run in CI/CD like any other code.

Scenario

A guest checkout on our own instance of OpenCart, from the test specification you would write to a job that gates your pipeline.

Prerequisites

• The Step AI Agent jar and the Step CLI 31 or newer: see Getting started
• Node.js 24 or newer
• An Anthropic API key exported as environment variable:

  
  export ANTHROPIC_API_KEY=sk-ant-...
  Copy

1. Create a test

Write the specification

It starts from one line: a visitor who is not registered can buy a product and receive an order confirmation.

The test summarizes in one line. We make steps explicit. For the CLI, the specification goes in instructions.json, the input file the Step AI Agent expects:

  
  {
  "testCases": [
    {
      "name": "opencart_checkout",
      "mode": "create",
      "spec": "OpenCart storefront - Guest checkout flow for a MacBook reached via the Laptops and Notebooks category. Steps: 1. Navigate to the Laptops and Notebooks category and find the MacBook product. 2. Add the MacBook to the cart. 3. Open the cart and proceed to checkout. 4. Select Guest Checkout and continue. 5. Fill in the billing/shipping form with customer details. 6. Confirm shipping method. 7. Accept terms and confirm payment method. 8. Place the order and verify the confirmation page. Test data: Storefront URL https://opencart-prf.stepcloud.ch/, customer Gustav Muster, customer@opencart.demo, Bahnhofstrasse 1, 8001 Zurich, Zürich, Switzerland."
    }
  ]
}
  Copy

Run the agent

Lay the repository out like this, and there is nothing to move between runs:

  
  my-repo/
  instructions.json          the specification you edit and commit
  opencart_tutorial_ap/      apDirectory — generated, deployable on its own
  Copy

Then, from its root:

  
  cd my-repo

INSTRUCTIONS=$(tr -d '\n' < instructions.json)

step ap execute --local \
  --package=/path/to/step-ai-agent \
  --includePlans="AI Agent - Automate" \
  --localAgentWorkDir=/tmp/step-ai-tutorial \
  -ep "apDirectory=$PWD/opencart_tutorial_ap|instructions=$INSTRUCTIONS|agentDebug=true"
  Copy

The specification stays outside the package because it is the input, not part of what you ship: the package is what CI runs and what you deploy to a Step instance. The request is not lost by separating them: the agent files a copy of every one under .ai/runs/<executionId>/ inside the package, so a package records what each run was asked for. That is the request only — logs/ holds the phase logs of the most recent run and is overwritten by the next, so keep a run’s logs elsewhere if you need to compare two.

The agent explores the storefront before writing any code, implements the test case, runs it, and repairs what fails before reporting success. A create run of this scenario takes several minutes.

It does not make assumptions or stop to ask for clarification. Before the first model call it rejects a duplicate test case name, a name Step would not accept as a plan name, or a request that reads as both a test and an RPA workflow. Later, while designing the keywords, it checks the requested steps against the running storefront, and a step the application cannot satisfy stops the run before any code is generated. Each case fails the command with a message naming what to change, so a broken specification costs you seconds rather than a full run.

The run closes with a report, written to opencart_tutorial_ap/logs/report.md. This is an extract:

  
  ## ✅ PASSED

✅ Run PASSED — 8 keyword(s) implemented, $1.7490, 6.42 min

## What this run produced

| Test case         | useCase | mode   | Verified by                          |
|-------------------|---------|--------|--------------------------------------|
| opencart_checkout | fct     | create | local execution (1/1 plans passing)  |

**Plan fidelity:** 1 covered · 0 partial · 0 missing
  Copy

The verdict PASSED or FAILED is the principal information, and it comes from running the automation: Step executes the generated keywords against the storefront, and a keyword whose check is not met fails its plan, which fails the run. Additional metrics are reported for information only, like plan fidelity, where a second model compares the plan against the specification you wrote and marks each test case covered, partial or missing. It never fails the run: it reads the plan rather than running it, and a plan the storefront cannot satisfy was already caught earlier, while the keywords were designed.

Read what came back

Everything the run produced is written in opencart_tutorial_ap, the directory set by the apDirectory execution parameter:

  
  opencart_tutorial_ap/
  automation-package.yaml          plan, stamped with the run that produced it
  fragments/
    nodejs-keywords.yaml           keyword declarations: role, inputs, outputs
  nodejs-keywords/src/keywords/    their TypeScript implementations
  keywords-specs/
    keywords-spec.md               the design, written before the code
  specs/
    plan.yaml                      the plan as designed, with the SUT URL
    plan-fidelity.json             request vs. plan: covered / partial / missing
  logs/                            one log per phase, with its cost and its reasoning
    design-browser-trace.zip       the agent exploring the application
    local-execution-traces/        one Playwright trace per keyword, on a remote run
  .ai/runs/<executionId>/
    instructions.json              the request, exactly as submitted
  Copy

The file automation-package.yaml contains the plan of the automation package.

The plan reads as the test case, one keyword per test step, and is stamped origin: ai with the execution that produced it, so AI-generated automation stays identifiable later. Eight keywords implement it, each performing its action and asserting the outcome it owns. openCartAndProceedToCheckout is typical:

  
  export async function openCartAndProceedToCheckout(input, output, session, properties) {
    const productName = str(input, 'productName', DEFAULTS.productName);
    const { context, page } = getPlaywright(session);

    await traced('openCartAndProceedToCheckout', context, output, async () => {
        await page.locator('#top-links').getByRole('link', { name: 'Shopping Cart' }).click();
        await page.waitForLoadState('networkidle');

        if (productName) {
            const cartLine = page
                .locator('#content form td.text-left')
                .getByRole('link', { name: productName, exact: true });
            try {
                await cartLine.waitFor({ state: 'visible' });
            } catch {
                throw new Error(`Product "${productName}" is not in the shopping cart (${page.url()})`);
            }
        } else {
            await page.locator('#content form').waitFor({ state: 'visible' });
        }

        await page.locator('#content').getByRole('link', { name: 'Checkout', exact: true }).click();
        await page.waitForLoadState('networkidle');
        await page.getByLabel('Guest Checkout').waitFor({ state: 'visible' });
    });
}
  Copy

What the agent worked out on its own

Most of that time went on questions nobody wrote down. The agent drove the storefront before writing any code, and a detail the specification left open is its to resolve: a selector that is wrong or ambiguous, a label that reads differently on screen, a screen that needs a wait. It keeps exploring until it has an answer that holds, and records the choice it made.

Examples from this run:

• The specification’s words are not the application’s. It says “Laptops and Notebooks”; the store’s menu says “Laptops & Notebooks”. The automation uses the menu label, and the difference is recorded rather than silently absorbed.
• The application behaves in a way nobody would think to specify. The region dropdown is re-loaded over AJAX when the country changes, so the keyword waits for the new options instead of selecting into a stale list.

That is what makes the selectors look fussy, and why keywords-specs/keywords-spec.md — the design the agent wrote before the code — is worth reading alongside the code itself.

Regenerate or extend a test case (optional)

Because the package is code, the agent can build on it rather than start again. Run it a second time with the same test case name and "mode": "extends", and it reads what the package already contains, adds what is missing, and reports every keyword and plan it touched:

  
  {
  "testCases": [
    {
      "name": "opencart_checkout",
      "mode": "extends",
      "spec": "… the same journey, plus: in the cart, change the quantity to two and verify that Sub-Total, Eco Tax, VAT and Total are recalculated."
    }
  ]
}
  Copy

The name must match a test case the package already holds, or the run stops in its first phase, before any model call. extends rewrites the package in place, so run it on a copy if the current state is not committed. See Configuration for the three modes.

2. Validate a test

The finished package is a standard Step Automation Package and runs independently of the agent, with no model cost, which is what makes it usable as regression coverage. Run it once from where you are:

  
  cd /path/to/opencart_tutorial_ap
step ap execute --local --package=.
  Copy

The same deterministic automation runs each time, and a failing plan makes the command fail, so the result already works as a build verdict.

To validate it on your Step instance too, drop --local and point the command at the instance, where STEP_URL is its URL, STEP_API_KEY an API key, and STEP_PROJECT_NAME the project to run in:

  
  cd /path/to/opencart_tutorial_ap
step ap execute --stepUrl=$STEP_URL --token=$STEP_API_KEY --projectName=$STEP_PROJECT_NAME --package=.
  Copy

3. Integrate in CI/CD

Gate the pipeline

From here the package runs unattended, on a Step instance rather than your machine: a vanilla and controlled environment, results the whole team can read, and the JUnit export a pipeline needs. That is the instance command above with the report flags added, and there is nothing to run by hand.

Since Step 31 the CLI ships as a Docker image, so the runner installs nothing. GitLab reads .gitlab-ci.yml from the repository root, and this is the whole file:

  
  stages:
  - test

step-ap-execute:
  stage: test
  image: docker.exense.ch/step/cli:3.31.0
  script:
    - >
      step ap execute
      --stepUrl=$STEP_URL --token=$STEP_API_KEY --projectName=$STEP_PROJECT_NAME
      --package=opencart_tutorial_ap --reportType=junit --reportDir=reports
  artifacts:
    when: always
    paths: [reports/]
    reports:
      junit: reports/**/*.xml
  Copy

STEP_URL, STEP_API_KEY and STEP_PROJECT_NAME are CI/CD variables, the key masked. Tag the image with the version of your Step server.

The job starts at the repository root, so --package is the package directory relative to it — opencart_tutorial_ap in the layout above. Where the package is the repository root, use . instead.

Two things make this a gate rather than a log. A failing plan fails the command, so the job fails with it and the merge request is blocked. And reports: junit: publishes the results into the merge request, where a reviewer reads which test case failed instead of opening the job output.

What each run leaves behind

GitLab shows the pass or fail and the JUnit report. The details are in Step, where each run is kept as a full execution with a trace attached to every step:

Eight keyword calls, all passed. That is where a failure gets diagnosed: the failing step, the data it was working with, and the Playwright trace of the browser at that moment.

Hand the package over

The executions are there, but the package is not. step ap execute runs it in an ephemeral context that is destroyed afterwards, which is exactly what a merge-request gate wants and exactly what makes the package invisible between runs. To install it, deploy it:

  
  step ap deploy --stepUrl=$STEP_URL --token=$STEP_API_KEY --projectName=$STEP_PROJECT_NAME --package=.
  Copy

That makes its plans and keywords available on the instance and activates any schedule the package carries. Run it once when the package is ready to be used by people other than the pipeline, or add it to a job on your default branch so the instance always holds the current version.

Where it goes from here

The package gates your pipeline on every change, without the agent and at no model cost. Deployed to the instance, it is also readable automation: the plan reads as the test case in plain language, and every execution carries a trace for each step, which is what lets whoever owns the specification confirm it still matches, and schedule it as recurring coverage once the application is live.

👉 Agentic test automation with Step Studio tells the same story from that side.