Automation Packages are defined in a declarative way using YAML alongside the automation code. The Step Automation Package YAML syntax is specified by the Automation Package JSON schema, for more convenient work you can configure the embedded file editor in your IDE (IntelliJ IDEA or Visual Studio) to associate this schema with the Step YAML automation packages.

For Java Automation Packages, it is possible to directly declare certain entities in the code in addition to the YAML descriptor.

YAML descriptor

The automation package description is specified in a file name automation-package.yaml by convention. The top-level nodes of this descriptor consists of the following fields:

• name - the name of automation package
• version - specifies the version of the automation package schema used by this descriptor; required to ensure compatibility with future versions of Step
• keywords - fragment containing the keywords specifications
• plans - fragment containing the plans specifications
• parameters - fragment containing the parameters specifications
• schedules - fragment containing the schedules specifications
• presets - fragment containing the notifications presets specifications
• alertingRules - fragment containing the alerting rules specifications
• metadata - free-form metadata attached to the package, which Step stores but does not act on

Each fragment is optional.

Example:

---
version: 1.0.0
name: "load-testing-k6-automation-package"
keywords:
  - K6:
      name: 'OpenCart home'
      vus: 1
      iterations: 1
      scriptFile: "opencart-test.js"
plans:
  - name: "Opencart load test plan"
    root:
      testScenario:
        children:
          - threadGroup:
              users: 5
              iterations: 10
              children:
                - callKeyword:
                    keyword: "OpenCart home"

Name

The name of the automation package is not only used when deploying it to Step for organization purposes, it is also used as unique identifier. When deploying a package to Step, if a package with the same identifier already exists, it will be updated instead of deploying a new one.

For more advanced use cases, Step supports deploying multiple versions of a same automation package.

Schema version

This field defines the version of the automation package schema used by this descriptor. The available versions of the YAML automation package schema are described in the below table.

*the version 1.1.1 is available since Step 28.0, but wrongly kept the previous version number. It is corrected starting with version 28.1.

Backward compatibility

Step supports multiple schema versions to maintain backward compatibility. To ensure your automation package remains compatible with future versions of Step, you must explicitly set this field to the schema version used when editing the package.

In other words, an automation package using the version 1.0.0 can be deployed on a Step instance using the version 1.1.0 as long as the version field is set in your YAML descriptor.

Upgrading the automation package schema version

To access the latest features in existing automation packages, update the version in your YAML descriptors and ensure your IDE uses the corresponding schema version.

Depending on the type of version upgrade (major, minor or patch) following should be considered:

• A new patch version indicates the introduction of new optional fields; upgrading the schema is only required to be able to use and validate these new optional fields in your package.
• A new minor version indicates localized breaking changes; upgrading the schema is required to make use of the new functionalities and will require to manually migrate the content of your YAML.
• A new major version is currently not planned but would indicate major changes not compatible with previous major version.

Keywords

Automation Packages support all Step Keyword types in a declarative way except for QF-Test and PDF Test keywords. This includes for instance Java-based keywords (Playwright, Cypress, Selenium, etc), JMeter keywords and Grafana k6 keywords. You may read the related documentation for details on the configuration properties related to each keyword type, the same are available in YAML format.

Following sections give some details for a subset of these keyword’s types

Java keywords

Java keywords are supported natively with the Java @Keyword annotation and do not have to be specified in the YAML descriptor. In other words, any Keywords defined in the Java code contained in the automation package will be part of the package.

Please note that Java Keywords can include any third-party library and can thus be used to declare Keywords relying on Selenium, Playwright, or any other library supporting Java.

JMeter

Existing JMeter test plans can be declared as Keyword in automation packages using the JMeter keyword type as follows:

keywords:
  - JMeter:
      name: "JMeter keyword from automation package"  
      description: "JMeter keyword 1"
      jmeterTestplan: "jmeterProject1/jmeterProject1.xml"

Fields:

• name: an arbitrary name for the Keyword. This name will be used in the plan to reference the Keyword
• jmeterTestplan: the relative path to the JMeter test plan (xml)

Refer to the automation package schema for an exhaustive list of supported fields.

Cypress

Cypress Keywords can be declared in automation packages using the Cypress type as follows:

keywords:
  - Cypress:
      name: eCommerce - Typical visit
      cypressProject: cypress-test/
      spec: "opencart.cy.js"

Fields:

• name: an arbitrary name for the Keyword. This name will be used in the plan to reference the Keyword
• cypressProject: the relative path to the Cypress project containing the cypress.config.js
• spec: the filename of the Cypress spec to be executed

Refer to the automation package schema for an exhaustive list of supported fields.

Read more about the Cypress integration here

Grafana k6

Existing k6 tests can be declared as Keyword in automation packages using the K6 keyword type as follows:

keywords:
  - K6:
      name: 'OpenCart home'
      scriptFile: "opencart-test.js"
      vus: 1
      iterations: 1

Fields:

• name: an arbitrary name for the Keyword. This name will be used in the plan to reference the Keyword.
• scriptFile: relative path to the k6 script file to be executed.
  • If scriptDirectory is not specified, the path to scriptFile is considered relative to the root of the automation package.
  • If scriptDirectory is specified, then scriptFile should be defined relative to that directory.
• scriptDirectory (optional): The directory containing the script file and any supporting modules. If not set, the directory containing the defined script file will be used as the script directory.
  • This directory is transferred to the agent at execution time.
  • You can place and import local modules here using standard import statements in your k6 script.
  • To ensure proper module resolution, all required modules must reside within the specified scriptDirectory.
  • Modules must be placed inside the scriptDirectory. Modules located in the root of the automation package are not supported.
• vus (optional): number of VUs to run concurrently (k6 option ‘vus’).
• iterations (optional): total number of script iterations to be executed across all VUs (k6 option ‘iterations’).

Refer to the automation package schema for an exhaustive list of supported fields.

Read more about the k6 integration here

.NET

.NET keywords can be declared in automation packages using the DotNet type as follows:

keywords:
  - DotNet:
      name: "Open_Chrome_and_search_in_Google"
      dllFile: "DotNet/SeleniumKeywords.dll"
      librariesFile: "DotNet/AllDlls.zip"

Fields:

• name: the name of the Keyword. This name will be used in the plan to reference the Keyword. It should match the name of the Keyword contained in the referenced DLL
• dllFile: the relative path to the DLL file containing the Keyword
• librariesFile: optional path to the folder containing the library DLLs

Refer to the automation package schema for an exhaustive list of supported fields.

Read more about the development of C# keywords on the C# Keywords page

Node.js (JavaScript / TypeScript)

JavaScript / TypeScript keywords can be declared in automation packages using the Node keyword type as follows:

keywords:
  - Node:
      name: MyNodeKeyword
      jsfile: nodejs-keywords/

Fields:

• name: the name of the Keyword. This name will be used in the plan to reference the Keyword. It should match the name of the Keyword contained in the referenced project
• jsfile: the relative path to the Node.js project containing the Keyword definitions. This project should contain the package.json file and the keyword definitions in the folder keywords/

Refer to the automation package schema for an exhaustive list of supported fields.

Read more about the development of JavaScript/TypeScript keywords in the Keyword API

Oryon

Oryon keywords can be declared in automation packages using the Oryon keyword type as follows:

keywords:
  - Oryon:
      name: OryonKeyword
      scriptFile: oryon-script.groovy

Fields:

• name: an arbitrary name for the Keyword. This name will be used in the plan to reference the Keyword
• scriptFile: the relative path to the Oryon script

Refer to the automation package schema for an exhaustive list of supported fields.

Prebuilt library keywords

This section declares a prebuilt library Keyword directly inside your package, keeping it self-contained (see the self-containment trade-off versus deploying the library centrally and calling it by name).

A keyword from an officially prebuilt library is a Java keyword backed by the library’s jar file, and is imported the same way as any other Java script keyword: using the GeneralScript keyword type, with scriptFile pointing at the library jar. Each keyword you intend to call must be declared individually — there is no construct to import “the whole library” at once.

keywords:
  - GeneralScript:
      name: ExecuteBash
      scriptLanguage: java
      scriptFile: step-library-kw-system-1.0.31.jar

Fields:

• name: must match the keyword’s actual name inside the library (e.g. ExecuteBash, ExecuteCmd, HttpRequest, ExecuteQuery, …) — this is also the name used to call it from callKeyword.
• scriptLanguage: java for these libraries.
• scriptFile: relative path to the library jar. The jar itself is not published as part of your package’s source; download it from Maven (see Prebuilt Keyword Libraries) and bundle it alongside your automation-package.yaml.

For example, running an existing Playwright test project via the imported ExecuteBash keyword (generic command execution):

plans:
  - name: "Run existing Playwright suite"
    root:
      testCase:
        children:
          - callKeyword:
              keyword: ExecuteBash
              inputs:
                - Command: npx playwright test

Composite keyword

Composite keywords can be declared in automation packages using the Composite type.

keywords:
  - Composite:
      name: "Composite Keyword"
      plan:
        root:
          sequence:
            children:
              - echo:
                  text: "In Composite keyword"
              - return:
                  output:
                    - compositeOutput: "Composite Keyword output value"

The node plan contains the composite plan and follows the same structure as normal plans.

Read more about the Composite Keywords here

Plans

Plans are fully supported by the YAML format meaning that you can define plans in YAML identical to the ones created with the visual editor in Step. To understand the purpose and configuration of the many plan controls, refer to the dedicated documentation

Plans syntax

The plans are represented by a tree structure as in the visual plan editor and must contain a root node defining the type of plans (test set, test case, test scenario …). The different nodes share at least the following common properties:

• nodeName - [optional] name this node
• categories - [optional] list of categories assigned to this plan (categories can be used to filter plans for local and remote executions. See Filtering Plans).
• description - [optional] some textual description for the node
• children - [optional] the list of children of this node

Refer to the Controls documentation for the documentation and examples of the supported controls. Calling Keywords and Plans is detailed in Calling Keywords and Calling Plans.

For instance, the following YAML descriptor contains a single plan of type test case invoking a keyword named “My Keyword”

---
name: "My automation package"
plans:
  - name: "Simple plan"
    categories:
      - "FunctionalTests"
    root:
      testCase:
        children:
          - callKeyword:
              keyword: "My Keyword"

Convert visual plan to YAML

To convert existing plans to YAML or to help you develop plans in YAML syntax, you may generate the plan in YAML format from the visual plan editor.

Convert YAML to visual plan

You can create a visual plan by providing its YAML source in the UI:

Calling Keywords

Keywords are called from a plan with the callKeyword control. In its simplest form, only the name of the keyword is specified:

- callKeyword:
    keyword: "My Keyword"

In addition to the common node properties, callKeyword supports the following fields:

The following example combines these fields:

- callKeyword:
    nodeName: "Search product on Windows"
    keyword: "Search product"
    inputs:
      - productName: "Hand blender"
      - maxResults: 10
    routing:
      - OS: "Windows"

Selecting the keyword

The keyword field accepts either the name of the keyword as a string, or a map of selection criteria. The two following calls are equivalent:

- callKeyword:
    keyword: "My Keyword"
- callKeyword:
    keyword:
      name: "My Keyword"

Each criterion is matched against the attribute of the same name of the keywords, and only a keyword matching all the criteria is called. The keywords of an automation package are identified by their name attribute. Additional attributes are typically defined on keywords created in the Step UI, such as application or the custom attributes added through the screen templates:

- callKeyword:
    keyword:
      name: "Login"
      application: "Shop"

The value of each criterion can be a dynamic expression, for instance to call a keyword whose name is only known at runtime:

- set:
    key: "keywordToCall"
    value: "Login"
- callKeyword:
    keyword:
      name:
        expression: "keywordToCall"

When several keywords match the criteria, the keyword of the automation package being executed takes precedence over the keywords deployed elsewhere.

Keyword inputs

The inputs field is a list of entries, each defining the name and the value of one input. Values can be strings, numbers, booleans, maps or lists, or dynamic expressions evaluated at runtime. Strings may embed variables using string interpolation:

- callKeyword:
    keyword: "Create order"
    inputs:
      - customer: "John Doe"
      - quantity: 3
      - express: true
      - address:
          street: "Main Street 1"
          city: "Basel"
      - items: [ "item-1", "item-2" ]
      - orderDate:
          expression: "new Date().format('yyyy-MM-dd')"
      - url: "http://${host}:8080/orders"

Keyword routing

The routing field defines the criteria selecting the agent token on which the keyword is executed. Like inputs, it is a list of entries, each mapping the name of an agent token attribute to the expected value. The expected value is a regular expression that must match the whole attribute value, and may be a dynamic expression:

- callKeyword:
    keyword: "Open browser"
    routing:
      - OS: "Windows"
      - browser: "chrome|edge"
      - region:
          expression: "targetRegion"

These criteria are combined with the routing defined on the enclosing session control, on the keyword itself and with the route_to_ variables. Refer to Keyword routing for the order of precedence.

When defined within a session, the criteria of the session take precedence over the routing of the callKeyword for the same attribute. Additional criteria not matched by the token already reserved by the session cause a new token to be reserved, on which the keyword doesn’t share the session objects of the other keywords.

To execute the keyword on the controller instead of an agent, set remote to false:

- callKeyword:
    keyword: "Compute test data"
    remote: false

Keyword outputs

The output of a keyword is available as the output variable within the children of the callKeyword control, and as the previous variable in the following controls. Refer to Keyword outputs for details.

- callKeyword:
    keyword: "Search product"
    inputs:
      - productName: "Hand blender"
    children:
      - assert:
          actual: "productId"
          operator: "EQUALS"
          expected: "1234"
- set:
    key: "productId"
    value:
      expression: "previous.productId"

Alternatively, resultMap copies the string fields of the output into an existing variable. If the variable holds a map, the fields are added to it. If it holds a DataSet handle, a row is added for the output, as well as one row per object of the output’s list fields. Fields holding lists or objects are not copied, and the output, including the objects of its lists, must not contain number or boolean fields, which make the keyword call fail:

- set:
    key: "searchResults"
    value:
      expression: "[:]"
- callKeyword:
    keyword: "Search product"
    resultMap: "searchResults"
- echo:
    text:
      expression: "searchResults.productId"

Calling Plans

Plans are called from another plan with the callPlan control. The called plan is executed as if its content were part of the calling plan, which allows to build reusable sub plans. In its simplest form, only the name of the plan is specified:

- callPlan:
    plan: "Login sequence"

In addition to the common node properties, callPlan supports the following fields:

Selecting the plan

The plan field works like the keyword field of callKeyword: it accepts either the name of the plan, or a map of selection criteria matched against the attributes of the plans. The value of each criterion can be a dynamic expression:

- callPlan:
    plan:
      name:
        expression: "'Login sequence - ' + env"

When several plans match the criteria, the plan of the automation package being executed takes precedence over the plans deployed elsewhere.

Plan inputs

The input field has the same syntax as the inputs field of callKeyword. The inputs are available in the called plan as the input variable, a JSON object whose values are read with its getString, getInt, getBoolean, … methods:

plans:
  - name: "Order product"
    root:
      testCase:
        children:
          - callPlan:
              plan: "Login sequence"
              input:
                - user: "john"
                - password:
                    expression: "johnPassword"
  - name: "Login sequence"
    root:
      sequence:
        children:
          - callKeyword:
              keyword: "Login"
              inputs:
                - user:
                    expression: "input.getString('user')"
                - password:
                    expression: "input.getString('password')"

The variables defined in the calling plan are also visible in the called plan.

Dynamic expressions

As in the visual plan editor, most fields of controls and keyword inputs accept either a plain value or a Groovy expression, for instance to resolve the value of a variable. In YAML, a Groovy expression is written with the expression key.

Example of a thread group defined with literal values:

- threadGroup:
  users: 5
  iterations: 10

Example of a thread group defined with variables:

- threadGroup:
  users: 
    expression: "nbUsers"
  iterations: 
    expression: "nbIterations"

String interpolation

The expression form above turns the whole field into a Groovy expression. When you only want to inject a variable into a piece of text, you can instead keep the plain value and embed the variable with the ${...} placeholder syntax. The expression form remains necessary for numbers, booleans and complex objects, or for logic the placeholder syntax cannot express.

Without interpolation, injecting a variable into a URL means writing the whole value as a groovy expression, with its quoting and concatenation:

- callKeyword:
    keyword: "Get product"
    inputs:
      - url:
          expression: '"http://" + host + ":8080/products/" + productId'
      - description:
          expression: '"Product " + productId + " on " + env'

With interpolation, the same plan reads as plain YAML:

- callKeyword:
    keyword: "Get product"
    inputs:
      - url: "http://${host}:8080/products/${productId}"
      - description: "Product ${productId} on ${env}"

This works in the plain values of all fields that also accept the expression form, in plans as well as in Parameters. The text outside the placeholders is used exactly as written, so backslashes, quotes and line breaks need no escaping:

- set:
    key: "logFile"
    value: 'C:\logs\${date}.txt'
- set:
    key: "payload"
    value: '{"user":"${name}","id":${id}}'

To write a literal ${, escape it by doubling the dollar sign. This is typically needed when the value is a template meant to be interpreted by another tool:

- callKeyword:
    keyword: "Deploy manifest"
    inputs:
      # passed through as ${JOB_NAME}, not interpolated by Step
      - template: "name: $${JOB_NAME}"

Refer to Embedding variables in text values for the complete syntax and the escaping rules.

Controls - YAML specificities

While most controls properties are identical to the one of the visual plans or self-explanatory, there a few exceptions.

AssertMetric

In YAML, the metrics names are slightly different than in the UI, here are the mapping from YAML to UI:

• response-time: “Response time”
• executions/count: “Execution count”
• executions/failure-percentage: “Execution failure percentage”
• executions/failure-count: “Execution failure count”
• executions/failures-count-by-error-code: “Execution failure count by error code”

Agent provisioning settings

When running in Kubernetes, Step supports automatic provisioning of Agents. This section explains how to define the provisioning settings in Automation Packages.

The provisioning settings can be defined for each Plan using the agents field.

Refer to the Agent provisioning page for more details about this feature.

Enable automatic determination and provisioning of agents

To enable the automatic determination and provisioning of Agents, the field agents has to be set to “auto_detect” as follows:

---
version: 1.0.0
name: "my-automation-package"
plans:
- name: "OpenCart okhttp Load Test"
  agents: auto_detect
  ...
  

Manual specification of agents

In order to specify the agents manually, a list of pools can be defined in the agents field:

---
version: 1.0.0
name: "my-automation-package"
plans:
- name: "OpenCart okhttp Load Test"
  agents:
  - pool: java-enterprise-agent
    replicas: 2
    image: docker-dev.exense.ch/test:agent-java-custom
    

It supports 3 fields:

• pool: specifies the name of the pool template to be provisioned. You can get the list of available pool in the UI (see previous section).
• replicas: specifies the number of agents to be provisioned.
• image: optionally specifies the Docker image to be used for the pool. Refer to Creating your own Agent images for details on how to create custom images.

Plain text plans

Plain text plans can also be included in automation packages. Simply include the files in the automation package and reference them in the descriptor using following syntax (only the file property is mandatory):

plansPlainText:
  - file: "plans/this-is-a-plain-text-plan.plan" # relative path to the file, supports wildcard 
    name: "this is a plain text plan" # optional plan name (cannot be specified when using wildcard for the file) 
    rootType: TestCase # optional root type of the plan (TestCase by default)
    categories: # optionally assign categories which can then be used for the selection of plans when triggering executions 
      - myTestCategory

Parameters

Parameters can be defined in the YAML descriptor as they would be in the Step UI. They also support dynamic expression in YAML.

Example:

parameters:
  - key: "baseURL"
    value: "http://test.com"
    activationScript: "env == 'TEST'"
  - key: "baseURL"
    value: "http://prod.com"
    activationScript: "env == 'PROD'"
  - key: "fullURL"
    value:
      expression: baseURL + "/something"

Schedules

Schedules can be defined in the YAML descriptor too using the same configuration properties as in the Step UI. Note if you need help to define the CRON expression we recommend using the Step UI wizard.

Example:

schedules:
  - name: "Opencart synthetic monitoring schedule"
    #CRON expression for one execution per minute
    cron: "0 0/1 * * * ?"
    planName: "Opencart synthetic monitoring plan"

Notification Presets

Here is a snippet showcasing the definition of Notification Presets:

notificationPresets:
  - EmailNotification:
      presetName: "Sample mail"
      upstreamPreset: "Mail"
      subject: 
        data: "${eventSummary}"
        protection: READONLY
  - WebhookNotification:
      presetName: "Sample Webhook call"
      upstreamPreset: "Webhook"
      url:
        data: "https://example.org"
      method:
        data: "POST"
      body:
        data: "{\"Hello\": \"World\"}"
      headers:
        data:
          "X-Step-URL" : "${controllerUrl}"

Alerting rules

Alerting rules can also be defined in YAML as you would do it in the UI. Example:

alertingRules:
  - description: "Send email for incident created for Opencart synthetic monitoring"
    name: "Opencart synthetic monitoring email"
    eventClass: IncidentOpenedEvent
    conditions:
      - BindingCondition:
          bindingKey: "executionParameters"
          predicate:
            BindingExistsPredicate:
              {}
      - BindingCondition:
          bindingKey: "executionParameters[env]"
          predicate:
            BindingValueEqualsPredicate:
              value: "PROD"
    actions:
      - NotificationAction:
          notification:
            EmailNotification:
              upstreamPreset: "Sample Mail"
              from:
                data: "me@example.org"
              to:
                data:
                  - "you@example.org"
                  - "them@example.org"
              bodyPlaintext:
                data: "An incident was opened, technical information: %{bindings}"

Metadata

The automation package and each of its entities - keywords, plans, parameters, schedules, notification presets and alerting rules - accept an optional metadata field. Its content is free-form YAML: any keys you like, holding nested maps, lists or plain values.

Metadata is purely informative: it has no effect on how an entity behaves. It is propagated to the deployed entities.

The Step AI workflow uses metadata to keep track of its work. You can also use it for your own bookkeeping, such as the requirements a plan covers or the team owning a package:

version: 1.3.0
name: "my-automation-package"
metadata:
  owner: "quality-engineering"
  repository: "https://git.example.org/tests"
plans:
  - name: "Checkout test"
    metadata:
      requirements: [ "REQ-42", "REQ-43" ]
      origin: "ai"
    root:
      testCase:
        children:
          - echo:
              text: "Hello world"

Values are unrestricted, but keys must not be empty, contain a dot or start with a dollar sign, at any nesting level. These characters are reserved by the databases the metadata is stored in, so a package using them is rejected when deployed.

Fragment files may declare metadata at their top level too. It is kept in the file, but only the metadata of the main descriptor is attached to the deployed package.

Fragment files

The concept of fragment files enables the creation of modular automation packages.

A fragment file declares automation package entities using the same syntax as automation package files and can be included in a main automation package to build packages in a modular way.

Fragment files are referenced in the main automation package file using the fragments field:

version: 1.0.0
name: "My Automation Package"
fragments:
- "fragment1.yml"
- "fragment2.yml"

Each entry of the fragment list can be either the relative path to the referenced fragment or a wildcard expression referencing a list of fragments:

version: 1.0.0
name: "My Automation Package"
fragments:
- "fragments/*.yaml"

Example

Here’s an example of a main automation-package.yaml referencing 2 fragments that each define a plan to be included in the automation package:

version: 1.0.0
name: "My Automation Package"
fragments:
- "plan1.yml"
- "plan2.yml"

The plan1.yml file would only contain the definition of the plan 1:

plans:
  - name: "Plan 1"
    root:
      testCase:
        children:
          - echo:
              text: "Hello world 1"

Setup your IDE for auto-completion and validation

The JSON schema can be downloaded from your Step instance by going to the Automation Package view and accessing the help menu. The underlying REST service can also be used directly for integration: <Your_Step_instance>/rest/automation-packages/schema

In IntelliJ IDEA add the JSON schema mapping as follows

1. In IDEA setting open the JSON Schema Mappings in Languages & Frameworks section.
2. Find and select the downloaded JSON Schema from your Step instance (Automation Package Schemas are currently not publicly available on schemastore.org )
3. Choose the files you want to link with the selected JSON Schema

After that you can use the chosen schema in the embedded YAML editor:

1. When you type something in the YAML editor linked with JSON Schema, IDEA shows the on-the-fly suggestions and validates entered data according to the selected JSON Schema
2. For the current file you can change the link to the JSON Schema (switch schema from OS version to EE or update schema to the latest version)