Categories: DEVELOPER GUIDE CI CD DEV OPS ENTERPRISE

Automation Packages

Abstract: Step is a complete and established solution for Software Automation; its ability to cover a multitude of use cases – from functional and performance tests, to RPA tasks, synthetic monitoring, and more – allows to realize the most demanding automation needs. Based on the Automation as Code paradigm, Automation Packages extend Step with DevOps-native functionality, allowing to maintain application and automation code in tandem, and bringing powerful automation capabilities to DevOps pipelines, as well as simplifying Step’s own usage and maintenance.

As the IT landscape is evolving and adjusting to continuously changing requirements, state-of-the-art methodologies and paradigms such as agile development, containerized infrastructures, continuous integration/delivery are being employed; the combination of these approaches is commonly subsumed under the name “DevOps”. In the following, we will briefly summarize some common terminology used in this context.

DevOps refers to a set of practices and tools aiming to improve the software lifecycle. Notably, it encompasses both development and operations, not as separate disciplines, but as closely related ones. The process is considered to be a perpetual cycle iterating through the various phases, as shown in the figure below:

There are two important aspects to note:

  • Because this is an ever-repeating cycle, the word continuous is aptly describing the nature of the process and its phases.
  • As a result, and precisely because everything is happening in a repeated manner, it is logical to strive for as much automation as possible, to reduce the necessary manual workload and overhead.

CI/CD (Continuous Integration and Continuous Delivery/Deployment) is generally understood to encompass the Build, Test, Release, and Deploy phases of the cycle using automated tools tailored to the specific jobs (e.g. compiling and packaging the software, running unit tests, uploading software to repositories, provisioning it on servers).

The aforementioned phases of the DevOps cycle are well covered by tools such as GitLab pipelines and similar CI/CD tools, as they can be performed as automatic reactions to changes in a codebase. Such tools will orchestrate the operations required to perform the corresponding steps, often integrating specialized tools for particular tasks (examples include linters to perform code quality checks, documentation generators, solutions for running unit and integration tests, etc.). For a concrete example, Step could be integrated into the CI/CD pipeline during the test phase to perform end-to-end load tests, as outlined in more details below.

The last phase a CI/CD tool is typically responsible for when reacting to code changes is the deployment phase where software is put into production (of course, pipelines may also stop earlier and not cover this phase). The subsequent long-running phases, namely the Operation and Monitoring ones are usually out of scope for such tools. However, it is possible to reduce that “automation gap”, again by using Step’s functionality, thanks to the solutions presented below.

Automation as Code is the underlying concept. At its core, it means that all application automation logic is expressed in code; just like Unit Tests, said code (defining assets like keywords and plans used to define e.g. end-to-end acceptance tests, load and performance tests, synthetic monitoring, RPA, or any other automation workflow) is stored alongside the actual application’s code. This provides the following advantages:

  • Application code and the corresponding automation code are stored and maintained in the same place, instead of separately.
  • Therefore, established coding guidelines, branching policies etc. are automatically and seamlessly applied to automation code as well.
  • Versioning incompatibilities or similar problems between application and automation code are also less likely to occur, because both are developed in tandem.

Automation Packages are unique to Step, and are produced from the abovementioned automation assets code. While they are typically built and deployed by CI/CD tools during the respective phases, they allow to cover requirements present in many phases of the DevOps lifecycle. In particular, suitable Automation Packages can not only provide functional or performance tests, but also extend the scope to Operation and Monitoring, by including the corresponding definitions and configurations for RPA and/or scheduled synthetic monitoring.

This page introduces Automation Packages by presenting:

  • how they are structured
  • examples of which tasks they can be used for
  • how they are used

Automation Packages in a nutshell

The figure below gives a high-level overview of the content of an automation package, as well as the most important operations:

Assets

As can be seen, there are multiple kinds of assets involved:

  1. Keywords: Keywords encapsulate the automation logic and are the building blocks of plans. You can write keywords entirely from scratch, as well as integrate existing automation solutions.
  2. Plans: Plans define the operational logic that Step performs. They are defined in a YAML format.
  3. Schedules: Schedules define which plans are to be periodically executed. They are also defined in YAML.
  4. Alerting Rules: If plans fail to execute correctly (for example by failing, or by not meeting user-definable performance or other criteria), alerts can be sent using various mechanisms. Flexible rules for such circumstances may be defined, again expressed using a YAML notation.
  5. Resources: Plans may refer to additional resources, such as bundled data used during plan executions (e.g., Excel or CSV files containing test data).

Operations

At automation package build time, these assets are packaged into an archive file which can then be operated on. The most important operations are:

  1. Package: This operation creates an automation package from the assets and resources. This is the equivalent of the compilation step of traditional programs.
  2. Publish: This operation publishes an automation package to an artifact repository, but does not perform any further actions. This is an optional step that allows to deploy or execute an automation package on multiple Step servers.
  3. Deploy: This operation uploads an automation package to a Step server, makes all the assets it contains available for use in Step and activates the contained schedules. This is the equivalent of a full “installation”.
  4. Execute: This operation performs a one-off execution of the plans contained in an automation package on a Step server, in a temporary isolated context. The content of the automation package is therefore executed once and then removed again, but not permanently deployed or activated. This operation is useful for instance for performing load or performance tests during the Test phase of the DevOps cycle.

Integration with CI/CD pipelines

The figure below shows how automation packages are built and used within a CI/CD pipeline.

This illustrates the following aspects of the automation packages:

  • The code of automation packages is stored in a git repository, along with the actual application to automate.
  • The automation package is built by the CI/CD pipeline together with the application itself.
  • The automation package can be deployed to an artifact repository like the application itself.
  • The execution of the plans contained in the automation package is triggered by the CI/CD pipeline at the different stages of the pipeline. For example:
    • Test: Execute the plans contained in the automation package after deployment of the application to the test environment
    • Operate: Provide Self-Service Automation services, automate recurring tasks
    • Monitor: Schedule periodic monitoring of the application after its deployment to the productive environment

Applications of Automation Packages

The range of practical applications of automation packages is wide and in principle, they could be employed in almost every stage of a DevOps cycle. Still, it makes sense to use the most suitable tool for the tasks at hand, so a combination of traditional build tools/pipelines and automation packages is the recommended approach.

This section gives a quick overview of the main applications of automation packages, grouped by DevOps phase.

Plan

While automation packages don’t offer functionality to actively assist you in this phase, it is recommended to keep in mind their potential for the upcoming phases.

Code

This is where automation packages, i.e., their code and configuration, are actually defined, along with the application code.

Build

In principle, Step is flexible enough to automate build-related tasks as well; however, other established tools like traditional build tools and CI/CD pipelines are specifically designed for this purpose, and we recommend using these. In fact, this stage usually includes building the actual automation packages, along with the application itself.

Test

This is one of the phases that are explicitly in scope for Automation Packages. Traditionally, this phase includes at least functional tests to ensure correctness of the application (i.e., Unit Tests). The coverage can be significantly broadened by leveraging automation packages at this stage, enabling a wide range of additional test scenarios, such as end-to-end integration tests or performance tests. With a suitably configured Step instance – which may include thousands of execution agents – demanding large scale tests are possible.

Here is a brief list of various possible use cases:

Automated load and performance tests

  • Goal: Fully automated load and performance tests for early detection of performance and scaling issues
  • The scripts (VU), load model and SLOs are defined as code by the developer
  • Features:
    • Support for protocol and client (e.g. browser, mobile app) based automation
    • Possible integration of existing automation solutions
    • All configurations (Load model, SLOs) defined as YAML
    • Load test execution triggered by CI/CD pipeline and then orchestrated by Step controller
    • Scalable Step agent grid for large scale parallel, multi-zone executions

Automated large scale end-to-end tests

  • Goal: Fully automated large scale E2E tests for early detection of integration issues
  • The scripts and test plans are defined as code by the developer
  • Features:
    • Multi application support: design complex E2E tests across multiple applications
    • Support for protocol and client (e.g. browser, mobile app) based automation
    • Possible integration of existing automation solutions
    • Test configuration defined as YAML
    • Test execution triggered by CI/CD pipeline and then orchestrated by Step controller
    • Scalable Step agent grid for large scale executions

Automated creation of test data

  • Goal: Fully automated creation of test data after deployment to the test environment
  • The scripts and data are defined as code by the developer
  • Features:
    • Support for protocol and client (e.g. browser, mobile app) based automation
    • Possible integration of existing automation scripts
    • Support of CSV, XLS, JSON, YAML or database data sources
    • Test data creation triggered by CI/CD pipeline and then orchestrated by Step controller
    • Scalable agent grid for large scale parallel executions

Automated monitoring of the test environment

  • Goal: Permanent monitoring of the deployed application in the test environment for proactive detection of issues (performance or availability)
  • The scripts (monitors), schedules, SLOs, and alerting rules are defined as code by the developer
  • Features:
    • Support for protocol and client (e.g. browser, mobile app) based automation
    • Possible integration of existing automation solutions
    • All configurations (Schedules, SLOs, alerting rules) defined as YAML
    • Monitoring deployed by CI/CD pipeline and then scheduled and periodically executed by Step controller

Release

Whereas Step would be flexible enough to automate the tasks related to the release of an application, this is not its intended primary scope, therefore it is recommended to use pipelines or build tools specifically tailored to such tasks.

Deploy

While the Test phase is the most suitable one for demanding load, performance, or integration tests, one may wish to include somewhat smaller-scale tests during the deployment in production to ensure that the system is functioning correctly.

Automated smoke tests

  • Goal: Ensure correct operation of the application by running smoke tests after deployment in production
  • The scripts and test plans are defined as code by the developer
  • Features:
    • Support for protocol and client (e.g. browser, mobile app) based automation
    • Possible integration of existing automation solutions
    • Test configuration as YAML
    • Test execution triggered by CI/CD pipeline and then orchestrated by Step controller

Operate

One of the unique features of automation packages is that they can contain automation logic for interacting with target applications during their operation in production. Technically, such logic is deployed to a Step instance by the CI/CD pipeline, where it can then be invoked to perform the respective operations in the application. Examples include:

Self-service RPA tasks

  • Goal: Expose self-service RPA tasks
  • Example: you’re operating a purchased software for which you want to propose self-service jobs that automate repetitive tasks to your users
  • The scripts (RPA definitions), business inputs and access rules (who’s allowed to execute the service) are defined as code by the developer
  • Features:
    • Support for protocol and client (e.g. browser, mobile app) based automation
    • Possible integration of existing automation solutions
    • All configurations (SLOs, alerting rules) defined as YAML
    • Available RPA services can be triggered from the full-featured Step UI, or via REST

Automate recurrent tasks

  • Goal: Regularly automate otherwise manual routine tasks such as maintenance or cleanup jobs etc.
  • Features:
    • Support for protocol and client (e.g. browser, mobile app) based automation
    • Possible integration of existing automation solutions
    • All configurations (Schedules, SLOs, alerting rules) defined as YAML

Monitor

Similarly to the Operating aspects, where automation packages are used mainly to simplify otherwise manual tasks and interactions, they can also be used in a proactive way to detect issues on running production servers using Synthetic Monitoring. Again, the respective logic is deployed by the CI/CD pipeline to a Step cluster, which then actively monitors the running application after its deployment.

Synthetic monitoring

  • Goal: Permanent monitoring of the deployed application in production for proactive detection of issues (performance or availability)
  • The scripts (monitors), schedules, SLOs, and alerting rules are defined as code by the developer
  • Features:
    • Support for protocol and client (e.g. browser, mobile app) based automation
    • Possible integration of existing automation solutions
    • All configurations (Schedules, SLOs, and alerting rules) defined as YAML
    • Monitoring deployed by CI/CD pipeline and then scheduled and periodically executed by Step controller

 

Automation package formats

With the introduction of automation packages in Step 24, a single file format (Java archive) is supported for defining automation packages, but this may be extended in the future.

JAR (Java archive)

In its JAR format the automation package consists of a standard JAR file that contains all the automation code and libraries as Java classes, as well as the automation package descriptor as YAML.

The Java automation package is built just like any regular Java .jar archive. We recommend using maven for this task, but you can use any suitable build toolchain that is capable of producing correct jar files.

Other formats

Future versions of Step may include support for other formats for automation packages, such as files specifically tailored to the requirements of .NET keywords, or an “exploded” format (consisting of raw, non-packed directory structures). However, at this time, no specific roadmap exists for such formats, and they may be added as the need for them arises.

Layout

At its core, an automation package’s definition consists of a single file named automation-package.yaml, placed in the top-level directory of the AP archive.

This file will usually contain Step plans, which in turn reference Step keywords, which may themselves have dependencies on libraries etc.

Because the automation package is self-contained, it needs to include all those dependencies, and therefore, usually the biggest part of the automation package’s content will in fact consist of compiled code or other kinds of resources required for the execution of said code.

Below are annotated examples showing the content of actual automation package files:

okhttp load test example

This is the content of the automation package (in JAR format) built from this tutorial. It contains Step keywords defined in Java, and therefore the resulting classes, as well as their transitive dependencies.

META-INF/                 jar metadata
kotlin/                   Java classes - dependencies
okhttp3/                  Java classes - dependencies
okio/                     Java classes - dependencies
org/                      Java classes - dependencies
step/                     Java classes - keywords
automation-package.yaml   Automation Package definition

K6 load test example

This is the content of the automation package (in JAR format) built from this tutorial. In this example, the keyword is defined directly in the automation-package.yaml file, simply referencing the Grafana K6 script to be executed. Therefore, the automation package itself does not need to include any further dependency; however, it will only work on Step clusters where the Step agents include a working K6 installation.

META-INF/                 jar metadata
automation-package.yaml   Automation Package definition
opencart-test.js          K6 script

 

Command Line Interface (CLI)

The current version of the automation package CLI is based on Maven. In the following, we provide short, concrete examples of how to perform specific tasks. Please refer to the Plugin documentation for the full details.

Note that these commands must be run in the context of a maven project that includes the respective step plugin and configuration.

Define an automation package

Automation packages are defined in a file called automation-package.yaml. Please see the dedicated documentation for the file format.

Execute an automation package in Step

mvn package step:execute-automation-package "-Dstep.url=https://your.step.server/" "-Dstep.step-project-name=Common" "-Dstep.auth-token=your_token_in_text_format"

Execute an automation package locally

If you wish to execute an automation package locally on the development machine (instead of a Step cluster), you may do so by defining a trivial Java JUnit Test class with this minimal code:

import org.junit.runner.RunWith; import step.junit.runner.Step;
@RunWith(Step.class) public class RunAutomationPackageTest {}

This test can be executed using the test functionality of the IDE, or from the command line:

mvn test -Dtest="RunAutomationPackageTest"

Deploy an automation package to Step

mvn package step:deploy-automation-package "-Dstep.url=https://your.step.server/" "-Dstep.step-project-name=Common" "-Dstep.auth-token=your_token_in_text_format"

Getting Started

The easiest way to get started is to use one of our tutorial projects, which include a complete project definition containing a few simple keywords and plans along with a small load test definition. It should be straightforward to execute these projects and start from there by modifying them according to your needs.

See Also