The Azure DevOps integration plugin provides an out-of-the-box integration of Step with the Azure Test Plans module of Azure DevOps.

With the Azure DevOps plugin you create, design and manage your automated test cases in Azure Test Plans, run them on Step’s Execution Engine, and evaluate the run results and test reports at the same place in Azure Test Plans. This plugins is therefore the perfect solution for users requiring the best of the 2 worlds: the unique power of the Step Execution Engine with the test management and reporting features of Azure Test Plans.

The test case design relies on the standard plain text syntax of Step (The same as used by the other integration plugins like ALM and Jira).

Plugin Usage

Test Case Definition in Azure DevOps

Test Cases can be defined directly in Azure DevOps, using Step’s plain text syntax.

In the “Expected result” column, you can enter assertions which are expected to hold.

Running Tests in Step

To execute tests stored in Azure DevOps, start by selecting Plugins -> “Azure DevOps”:

You will be presented with a window where you can browse repositories, plans, suites, and test cases. From there, you can launch entire Test Suites, or individual Test Cases.

Run Evaluation in Azure DevOps

After tests were run in Step, the results are exported back to Azure DevOps, where they can be found under the “Runs” menu item:

Installation and Configuration

Azure DevOps

In order to interact with Azure DevOps, Step must have the permission to do so. This is achieved by creating an Access Token which has at least full read/write access to the “Test Management” functionality. If you want to use Shared Steps, read/write access to “Work Items” is also required.

Step

In Step, the integration is configured using the step.properties file on the controller.

There are two kinds of entries to create.

repository.azuredevops.projects

This configuration key contains a list of organization/project combinations that can be interacted with, separated by semicolons. Spaces within project names are permitted and used verbatim; spaces around the semicolon separators are permitted and discarded.

For example, if your project’s root URL on Azure devops is https://dev.azure.com/myorganization/myproject , the corresponding configuration entry is:

repository.azuredevops.projects = myorganization/myproject

If you wish to access multiple projects (possibly even in a different organization), add additional entries, like so:

repository.azuredevops.projects = myorganization/myproject; myorganization/otherproject; anotherorg/test project

repository.azuredevops.token.<organization>

For each organization that is to be accessed, add a configuration entry with the organization name and the respective access token.

Example:

repository.azuredevops.token.myorganization = thetokenformyorganizationabcdefghijlkmnopqrstuvwxyza

Further configuration

Starting with Step version 3.17.2, a number of further configuration options are available. These include:

• Specifying whether, and how, the numerical DevOps IDs of test plans, suites, and cases, are to be displayed while browsing them in the Step UI (default: disabled)
• Specifying the name that will be appended to test runs in the DevOps UI (default: STEP)
• Specifying the format of the attached reports (HTML and/or plain-text, default: HTML)
• Controlling whether the original test step formatting is to be preserved in reports (default: false)

Please refer to the step.properties file for further documentation on configuring these options.

Disabling reporting for executions

By default, all executions of test suites or cases defined in Azure DevOps are also reported back to DevOps as Test Runs. This may not be desirable in all cases.

Starting with Step version 3.17.2, it is therefore possible to define an execution parameter which allows to control the reporting.

The execution parameter MUST have the ID azure_devops_reporting and have a boolean value (true/false). Here is an example on how to define it:

Known Issues and Limitations

There are a number of limitations imposed by Azure DevOps. Below, we list the ones we are currently aware of:

Access Tokens

• Access Tokens are bound to a specific Azure DevOps User. Executions will always be shown as belonging to that user. It is therefore recommended to use a technical user for that purpose, if possible.

• Access Tokens have a maximum allowed lifetime of 12 months. After the token expires, a new one must be created and the configuration must be updated accordingly.

Running tests, evaluating test results

• It is unfortunately not possible to directly launch test runs in Step from Azure DevOps. In other words, tests must be initiated from within Step using the “DevOps browser” functionality.

• Test runs are always reported as “Manual”. To mitigate this, we append “(STEP)” (or the value configured in the configuration file) to the names of the runs which were executed in Step.

Reporting

Reporting in Azure DevOps is limited in a number of ways.

• Test steps which were run repeatedly can only show up once in the report.

• Only a very short “comment” field without support for formatting is available.

For these reasons, Step attaches reports (in a configurable format, by default HTML) to each Test Case that was run, providing more detailed information.

Miscellaneous

Executing nested test suites recursively is not supported.