Continuous end-to-end testing

In this tutorial you’ll learn how to set up continuous end-to-end testing across several applications based on Playwright tests in your DevOps pipeline using Step.

For this example our end-to-end test case will be the following:

• Purchase an article in exense’s Demo online shop
• Login to exense’s Demo Webmail and wait for the order confirmation Email

Prerequisites

• Access to a Step cluster: Get started quickly by setting up a free SaaS cluster in the Step Portal, or, if preferred, follow the Installation page to configure your own on-premise cluster.
• GitLab or any other DevOps platform
• Maven (optional): to optionally build and execute the load test locally

Set up the project

Checkout

While you could set up your project from scratch, we recommend to get started with one of the samples available in Git. In this tutorial we’ll start with the project e2e-testing-playwright

git clone https://github.com/exense/step-samples.git
cd step-samples/automation-packages/e2e-testing-playwright

Project structure

The project “e2e-testing-playwright” is a standard maven project ready to build. It contains the following files:

Playwright Keyword (scripts)

The playwright scripts automating the 2 test cases are contained in the classes OpenCartPlaywrightKeywords.java and WebmailPlaywrightKeywords. It provides 2 keywords (“Buy MacBook in OpenCart” and “Read order confirmation in Webmail”) that encapsulate the automation scripts implemented with Playwright.

Automation package with load test plan

The plan configuring the end-to-end test case is defined in the descriptor of the automation package: src/main/resources/automation-package.yaml:

---
schemaVersion: 1.0.0
name: "end-to-end-testing-playwright"
plans:
  - name: "OpenCart Test Set 01"
    root:
      testSet:
        children:
          - testCase:
              nodeName: "OpenCart Test Case 01"
              children:
                - callKeyword:
                    keyword: "Buy MacBook in OpenCart"
                - callKeyword:
                    keyword: "Read order confirmation in Webmail"
        threads: 1

Maven Project

The file pom.xml defines how the maven project and thus the automation package is built and how it is executed in Step. The section relevant to the execution of the load test is shown here:

<build><plugins><plugin>
    <groupId>ch.exense.step</groupId>
    <artifactId>step-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>execute-automation-package</id>
            <phase>integration-test</phase>
            <configuration>
                <url>${step.url}</url>
                <authToken>${step.auth-token}</authToken>
                <stepProjectName>${step.step-project-name}</stepProjectName>
                <executionParameters>
                    <env>TEST</env>
                </executionParameters>
                <waitForExecution>true</waitForExecution>
                <executionResultTimeoutS>600</executionResultTimeoutS>
                <ensureExecutionSuccess>true</ensureExecutionSuccess>
            </configuration>
            <goals>
                <goal>execute-automation-package</goal>
            </goals>
        </execution>
    </executions>
</plugin></plugins></build>

These definitions ensure that during the integration-test phase of maven (which is performed as a precondition of the mvn verify command), the execute-automation-package goal provided by the Step maven plugin is run. For more information about the various maven phases, see this reference.

You will need to provide the URL to your Step cluster, as well as the project name and API key (refer to Generate an API Key). In this example, the actual values are referencing variables which are conveniently defined at the beginning of the file. Make sure to adjust these values to your setup.

The other configuration parameters specify parameters to pass to Step for the actual execution (executionParameters), tell maven to wait for the execution result for a maximum of 10 minutes (waitForExecution and executionTimeoutS), and finally, to check that the execution in Step was indeed successful (ensureExecutionSuccess).

Execute locally

During development or for debugging purposes, you can execute the test with JUnit using the following command:

mvn test

Execute in Step

To trigger the execution in a Step cluster, run the following maven command:

mvn verify -DskipTests

The test is executed in Step according to the plan specified in resources/automation-package.yaml. The maven project is configured to wait for the execution to complete.

Integrate into the CI/CD pipeline

As the project is a standard maven project, the integration into any CI/CD pipeline is straightforward.

In this tutorial we’ll show how to integrate it using GitLab. Doing the same with any other CI/CD pipeline supporting Maven is very similar.

Create a project in GitLab, clone it, then copy the contents of the sample maven project “load-testing-playwright” directly into the git project directory.

The project already includes a suitable .gitlab-cy.yml:

image: maven:3-openjdk-11

variables:
  STEP_URL: "https://your-step-instance.url/"
  STEP_PROJECT: "Common"
  STEP_TOKEN: "Your Step API token"
  MVN_ARGS: >-
    --settings gitlab-maven-settings.xml
    -Dmaven.repo.local=.m2/repository
    --batch-mode
    --no-transfer-progress    

cache:
  paths:
    - .m2/repository

stages:
  - verify

maven-verify-job:
  stage: verify
  script:
    - mvn ${MVN_ARGS} verify -DskipTests -Dstep.url=${STEP_URL} -Dstep.step-project-name=${STEP_PROJECT} -Dstep.auth-token=${STEP_TOKEN}

You only have to adjust the Step URL, project name, and token in this file. In addition, please modify the gitlab-maven-settings.xml to include your credentials for accessing the Exense maven repository.

Pushing these changes to the remote repository will trigger the execution of the build pipeline in GitLab, which will run the mvn verify command. This command in turn will trigger the execution of the test in your Step cluster.

Analyze the results

After execution of the pipeline in GitLab the job report should look similar to this:

To analyze the results of the load test execution in Step, navigate to the execution link printed in the logs of the job: