DevOps Synthetic Monitoring with Playwright - Advanced

This tutorial demonstrates how Playwright tests can be reused for synthetic monitoring of a productive environment in a DevOps workflow

Illustration for playwright synthetic monitoring in a devops workflow

Synthetic monitoring, also known as active monitoring, is a technique using automation to simulate users interactions on a target applications at regular intervals to monitor its performance, functionality and availability in production.

This tutorial will demonstrate how Playwright automation can be turned into full synthetic monitoring by configuring all aspects such as scheduling, assertion and alerting as code and integrate it into a CI/CD pipeline.

The example for this tutorial simulates online shoppers using exense’s demo online shop based on opencart.

Prerequisites

Access to a Step cluster: Get started quickly by setting up a free SaaS cluster in the Step Portal, or, if preferred, follow the Installation page to configure your own on-premise cluster.
Step gateway configuration: this tutorial defines an email alerting rule in case an incident is opened. An email gateway named Email notification gateway must be configured in Step
GitLab or any other DevOps platform
Maven (optional): to optionally build and execute the synthetic monitoring scripts locally

Set up the project

Checkout

While you could set up your project from scratch, we recommend to get started with one of the samples available in Git. In this tutorial we’ll start with the project synthetic-monitoring-playwright-advanced

  git clone https://github.com/exense/step-samples.git
cd step-samples/automation-packages/synthetic-monitoring-playwright

Project structure

The project “synthetic-monitoring-playwright” is a standard maven project ready to build. It contains the following files:

Playwright Keyword

Step’s keyword enable seamless integration of automation scripts with the Step platform using the Step Keyword API.

The playwright script, that automates the user path for the synthetic monitoring, is contained in the class PlaywrightKeywordExample.java. It provides a single keyword (“Buy MacBook in OpenCart”). Note that for simplicity, this keyword stops just before actually placing the order, so as not to empty our virtual stock too quickly :-)

Automation package with Synthetic Monitoring configuration

The synthetic monitoring configuration is defined in the descriptor of the automation package: src/main/resources/automation-package.yaml.

It contains following fragments:

plans:
- The test case plan ‘Opencart synthetic monitoring plan’ defines the execution of the automated workflow and the assertions per single execution
- The assertion plan ‘Opencart schedule assertion plan’ defines assertions for the scheduled executions on a sliding time window
schedules configures the scheduling of the synthetic monitoring (frequency, plan to be executed, assertion plan to be evaluated)
alertingRules configures one alerting rule to send an email whenever an incident is opened for this synthetic monitoring package

As you can see, the specification is straightforward – the complete definition is shown below.

  ---
name: "synthetic-monitoring-playwright"
plans:
  - name: "Opencart synthetic monitoring plan"
    root:
      testCase:
        children:
          - callKeyword:
              keyword: "Buy MacBook in OpenCart"
          - performanceAssert:
              measurementName: "Buy MacBook in OpenCart"
              comparator: "LOWER_THAN"
              expectedValue: 5000
  - name: "Opencart schedule assertion plan"
    root:
      assertionPlan:
        description: "Assert the scheduled executions did not fail more than once in the last 10 minutes"
        children:
          - assertMetric:
              description: "Make sure executions did not fail more than once in the last 10 minutes"
              comparator: LOWER_THAN
              aggregation: SUM
              expectedValue: 2
              metric: "executions/failure-count"
              slidingWindow: 600000
              errorCode: 10 #custom error code
schedules:
  - name: "Opencart synthetic monitoring schedule"
    #CRON expression for one execution per minute
    cron: "0 0/1 * * * ?"
    planName: "Opencart synthetic monitoring plan"
    assertionPlanName: "Opencart schedule assertion plan"
alertingRules:
  - description: "Send email for incident created for Opencart synthetic monitoring"
    name: "Opencart synthetic monitoring email"
    active: true
    eventClass: IncidentOpenedEvent
    conditions:
      - BindingCondition:
          bindingKey: "executionDescription"
          predicate:
            BindingValueEqualsPredicate:
              value: "Opencart synthetic monitoring plan"
    actions:
      - NotificationViaGatewayAction:
          gatewayName: "Email notification gateway"
          gatewayParameters:
            - "changeme@changeme.changeme"

Note: You’ll have to set your own email address in the gatewayParameters to effectively receive a notification.

Maven Project

The file pom.xml defines how the maven project and thus the automation package is built and how it is deployed to Step. The section relevant to the deployment to Step is shown here:

  <!-- Deploy the automation package to Step -->
<plugin>
    <groupId>ch.exense.step</groupId>
    <artifactId>step-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>deploy-automation-package-on-step</id>
            <phase>integration-test</phase>
            <configuration>
                <url>${step.url}</url>
                <authToken>${step.auth-token}</authToken>
                <stepProjectName>${step.step-project-name}</stepProjectName>
            </configuration>
            <goals>
                <goal>deploy-automation-package</goal>
            </goals>
        </execution>
    </executions>
</plugin>

These definitions ensure that during the integration-test phase of maven (which is performed as a precondition of the mvn verify command), the deploy-automation-package goal provided by the Step maven plugin is run. For more information about the various maven phases, see this reference.

You will need to provide the URL to your Step cluster, as well as the project name and API key (refer to Generate an API Key). In this example, the actual values are referencing variables which are conveniently defined at the beginning of the file. Make sure to adjust these values to your setup.

Execute locally

During development or for debugging purposes, you can execute the automation workflow locally using the following command:

  mvn test

Deploy to Step

To deploy this automation package to a Step cluster, run the following maven command:

  mvn verify -DskipTests

The automation package is deployed to Step enabling the synthetic monitoring on the demo application.

Integrate into the CI/CD pipeline

As the project is a standard maven project, the integration into any CI/CD pipeline is straightforward.

In this tutorial we’ll show how to integrate it using GitLab. Doing the same with any other CI/CD pipeline supporting Maven is very similar.

Create a project in GitLab, clone it, then copy the contents of the sample maven project “load-testing-playwright” directly into the git project directory.

The project already includes a suitable .gitlab-cy.yml:

  image: maven:3-openjdk-11

variables:
  STEP_URL: "https://your-step-instance.url/"
  STEP_PROJECT: "Common"
  STEP_TOKEN: "Your Step API token"
  MVN_ARGS: >-
    --settings gitlab-maven-settings.xml
    -Dmaven.repo.local=.m2/repository
    --batch-mode
    --no-transfer-progress

cache:
  paths:
    - .m2/repository

stages:
  - verify

maven-verify-job:
  stage: verify
  script:
    - mvn ${MVN_ARGS} verify -DskipTests -Dstep.url=${STEP_URL} -Dstep.step-project-name=${STEP_PROJECT} -Dstep.auth-token=${STEP_TOKEN}

You only have to adjust the Step URL, project name, and token in this file. In addition, please modify the gitlab-maven-settings.xml to include your credentials for accessing the Exense maven repository.

Pushing these changes to the remote repository will trigger the execution of the build pipeline in GitLab, which will run the mvn verify command. This command in turn will trigger the execution of the load test in your Step cluster.

Analyse the synthetic monitoring in Step

Once the synthetic monitoring in place, you can follow the performance and availability of the demo application using the Step’s analytics dashboard.

In case an incident is raised, you will receive an email containing all important information such as the reason of the incidents and direct link to Step. The link of the incident will bring you to the detailed page of this incident:

Refer to the knowledge base for more information on alerting or incident managements

Using Step with Grafana

This article demonstrates how to connect Grafana to data generated by Step.

Setting up system monitoring with a Step agent

This article demonstrates how to set up distributed system monitoring using Keyword executions, and analyze the results as measurements.

NET tutorials: Microsoft Office automation with Step

This tutorial demonstrates how to automate interaction with Microsoft Office applications using the Office Interop Assembly.

JUnit Plan Runner

This article provides documentation for how to integrate JUnit tests into Step.

How to monitor services availability and performance

This tutorial demonstrates how Step can be used to monitor services, availability and performance metrics.

.NET tutorials: AutoIt with Step

This tutorial demonstrates how to utilize the AutoIt C# binding to automate interactions with Windows applications.

Android Testing using Step and Appium

This article demonstrates the automation of mobile applications on Android using the Appium framework.

Browser-based automation with Step and Selenium

This article defines three Keywords which will be used in browser-based automation scenarios, using Step and Selenium, as general drivers.

Illustration for Load Testing with Cypress

Load Testing with Cypress - advanced

This tutorial shows you how to efficiently set up a browser-based load test using existing Cypress tests in the Step automation platform.

Adding and Configuring New Agents

In this short tutorial, we show how to quickly implement a simple browser-based load test based on Cypress scripts in Step.

Illustration for Load Testing with Playwright

Load Testing with Playwright using Step UI

This tutorial shows you how to set up a browser-based load test using existing Playwright tests in the Step UI.

Basic Keyword Development

This article explains Keywords in Step and demonstrates how to create simple ones.

Designing functional tests

This tutorial demonstrates the design, execution, and analysis of functional tests using the web interface of Step.

Robotic Process Automation (RPA) with Selenium

This tutorial will demonstrate how to use Step and Selenium to automate various browser tasks.

Robotic Process Automation (RPA) with Cypress

This tutorial demonstrates how to use Step and Cypress to automate various browser tasks.

Synthetic Monitoring with Selenium

This tutorial demonstrates how Selenium automation tests can be turned into full synthetic monitoring using Step.

Load Testing with Cypress

In this tutorial, you'll learn how to reuse existing Cypress tests to quickly set up and run a browser-based load test using the automation as code approach.

Load Testing with Serenity BDD and Cucumber

In this tutorial, you'll learn how to reuse existing tests written with Serenity BDD and Cucumber for load testing.

Synthetic Monitoring with Cypress

This tutorial demonstrates how Cypress automation tests can be turned into full synthetic monitoring using the automation as code approach.

Load Testing with Cypress using Step UI

In this tutorial, you'll learn how to reuse existing Cypress tests to quickly set up and run a browser-based load test using the Step UI.

Load Testing with Selenium

This tutorial demonstrates how to leverage existing Selenium tests to set up and execute browser-based load tests, following a full code-based approach.

Load Testing with Selenium using Step UI

This tutorial demonstrates how to set up a browser-based load test in the Step UI using existing Selenium tests.

Synthetic Monitoring with Playwright

This tutorial demonstrates how Playwright automation tests can be turned into full synthetic monitoring using Step.

Synthetic Monitoring with Cypress using Step UI

This tutorial demonstrates how Cypress automation tests can be turned into full synthetic monitoring using the Step UI.

Robotic Process Automation (RPA) with Playwright

This tutorial will demonstrate how to use Step and Playwright to automate various browser tasks.

Distributed load testing with JMeter

This tutorial shows how to distribute JMeter tests across multiple nodes.

Load Testing with Playwright for Java

In this tutorial, you'll learn how to reuse existing Playwright tests written in Java to quickly set up and run a browser-based load test using the automation as code approach.