Intro

This tutorial will walk you through the steps needed to connect Grafana to the data generated by step. At the end of the tutorial, you will have:

configured Grafana to connect to data generated by step in a database of your choice
imported a performance dashboard
visualized step data in Grafana

Why Grafana

Grafana is a well-known open-source dashboarding product with versatile functionality, broad community and vast ecosystem. Moreover, it is frequently updated and very devops friendly. It is especially well-suited for analyzing time-series data, exactly such as the data generated by step. We believe that enabling users to access step data in Grafana alongisde other metrics that they have already available will allow users to generate much more value out of their data.

Grafana relies on “data source plugins” to ingest data from various databases and applications. While there is no “step data source” plugin, step users can connect Grafana to the databases that contain step measurement data. In the open-source step measurements are stored in MongoDB, and in step Enterprise there is a possibility to use a plugin to concurrently store measurements in a different database.

Grafana dashboard is built out of “panels”, and besides built-in panel types shipped with Grafana, there is a collection of community-developed panel plugins as well. In step dashboards we have included some of these plugins for better usability (see below), however the most basic version of the dashboard will also work without them.

Grafana and MongoDB

By default, step writes the measurement data into a MongoDB collection. At this moment, Grafana doesn’t ship with a MongoDB data source within the open-source edition. However, there is a community-developed open-source MongoDB plugin available, which relies on a node.js proxy server serving the aggregation requests from Grafana UI to the Mongo backend and back. This tutorial has been written for the users who are using an open-source setup and therefore it requires this community plugin to be installed, in order to connect to the step data stored in MongoDB.

If you have an Enterprise version of Grafana, you may be able to use an official MongoDB data source instead. In this case, slight re-working of the aggregation queries may be needed, as these two data source plugins expect slightly different data formatting (such as renaming some fields in the output of the last aggregation stage).

The dashboards require MongoDB v4.0.x or later.

step data and other databases

Through the step Enterprise analytics plugins the user can enable step to concurrently write data into databases of choice such as PostgreSQL or Prometheus. Both PostgreSQL and Prometheus data sources are available in the open-source edition of Grafana and will work out of the box.

Performance considerations

Grafana is designed to re-run all the active queries of all the open panels every time the dashboard is refreshed, time range changed or a different variable selected. This may lead to performance problems if you have a huge number of measurements already present in the database of your choice. While we cannot assist you in optimizing your database setup, there is a couple of things that you may want to try:

optimize indexes in your database, especially for the fields which are declared as dashboard variables, as they are often used in queries (eId, name, status, type)
in the dashboard, hide rows with the panels you don’t need at the moment - the queries inside the panels in hidden rows are not being fired
use the dashboards provided as a starting point to create your own dashboards - remove panels and variables which you don’t use frequently
consider using a different database - in particular, TimescaleDB may be significantly faster than MongoDB - or Prometheus as an approximate monitoring solution

Overview of the essential steps

The general steps needed to monitor step data in Grafana are the following:

Install Grafana according to the official instructions. We recommend version 7.5.5+ (The provided dashboards will be partially working from 7.2+).
Decide on the data source:

If using other databases than MongoDB - enable step to write data to external databases

Configure Grafana

If using MongoDB as a data source, install the community plugin
Install additional panel plugins as described below
Configure the data source
Import a step dashboard

Run an execution in step to produce data
Monitor and analyze your step data in Grafana

Below we elaborate on each of them in detail.

Install and configure Grafana to read step data

If you don’t have a running Grafana instance, install one following the official documentation. Once Grafana is installed, run the Grafana server and open Grafana in the browser. You will be asked to log in (admin/admin) and create a new user. With these initial steps, you should have Grafana up and running.

Plugins

While Grafana offers an automated way to install plugins using grafana-cli, in our experience many plugins hosted in the official repository are not up-to-date with their latest releases and the waiting time for a new release of a plugin to be accepted in the official repository is considerably long (months). Therefore for trusted plugins we recommend manual installation of the latest release, not yet visible in grafana-cli:

Go to a relevant git repository
Download the most recent plugin release
Unpack the plugin to the plugin directory you have selected in the Grafana configuration (default is GRAFANA_DIR/data/plugins)
Restart Grafana server (and force refresh the Grafana UI if kept running)

Please make sure that using unsigned or manually installed plugins is compliant with the guidelines in your organization.

Plugins relevant for step dashboards

The following Grafana plugins are relevant for step dashboards:

MongoDB data source

Each of them can be installed following the steps described above.

Configure the data sources

Once you have installed the plugins, it is time to connect Grafana to step data by defining a Grafana data source. If you run your database at a different configuration than the standard one, have this information ready.

MongoDB with community plugin

Before the first start:

Go to the directory where you installed the plugin (e.g. GRAFANA_DIR/data/plugins/MONGO_PLUGIN_DIR)
Install the node.js dependencies as described in the plugin documentation

To set up a data source:

In the plugin directory, run the proxy server (npm run server). By default it listens to requests from Grafana on http://localhost:3333
In Grafana:

Open Configuration -> Data Sources:
Select “Add Data Source”:
Select “MongoDB datasource”:
You will see the screen with data source settings.
Now we have to fill them in. Use the following values:
- Name: step-mongo
- URL: PROXY_SERVER_URL (default localhost:3333)
- MongoDB URL: MONGO_DB_URL (default mongodb://localhost:27017)
- MongoDB Database: step
- Additionally configure your authentication details in the Auth section
Click on “Save & Test” - you should see a green message confirming that the connection test was successful.

Congratulations! You have set up a data source that will be talking to the measurement data from step.

PostgreSQL/TimescaleDB

Connection to an existing PostgreSQL instance is done in a similar way, using a built-in PostgreSQL data source provided by Grafana.

In Grafana:

Open Configuration -> Data Sources:
Select “Add Data Source”:
Select “PostgreSQL datasource”:
You will see the screen with data source settings.
Now we have to fill them in. Use the following values:
- Name: step-postgres
- Host: POSTGRES_URL (default localhost:5432)
- Database: step
- User and Password: credentials of the account that you wish to use to connect to the DB - we recommend to set up a Grafana-specific, read-only user
- SSL Mode, Connection limits: configure as required by your needs
- PostgreSQL details:
  - Version: your PostgreSQL version, we recommend at least v12
  - TimescaleDB: if you are using TimescaleDB, enable this option to use Timescale-specific functions for time-binning and quering for better performance; consult step documentation for instructions regarding Timescale and step data
  - Min time interval: this will set minimal granularity of the time binning on the horizontal axis on all the dashboards created using this data source. If you are working with data which changes very rapidly in time and want to have ability to zoom in very deeply, you may want to set it to 1ms, which is the precision of the timestamps created by step, otherwise 10ms or 100ms should be fine as well. This setting only affects querying and plotting the data.

Congratulations! You have set up a data source that will be talking to the measurement data from step stored in PostgreSQL.

Prometheus

Configure your Prometheus data source similarly to MongoDB and PostgreSQL data sources. Look up the port where your Prometheus service resides in the Prometheus configuration (prometheus.yml). Name your data source as step-prometheus.

Automatic data source provisioning

Grafana offers you a possibility to automatically provision data sources using yaml files instead of configuring them from the UI. Below we provide you a starter template for a data source yaml file, which you can use (and modify according to your needs and exact database configuration) to provision your step data sources in Grafana:

# # config file version
apiVersion: 1

datasources:
- name: step-postgres
  type: postgres 
  url: localhost:5432
  database: tutorial 
  user: $STEP_POSTGRES_USER
  secureJsonData: 
    password: $STEP_POSTGRES_PWD
  jsonData: 
    sslmode: "disable" 
    postgresVersion: 1200
    timescaledb: true 

- name: step-mongo 
  type: grafana-mongodb-datasource
  url: localhost:3333
  jsonData:
    mongodb_url: mongodb://localhost:27017 
    mongodb_db: step

step performance dashboards

Now that you have configured the data sources, you can import the step performance dashboards. Grafana stores dashboards as json files with all the queries and panel configuration embedded.

There are 2 interconnected dashboards that make use of step data:

Execution list dashboard showing available executions in a given time frame. Clicking on an execution ID in a table redirects to the Execution overview dashboard with the execution ID, its start and end times selected
Execution overview dashboard showing measurements associated with a particular execution, within a given time frame. Clicking on a link in the top right corner redirects to Execution list dashboard

We have prepared one dashboard set per each data source, with equivalent data source queries. The only difference between the data sources is calculating percentiles and some statistical parameters, since MongoDB doesn’t provide the built-in percentile accumulator yet. For rapid percentile calculations in PostgreSQL we recommend installing the t-digest extension.

Grafana offers you multiple ways to import a dashboard. We describe few of them here:

Importing step dashboards

We provide you the following versions of the dashboards as a basis. You are free to use and extend them. We’ll be glad if you want to share your feedbacks and contributions back with us and the community, however please note that these dashboards are not officially supported and maintained as of now.

Dashboard with MongoDB data source without percentile calculation: Execution List, Execution overview, Execution overview (embedded version)
Dashboard with PostgreSQL/TimescaleDB data source: Execution List, Execution overview with native PostrgreSQL percentile calculation Execution overview with t-digest percentile calculation
Dashboard with Prometheus data source: Execution List, Execution overview, Execution overview (embedded version), Scheduled execution overview

They all have about the same UI and differ mainly in the datasource implementation. For importing them, download them first to a local directory, and import them in Grafana UI (see below). To make the interconnection between the dashboards work, you have to either preserve the UIDs, or set up the link in the dashboard variables (see below).

Importing by hand

Go to Dashboards -> Manage
Click on “Import” The following screen will appear:
Select dashboard:

Use “Upload JSON file” to choose the dashboard you downloaded and stored locally

Now you should see the following screen, where you can modify the dashboard title and control the dashboard ID number (uid). For better organization, you can also select the folder where you want to place the dashboard. If you already have a dashboard with the same uid, Grafana will notify you and give you an option to choose a different uid - it needs to be unique within your Grafana instance. Grafana uses the uid also as part of the dashboard URL and for querying its APIs, so you can use a human readable name as well. If the uid is empty, Grafana will give automatically a new, unique uid. If you change the UID, remember to adjust the dashboard settings to make the interconnection work.

Setting up the interconnection between the dashboards

If you want to change the existing UID and still have a working connection between the dashboards, or if the link between the dashboards appear to be broken (e.g. because of the bug happening in Grafana versions prior to 7.4.0), follow these steps:

Go to the dashboard settings
Go to “Variables”
Click on a variable “executionlistUID” or “drilldownUID”
Put the correct UID of your target dashboard
Update

In case of a known Grafana internal bug that prevents you from saving the variable value, you may need to delete the variable alltogether, save the dashboard, and re-create the variable again.

That’s it! You should now be able to use the step dashboards.

Provisioning

Grafana also offers you a possibility to automatically provision the dashboards. To use it with step, download the json files to a directory of your choice (defined with path property) and use this yaml file as a template:

# # config file version
apiVersion: 1

providers:
- name: 'step'
  orgId: 1
  folder: ''
  folderUid: ''
  type: file
  disableDeletion: false 
  updateIntervalSeconds: 0 
  allowUiUpdates: false
  options:
    path: PATH_TO_YOUR_DASHBOARDS_FOLDER
    foldersFromFilesStructure: true

Consult Grafana documentation for modifying the template.

Using step dashboards

step dashboard is built out of set of panels, grouped into rows. Rows may be expanded and collapsed as needed. Filtering for execution and keyword is done using a set of filters on the top.

Using dashboard variables

The dashboard variables are sets of values which can be used to choose an execution, pick one or more keywords, filter for status (errors), keyword type, and choose a metric to display on a time-trace chart.

They are located at the very top of the dashboard and cannot be hidden. They are generated using dynamic queries and are regenerated every time an eId, name or time range is selected.

Tip: If you have many executions available, you can also use the top field as a text search, or directly paste the eId number you are looking for.

Picking the time range

To display the data in the statistics tables, time traces, errors and threadgroups rows, queries only look for the data between the set time range, which is defined once for the whole dashboard. Currently selected time range is displayed in the top-right corner.

Grafana has been designed to support workflows relying on streaming data. Accessing past data by setting the time range manually is also supported, however finding your past data by hand may be cumbersome.

Here we provide you some helper functionalities to make this task easier. The time range can be changed by:

Using the time range control in the top-right corner
Using the statistics table: clicking on the name of execution ID, or on the start or end time will set the time range.
Selecting a particular sub-section of a time trace

Upon changing the time range, Grafana will recalculate the time interval used to plot the data, unless it is smaller than the minimum interval defined in your data source.

Statistics tables

Statistics contain the summary of statistics of selected keywords for a given eId. This table contains measurement of all types and statuses as picked in the filters on the top. That means that if you have some erroneous measurements, you are able to filter them in or out of this table by choosing the right Selected status.

Time traces

We provide different time traces so that you can follow the time evolution of your metrics.

Average response time per keyword
Transaction rate per keyword (stacked)
Response time vs transaction rate (only in Prometheus dashboard)
Transaction rate for all keywords together, split by status

This row lets you inspect the errors. You can filter for the measurements with a given error status either by choosing the status_current variable or by clicking on the legend.

In this panel, all the measurements from selected keywords are grouped together.

Threadgroups

As of step 3.16+, it is possible to monitor the number of threadgroups active at any point in the execution. This panel visualizes the threadgroups.

Summary

In this tutorial you have:

configured Grafana to connect to data generated by step in a database of your choice
imported a performance dashboard
visualized step data in Grafana

Happy monitoring!