From the execution table view, clicking on an execution will redirect you to the execution details, which are composed of the below views.

Execution steps

This table display the test plan execution steps in details, by default in reverse chronological order (last executed step first).

Again, you can use the table headers to filter the execution steps table and as well change the chronological display order :

If you are using TestSet and TestCase object like the above example, you can see that the execution steps are grouped by test case : you can open a test case details by clicking on it :

In addition, you can re-execute only selected test cases instead of the whole test plan by ticking its checkbox and click on the execute button :

You can toggle the display of the step execution details as follow. For Keywords it will show you for instance the input and output from the keyword call as well as the details of the keyword measurements:

(1) Clicking on the right icon next to a measurement will bring you to the performance interactive analysis for this measurement.

Finally, you can open directly an execution step in the Execution tree (see below) by clicking on its associated button :

Execution tree

The table display your test plan execution steps as a tree in chronological order. Any node can be expanded to display its content :

Performance

The performance tab of the execution view will provide you with the most frequently required metrics (keyword throughput, average response time, etc) and filters out-of-the-box. These metrics provide both statistic over time as well as in a summary form, covering the entire execution. Furthermore, the different charts and tables can be refreshed in real time or browsed as in a stationary version of the view. Lastly, for more advanced queries, drill-downs based on custom dimensions or raw measurement browsing, an interactive session can be opened.

Real-time monitoring

For on-going execution, the performance view will be automatically covering the test current time range and be refreshed every 5 seconds, as illustrated on the screenshot below.

You are presented, from top to bottom and left to right with the following information:

• a performance overview chart showing aggregated results overall measurements
• an aggregates statuses chart showing the overall calls grouped by statuses
• a response times chart by measurements showing by default the average, the metric can be changed using the button at the top right of the chart
• a throughput chart by measurements, the total throughput is displayed as bar chart and using the right axis
• a summary table of all metrics applicable to the measurements (including percentiles, median, counts, etc), aggregated over the selected time interval
• a chart showing the number of parallel threads over time

Interacting with the dashboard

• You can change the refresh rate for running execution using the dedicated picker at the top left of the view.
• The time range selection can be done at different level and will be applied to all dashboard widgets
  • use the top time range widget
  • use the top time range picker
  • zoom in and out directly in the charts
• Most important filtering and grouping can be done directly using the top filter selections. The measurements to be displayed or hidden can be selected from the statistics table or be configured in the charts options.

Interactive session

While we believe the real-time and summary views along with time-based selection will cover most analysis needs, users also have the option to further investigate and mine their data by starting an interactive session. From this view, you will also be able to export raw measurements to a file in case you’d like to use a 3rd party tool to process the data.

The “Interactive session” button in the top left corner of the view will take you to the following view which an instance of the “RTM dashboard” (originally for Response Time Manager).

The response time manager offers data selection, aggregation and visualization capabality, allowing users to filter and drill down into custom measurements in order to diagnose complex issues.

Measurement Selectors

By default, two text filters will automatically be set in the default Measurement selector in order to only display data relating to the execution you had opened prior to clicking the interactive session button (eId=a_certain_execution_id) and focusing by keywords by default (type=keyword). However, various other attributes such as rnStatus (providing information on the keyword’s execution status) can be used to filter data.

For instance, by removing the eId filter and instead adding a date filter using the begin attribute (which points to the timestamp of each measurement), users can perform cross-execution queries and visualize data over multiple executions. In this example, we will query the raw measurement values of all executions over the last hour:

Standard Selectors

While the list of selectors depends on the custom measurements that you choose to create, below is a list of standard selectors and some of the possible values:

• begin: start time of the measurement
• eId: execution id of your test execution
• planId: id of the executed plan
• taskId: id of the related scheduled tasks if any
• project: id of the related project if any
• name: name of the measurements (either the keyword name or the custom name given while creating the measurement)
• origin: name of the parent keyword measurement for custom measurements (same as name for keyword’s measurement)
• type: type of the measurements (by default ‘keyword’ for automatic keywords measurements and custom for all others)
• rnStatus: status of the transactions (PASSED, FAILED, TECHNICAL_ERROR…)
• value: the value of the measurements (long number)

Service Configuration

Different types of queries can be sent to different types of services in order to browse measurement data. Currently and by default, two main services are made available:

• Raw values: this service allows users to browse individual measurements, which is great for investigating fine grained anomalies or just making sure that custom measurements are working as intended
• Aggregates: this service performs overtime or summary-style aggregations based on dynamic criteria such as the name of a measurement, its status or any additional metadata attribute that was provided through the measurement API.

These services can be toggled by using the following button:

In either scenario, you can fire a new query (and subsequent aggregation) by clicking the Search button on either side of the dashboard:

The top part of the configuration dashlet contains service-specific configuration whereas the lower part contains configuration which is global to all services.

Raw values

As mentioned above, by default the raw measurement values are displayed upon loading the dashboard. These values are however paged and presented in groups of 100 values. In order to browse the values, you can use the small arrows available on both sides of the Chart:

The table will update automatically.

Another important point to note is that the scattered value chart will always display the value of the measurement, while the table can also display metadata attribute values. These can be selected using the Table metrics dropdown selector:

Aggregates

In order to perform over-time or summary aggregation, two key inputs must be kept in mind and are exposed under the tab “Grouping” of the Service configuration pane:

• interval size: tells RTM how large the aggregation intervals need to be and indirectly, how many datapoints will be returned and displayed. Defaults to “auto” which will automatically compute an aggregation with a target of 30 datapoints. However, arbitrary intervals can be provided in milliseconds, and the value “max” will compute an aggregation with a target of a single datapoint (also called summary mode).
• split attribute: tells RTM how to split series (or group measurements for aggregation). By default, a distinct series is created for each keyword name, however, any attribute can be used for grouping & splitting (such as rnStatus for error analysis or an empty or non-existing attribute for merging all keywords into a single series). Mutliple attributes can even be combined with a “;” for drill-down oriented queries. For instance: entering name;rnStatus will lead the computation of a distinct series for each keyword name and status combination.

Additional advanced parameters are made available under the Perf tab. These allow for the partitioning and parallel computation of complex queries. The default values should only be increased if sufficient hardware is available on the backend (please contact our support for more specific information).

Much like in Raw values mode, various metrics can be selected for display in the table below. These, however are not related to metadata attributes, but are numeric results of computations based on the measurement’s values (for instance, mean/average, counts, sums, percentiles, etc).

Note: the actual computation of metrics is not yet configurable as of step 3.13 but the underlying framework allows for extending metrics and we plan on exposing metric computation to our users in the future. New metrics such as standard deviation or concurrent transaction count, and new services such as a service allowing the visualization of the entire response time distribution or all percentile,s are also on our roadmap.

Global parameters

• Table sort: allows to organize datapoints based on time series or grouping series (keywords) first
• Table export: performs a CSV export of the table content and downloads the file (see Exporting measurements for raw measurements export)
• Global parameters > Fields: the name of the key fields used for aggregation and browsing, should not need to be modified as part of step’s use cases

Visualizations

In the RTM dashboard, the idea is to keep both a chart and a table visualization in sync at all times, regardless of whether the user is browsing raw values or aggregating data.

Lastly, a CSV export of the table can be ordered by clicking the export button of the service configuration pane.

Exporting measurements

In case you’d like to use your own toolset to process the raw data, you can use the export functionality. The exported data still depends on the provided selectors.

Error

The error view display an overview of the errors (grouped by error type) which occurred during the execution :

You can jump directly to the nodes producing the error by click on the error type :