Performance Comparison Report

The overview section shows some general information about both load tests. This enables you to compare settings, runtime, and profiles. The initial test run is labeled baseline. The test run that is compared to the baseline is labeled measurement run.

In the later sections, the percentage values show you the development of the performance in comparison to the baseline. Please note that the total columns (total throughput and total errors) might present misleading values if the two load tests used different runtime configurations. All other values are normalized in respect to the runtime and therefore easily comparable. Positive numbers in the count section mean an improvement of the throughput over the baseline. Negative values indicate a decrease of throughput. An increase in the number of errors is indicated with positive numbers, while a decrease in errors is shown in negative numbers. An infinite sign indicates the occurrence of errors in comparison to an error-free baseline.
For all runtime numbers, positive values indicate a poorer performance, while negative value show an improvement - smaller runtime values - over the baseline.

Added or removed transactions, actions, or requests are indicated. No comparison is provided for these.

A transaction is a completed test case, and a test case consists of one or more actions. The runtime of a transaction contains the runtime of all actions within the test case, think times, and the processing time of the test code itself. If the test path of the test case is heavily randomized, the runtime of transactions might vary significantly. The average runtime shows the development of tests over time and is especially helpful in evaluating the outcome of long-running tests.

*) numbers might be projected

An action is part of a test case and consists of a prevalidation, an execution, and a postvalidation. While the prevalidation ensures that the necessary data is available for the execution of this test step, the postvalidation evaluates the results and compares them to the expectations. The data shown here is the time spent in the execution routine of an action. Therefore its runtime includes the runtime of a request, e.g. an http operation, and the necessary time to prepare, send, wait for, and receive the data.

*) numbers might be projected

The request section is the most important statistical section when testing web applications. It directly reflects the loading time of pages or page components. Each row holds the data of one specific request. Its name is defined within the test case as timer name. The Count section of the table shows the total number of executions (Total), the calculated executions per seconds (1/s) and per minute (1/min), as well as projections or calculations of the executions per hour (1/h) and day (1/d). The Error section shows the total number (Total) of errors that occurred during loading of the page or page component. The error count does not include errors detected during the postvalidation of the data received. Typical error situations are http-response codes such as 404 and 505, timeouts, or connection resets.

The runtime section of the table shows the median, the arithmetic mean, and the minimum and maximum runtime encountered, as well as the standard deviation of all data within that series. The runtime segmentation section shows several runtime segments and the number of requests within the segment's definition.

If the runtime of the test case is shorter than a displayed time period--for example if the test runtime was 30 minutes and the time period is in hours--the numbers are a linear projection. That means they show a possible outcome of a longer test run, assuming load and application behavior remain the same.

*) numbers might be projected

The custom timers includes all timers that have been placed individually within the test code. The chart and data description is identical to the request section.

*) numbers might be projected