XLT – Quick Start Guide

Introduction

What is XLT?

XLT stands for Xceptance LoadTest1. XLT offers a comfortable way to run regression and load tests automatically. Since XLT is completely written in Java it can be run on any platform that is supported by Java. Furthermore, your tests can profit from the rich set of features offered by the Java programming language and its libraries. Test reports can be generated using the XLT report generator which processes the test results and generates an HTML test report along with an XML file with the computed statistics.

1 We will use the abbreviation XLT throughout the document.

Which Technologies are Supported by XLT?

Nearly every software that provides access via HTTP/HTML can be tested using XLT. This is possible because XLT simulates a web browser as closely as possible while running tests. There is also extensive JavaScript support for testing applications that use Web 2.0 technologies. But XLT supports not only web testing but also SQL tests, RCP-based application tests, or any other test that should run as a load or performance test in a distributed manner.

Because XLT uses Java as its scripting language, you can basically use any technology in your tests that is available for Java.

Who Should Read this Document?

Everyone who is interested in writing regression and load tests for web applications is invited to read this document. Of course, it is all about using XLT, so this document can be seen primarily as the Quick Start Guide for XLT.

This document gives you a short overview of XLT itself, its features, and how it can be used to perform regression and load tests in a comfortable, platform-independent approach. It describes the basic steps of writing a web test suite, testing the suite itself, and finally rolling it out to perform a load test.

To get the most out of this document and to be able to work effectively with XLT, you should have some basic knowledge about web technologies, the Java programming language, and JUnit4. Being familiar with these topics will help you better understand the next sections.

Installation Instructions

Before we go on, you need to install XLT to your hard disk.

System Requirements

Hardware

Typically, XLT requires a CPU running at 1.5GHz at least and about 1GB of RAM. XLT can run on smaller hardware but this depends on the tests themselves. The default installation requires about 85MB of hard disk space, but as load test results might require much more, at least 1GB of free disk space is recommended.

Software

XLT runs on any operating system supported by Sun’s Java version 5 (or higher), such as Microsoft Windows, Sun Solaris, Linux, HP-UX, or Mac OS X. We recommend using Sun’s JDK if it is available for your platform. XLT will also run in JVMs provided by other vendors (e.g. BEA, HP, IBM), but this has not been tested extensively. In order to view the HTML load test reports generated by XLT, a modern web browser is required, such as Firefox 3, Internet Explorer 7, Opera 9 or Safari 3. Please note that JavaScript needs to be enabled to fully utilize all functionality.

Installing XLT

The installation of XLT is simple. Just unzip the XLT archive to a file system location of your choice. The root directory is part of the archive, so you do not need to create it separately. Please note that although spaces in the path are supported by XLT, it is easier to code tests when the path is free of them.

On Unix-like systems, you need to grant execution rights to all files in the bin directory that end in .sh:


chmod u+x <XLT>/bin/*.sh

Next, copy the license XML file, which you should have received separately, to the directory <XLT>/config. Note that a Basic License of XLT does not require the installation of a license file. In this case, XLT is not limited in time or functionality, but the number of virtual users is restricted to five. Also notice that in this case the Basic License Terms apply. See the file <XLT>/doc/license.html for more information.

Finally, make sure that the executable directory of your Java installation is listed in your PATH environment variable so that the XLT start scripts can find the virtual machine runtime.

In order to install the XLT Script Recorder, an extension to the Firefox browser, you need to carry out the following steps:

  1. Start Firefox.
  2. Click File > Open File....
  3. Navigate to the <XLT>/tools directory and select the .xpi file. The Add-On installation dialog should appear.
  4. Click Install to finish the installation of the Script Recorder extension.

Alternatively, drag the .xpi file and drop it onto the Firefox window to install it.

Uninstalling XLT

Before uninstalling, make sure that you have backed up all the test results and test reports that you want to save. To uninstall XLT simply delete its installation directory. If you installed the Script Recorder Firefox extension, use the Firefox Add-On dialog to remove the extension.

Writing Web Tests

This section explains the basic concepts of XLT. It also covers writing and running web tests with XLT.

XLT Basics

XLT builds upon JUnit4 principles and its annotations to implement and tag test cases. This way, each XLT test is a JUnit test, so XLT tests can be executed just like any other unit test within an existing build process. The only difference between XLT and JUnit4 tests is that XLT tests can only have one active test method per test class.

The foundation of XLT for testing web-based applications is the HtmlUnit framework, which also includes the Mozilla Rhino engine for JavaScript support. Web tests are performed by a low-level web browser simulation. Basically this means that – as in real browsers – a web page is translated to a DOM (Document Object Model) tree. Analysis, validation and any other access is performed afterwards on the constructed DOM tree.

When performing web-based application tests the possible paths from page to page define the web page flow, or the “navigation map” of the web site. The web page flow can be represented as a directed graph where the vertices are the web pages and the transitions represent the actions to get from one page to another. Typically, a test scenario covers a certain part of the page flow only, i.e. a certain path through the application.

XLT provides a programming paradigm to translate a scenario into a unit test. For this, XLT uses a three-level architecture (transactions, actions, and requests). These levels will be described in the following subsections.

Transaction

A transaction is an execution of exactly one test case or test scenario. In order to perform the scenario, the page flow is modeled in code. The test scenario is implemented as a test case class, which itself executes a sequence of one or more actions.

Action

An action can be defined as one irreducible step within a test case. For example, for a web search engine the most important action would be to “search”: to fill in the search phrase and click on “Go”, “Search” or something similar to load the list of results. So an action interacts with the “current” page and – as a result – loads the “next” page. That page is associated with this action and becomes the “current” page for the next action in the test scenario.

An action is implemented using a special Action class. Generally, an action triggers one or more requests.

Request

This level is equivalent to the familiar HTTP request level used in web browsers or in any other application that relies on HTTP communication. Since requests are automatically generated by the underlying HtmlUnit framework when performing actions on HTML elements, we will not have to deal with them directly.

Example

Imagine the following web page flow through an online shopping application: open the shop’s start page, search, browse a product from the search result, and add it to the shopping cart. Typically, the start page action has no specific preconditions except a valid entry URL. Thus, we are simply going to get the URL from a property file, get the page, and check whether or not the downloaded page matches our expectations. But note that these three steps, namely

  1. checking the preconditions,
  2. sending the request(s) to the server, and
  3. validating the returned data,

are the typical steps that constitute an action.

The action that loads the next page with the search results has a dependency. It depends on the start page being loaded successfully in the first action. Because we require the results of the first action (the start page), we pass the start action to our search action (as a constructor parameter). This ensures that the search action can access the start page and interact with it. But before it can do so, it checks the preconditions, for example that there is indeed a search phrase input field on the current page. In the second step, the search action fills in the search phrase and submits the search request form. Finally, after receiving the page with the search results, the search action has to examine the page to make sure that there is indeed a list of search results on the page as expected.

The following pages in the test scenario are loaded exactly the same way. The test completes successfully if and only if all pages in the flow have been loaded and validated successfully.

XLT API

As mentioned earlier a test class can only have one test method within it. That means that although there can be an arbitrary number of methods within a class, only one method is permitted to be annotated with Test. This limitation simplifies things, and does not result in any actual limitations in real life.

All test classes should inherit the abstract class AbstractTestCase, which supplies some basic features like logging the test results to disk and flushing all caches. The system won’t break if a test class does not inherit AbstractTestCase, but then no logging and cache flushing will be performed.

Because a test case models a transaction and transactions rely on actions, defining the appropriate actions is the first step.

All actions must inherit the abstract class AbstractAction which forces you to implement the three methods execute(), preValidate() and postValidate(). As mentioned earlier, the preValidate() and postValidate() methods are used to perform validations before and after the execution of that action. Therefore the call sequence of an action generated by the XLT framework will always be:

  1. preValidate()
  2. execute()
  3. postValidate()

This call sequence will be executed exactly once when the instance method run() of an action is called. As you might have already guessed, the run() method starts the appropriate action by executing the call sequence listed above.

As you see, XLT forces us to implement the validation methods and that is the whole purpose of testing: validating data. Thus, implementing the abstract validation methods in a non-trivial way (i.e. not leaving them empty) is strongly recommended. Otherwise you will sacrifice test quality.

Each of the three methods may throw an exception, which is always an indication for a failure. To check if an action can safely be executed, the abstract class AbstractAction provides a method called preValidateSafe(). This method internally calls preValidate() and catches any thrown exception. If no exception is thrown, preValidateSafe() returns true; otherwise it returns false.

Note that AbstractAction does not offer any web support. Therefore, any web-based test should inherit the abstract class AbstractHtmlPageAction which is a specialization of AbstractAction and offers support for web testing.

Validation

Because we are testing the functionality of an application or piece of software, we have to have a way to check the correctness of all the responses. No matter whether we run a load or regression test, the action concept forces us to write validations.

Assertions

JUnit provides the concept of assertions and we use this concept for all validations. Because XLT does not change JUnit in any way, you can use assertions just as you learned to with JUnit. We strongly recommend that you handle all potential situations and use assertions as often as possible. It is better to use too many assertions rather than too few! They can’t do any harm, and will increase your confidence that your application works correctly.

Pre-validation

Each action has a preValidate() method that checks whether or not all of the required data is available to advance to the next page. To look at it from the end user’s point of view, we simply look for the information on the page that we need to continue our Internet experience, such as a form to fill out or a link to click. In the case that the required information cannot be found, we throw an AssertionException. If we run a load test, XLT will catch it and log all relevant information. This makes it possible to evaluate the results after the test and to narrow down any error condition.

XLT offers two ways of using the preValidate() method. Any exception on the direct path will stop the test with an error message. In case we just want to check whether or not a requirement is fulfilled, we can call the preValidate in a safe way (by using preValidateSafe()), so that any exception will be caught and no error reported. Should we accidentally cause any Java exception different to AssertionException, such as NullPointerException or IndexOutOfBoundException, XLT will issue a warning, because the code might be faulty from a programming point of view. Errors from the application being tested should always come up as assertion failures.

Post-validation

The postValidate() method works similarly to the preValidate() method. It is used to validate the page just loaded in execute() and ensures that the data matches the expectations. The full set of JUnit assertions is available.

You should insert as much validation as necessary to detect any abnormal application behavior of your software being tested. For instance, when we run HTTP/HTML-based load tests, it might happen that the returned page was delivered using HTTP response code 200. This basically just means that the application responded. It does not mean that the response matches the expected result, so in this case further validation is needed. And of course, HTTP is the transport protocol, so we have to keep in mind that protocol errors do not necessarily mean application errors and vice versa.

The postValidate() method cannot be called explicitly; the framework does it for us. Additionally, its error messages cannot be suppressed.

Validators

We strongly encourage you to write individual validation classes for easy reuse. If a certain check has to be done more than once, it becomes a candidate for a validator implementation. This simplifies the maintenance of the tests and makes them less error-prone, because the copy-paste disease is one source of typical programming errors.

Some common validation routines are already covered by default validators, such as a HTTP response code, HTML end tag, and HTTP content length validation. See package com.xceptance.xlt.api.validators in the API documentation for more information on this topic.

Example

To explain the XLT API with a more concrete example, consider the previously mentioned action for a web search page. The preconditions are the existence of the search input field and of an appropriate button labeled with “Search” or “Go”. The execute() method should fill in the search phrase and click the button.

After the new page has been loaded, it should be validated. This validation consists of general validation, performed by validators, and action-specific validation.

The resulting implementation of the search action would look like this:


public class Search extends AbstractHtmlPageAction
{
    // XPath to search input field
    private static final String searchInputFieldPath = "/id('search')/input";

    // XPath to search button
    private static final String searchButtonPath = "/input[@type='submit' and @name= 'Search']";

    // XPath to search result block
    private static final String searchResultPath = "/div[contains(text(), 'result')]";

    // Search input field HTML element
    private HtmlInput searchField;

    // Search button HTML element
    private HtmlInput searchButton;

    // search phrase
    private String searchPhrase;

    /**
     * Performs a new search.
     * @param prevAction
     *            Previous action.
     * @param searchPhrase
     *            Phrase to search for.
     */
    public Search(AbstractHtmlPageAction prevAction, String searchPhrase)
    {
        super(prevAction);
        this.searchPhrase = searchPhrase;
    }

    /**
     * Validation prior to execution.
     * @throws Exception
     *             if some of the required input elements couldn't be found.
     */
    public void preValidate() throws Exception
    {
        HtmlPage p = getPreviousAction().getHtmlPage();

        List result = p.getByXPath(searchInputFieldPath);
        Assert.assertEquals(1, result.size());

        searchField = (HtmlInput) result.get(0);
        Assert.assertNotNull(searchField);

        result = p.getByXPath(searchButtonPath);
        Assert.assertEquals(1, result.size());

        searchButton = (HtmlInput) result.get(0);
        Assert.assertNotNull(searchButton);
    }

    /**
     * Executes the search. Primarily this includes the input of the search
     * phrase and a click on the proper search button.
     * @throws Exception
     *             if some of the inputs have become invalid or setting the
     *             value attribute of the search input field has failed.
     */
    public void execute() throws Exception
    {
        // Fill in search phrase
        searchField.setAttribute("value", this.searchPhrase);
        // Click on 'Search'
        loadPageByClick(searchButton);
    }

    /**
     * Validation after search has become complete.
     * @throws Exception
     *             if no search result block element could be found or the
     *             search result block element doesn't contain the search
     *             phrase.
     */
    public void postValidate() throws Exception
    {
        HtmlPage p = getHtmlPage();

        // response code = 200?
        HttpResponseCodeValidator.getInstance().validate(p);
        // does the length match?
        ContentLengthValidator.getInstance().validate(p);
        // does the page end with </html>?
        HtmlEndTagValidator.getInstance().validate(p);

        List result = p.getByXPath(this.searchResultPath);
        Assert.assertEquals(1, result.size());

        HtmlElement el = (HtmlElement) result.get(0);
        Assert.assertNotNull(el);

        Assert.assertTrue(el.asText().contains(this.searchPhrase));
    }
}

Notice that the constructor of this class has two parameters. One of them is the search phrase that the action trivially has to know about. The other parameter is the action performed previously. All actions should provide at least one constructor with a parameter representing the previously performed action to enable validations prior to action execution.

You’ll notice that the postValidate() method uses some of the predefined validators. XLT also offers a StandardValidator which performs the most common validations in one go. In detail, this includes:

Of course, each of these validations can be run separately and in a different assortment.

Having the search action at hand, the implementation of a test case using this action is already almost done.

A very simple test case would be repeatedly searching for some phrases. These phrases can be stored in a data file and obtained using the XLT data provider mechanism.


public class TSearch extends AbstractTestCase
{
    // Container that holds all search phrases
    private static CustomDataProvider phrases = null;

    @Before
    public void initialize() throws Exception
    {
        // Data container already initialized?
        if(phrases != null) return;
        // No. Go for it.
        phrases = new CustomDataProvider(
                        getProperty("searchphrases.filename", "phrases.txt"),
                        CustomDataProvider.DEFAULT);
    }

    @Test
    public void search() throws Throwable
    {
        // Start on Homepage.
        Startpage start = new Startpage();
        start.run();

        for (int i = 0; i < XltRandom.nextInt(10); i++)
        {
            // Take a random search phrase.
            String searchPhrase = phrases.getRandomRow(false);

            // Search.
            Search search = new Search(start,searchPhrase);
            search.run();
        }
    }
}

The above example also demonstrates the use of the XltRandom class, which offers some nice features regarding randomization. Note that the package com.xceptance.xlt.api.util provides many more functionalities that help in implementing the tests.

Each execution of the search action requires a proper search phrase, which is obtained from a CustomDataProvider object. This class provides a generic mechanism to handle and provide test data that is stored in a text file. The name of the text file, along with a boolean value that tells the appropriate parser whether whitespace should be removed, are passed as parameters to the constructor. When the class is instantiated all data is kept in memory, allowing easy and fast access. XLT is shipped with a predefined set of data files which contain email addresses, first and last names, street and city names, and so on. This data can be obtained from the GeneralDataProvider class which uses the appropriate text files located in directory <testsuite>/config/data where <testsuite> refers to your test project directory.

The example also demonstrates the use of JUnit4 annotations, which can be used in the standard manner.

The full documentation of the XLT API can be found in the directory <XLT>/doc/apidoc.zip.

Demo Application and Test Suite

XLT ships with a real-world demo web application as the system under test (called “Pebble” )and a test suite to test this application. Both can be found in the directory <XLT>/samples.

Pebble is a blog software written in Java and licensed under a BSD license. Because Pebble is small and easy to deploy, it is a perfect fit for our test application.

Please do not use the provided installation for purposes other than testing, because it has been modified to run without problems from the sample directory. Feel free to download the latest version.

Thanks to Simon Brown for this excellent piece of software and licensing it as Open Source.

Running Pebble

To start the demo application, use the following command:


cd <XLT>/samples/app-server/bin
./start.sh

Windows users have to use the appropriate .cmd file which is located in the same directory.

This will start an application server which in turn starts the Pebble application. To access Pebble, simply open a browser with this URL: http://localhost:8080/pebble/.

Now take the time to become familiar with this application since the following sections will assume that you are.

Importing the Pebble Test Suite into Eclipse

Users of other IDEs have to carry out similar steps.

After starting Eclipse and creating a workspace, if you have not already done so, you have to import the sample test suite that is part of XLT. Open the import dialog in Eclipse and point to the samples directory in your XLT installation directory (File > Import > Existing Project Into Workspace).

  1. Select the root directory to search in, and point to <XLT directory>/samples.
  2. Select the test suite project from the list.
  3. Click Finish and you are done.

Because the imported project has dependencies to the libraries of XLT, we have to adapt these dependencies to your machine.

  1. Simply right-click on your project and select Properties.
  2. Choose Java Build Path > Projects.
  3. Remove the required project xlt, if there is such a project configured.
  4. Select Libraries and click Add External JARs (if there are already other libraries besides the “JRE System Library”, remove them first).
  5. Go to <XLT>/lib and select all JARs. Then click Open.
  6. A list of all these JARs should be visible now. Close the dialog with OK.

Eclipse will rebuild the project now and should not bring up any build problems if configured properly.

The Directory Structure of the Test Suite

An XLT test project has a simple directory structure. The following directories have to exist to make everything run smoothly:

<project>/classes contains the compiled code of your project. Normally your favorite development environment will do the job and place the files there. Optionally, you can build a JAR and place it in the <project>/lib directory.

<project>/config contains all the properties files used to configure the project.

There can be an optional <project>/config/data directory, where you can place any data file you need for the test, such as address data, logins and so on. All files will be uploaded to the agent before a load test takes place. The programming API provides a way to access this data easily.

You can place all your required libraries in <project>/lib. The content will be uploaded to the runtime agent and included in the class path. For your local development within an IDE, you have to add the libraries manually to your project; for help with this, please consult your IDE manual.

The <project>/src directory holds the source code of your project. This code will be compiled into classes by your IDE or build environment. The code is organized in main packages, typically one for test cases, one for the actions, and one for utility classes.

Understanding the Test Scenarios

As Pebble is a blog software, the test scenarios have to cover the typical use cases of a blog:

Note that both scenarios share some common steps, thus allowing us to demonstrate the re-use of code in different test cases.

Run the Tests

Any test case can be run directly from within Eclipse. Just select the test case class, for example TAuthor, and run it as JUnit test via the context menu. This mode is often used during development of the test cases.

Write Your Own Test Suite

We encourage you to study the demo test suite carefully with respect to the approach as well as the code. Since all web tests are similar in structure, you will soon get a feeling for how to write your own tests. The sample test suite can serve as a template for your own test projects.

Performing Load Tests

Now that you know how to write web tests, we will go a step further and explain how to turn these tests into load tests.

Understanding the Load Generation Environment

Typically, a distributed load generation environment is needed to generate enough load. For this we need a cluster of test machines. Install XLT on all these load machines.

XLT ships with several runtime components, namely:

The master controller is the brain of the load test environment. It deploys the test suite to all load machines, distributes the load evenly, and starts and stops the load test. There is only one master controller in the test cluster.

Since the master controller does not have direct access to the remote load machines, it needs a counterpart on these machines, which is the agent controller. The agent controller acts on behalf of the master controller. Typically, there is one agent controller per machine.

The agent is the component that actually executes the test suite against the system under test. The agent is started by the agent controller.

Configuration

Before you can start the load test, some configuration needs to be done. The configuration of XLT itself is discussed in the next section, followed by an explanation of the configuration of your test suite.

Configure XLT

The configuration of the XLT load generation runtime is separated into three files that set up the main components of XLT:

The first property file is used to configure the agent controller of the appropriate test machine. This includes

The second configuration file contains the properties of the master controller. Edit this file if you need to adjust

Last but not least, you can adjust the report generator to meet your needs by changing the appropriate properties in the reportgenerator.properties file. These include

Configure Your Test Suite

The test suite configuration is made up of at least two configuration files which are included in this order:

The first property file is used to configure the general behavior of the the XLT test framework. These include

To configure your test project you’ll have to edit the file named project.properties. There you can specify

Whether or not you need test-run-specific configurations, you can have any number of additional property files in your <testsuite>/config directory. One of them will be included when you specify its name in the com.xceptance.xlt.testPropertiesFile property in the project.properties file. Please note that you can specify only one property file per test run.

In case you want to modify the behavior of the logging facility of the load test agents, the test suite configuration directory contains a file named log4j.properties, which can be changed to meet your needs.

In case the Java Virtual Machine which the agents will run in should be launched with additional parameters, specify them in a file named jvmargs.cfg.

Run the Load test

Make sure all agent controllers are up and running on all load test machines:


cd <XLT>/bin
./agentcontroller.sh

Also check that the test suite has been compiled successfully.

Interactive Mode

Start the master controller with the following command line:


cd <XLT>/bin
./mastercontroller.sh

A screen like this will appear:


Xceptance LoadTest 3.2.0 (Build 1) - 1000 user license
Copyright (c) 2005-2009 Xceptance Software Technologies GmbH. All rights reserved.


(u) Upload agent files
(s) Start agents
(a) Abort agents
(d) Download test results
(r) Show agent report
(q) Quit
=>

You have a choice as to what to do next. The following list describes the available options.

Once you have chosen an option (by pressing the associated key followed by ENTER), the appropriate action is executed. Afterwards you will return to the menu immediately (unless you have chosen to quit, of course).

A typical usage scenario for a load test is as follows:

  1. Upload the test suite.
  2. Start the agents.
  3. Check the agent status regularly.
  4. Download the test results once the test has finished.
  5. Quit the master controller.

However, this may vary. Especially if your test suite or your test environment is not stable yet, you may want to skip one or more steps.

Auto Mode

As you have seen in the previous section, there is a typical sequence of steps to be executed when running a load test. It may quickly become tedious and error-prone to type the necessary keys over and over. To avoid this repetition, XLT provides another operating mode: the auto mode. In this mode, all the steps mentioned above are executed in succession, without user interaction. To start XLT in this operating mode, use the following command line:


cd <XLT>/bin
./mastercontroller.sh -auto

If the test suite files were uploaded and the load agents started successfully, XLT automatically refreshes the agent status regularly. Once the test has finished, the test results are downloaded. Afterwards, XLT quits. See below what the screen shows in that mode:


Xceptance LoadTest 3.2.0 (Build 1) - 1000 user license
Copyright (c) 2005-2009 Xceptance Software Technologies GmbH. All rights reserved.

Uploading agent files ...
 -> Agent Controller ac1 <https://localhost:8443> ... OK

Starting agents ...
 -> Agent Controller ac1 <https://localhost:8443> ... OK

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Running        10 of     10            0      0,00 s      0,00 s      0:00:00           0        0         0%
TAddToCart      Running        10 of     10            0      0,00 s      0,00 s      0:00:00           0        0         0%
TCreateUser     Running        10 of     10            1      0,72 s      0,72 s      0:00:01           0        0         0%

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Running        10 of     10           72      0,67 s      0,77 s      0:00:06           0        2         5%
TAddToCart      Running        10 of     10           55      0,70 s      1,03 s      0:00:06           0        0         5%
TCreateUser     Running        10 of     10           83      0,95 s      0,67 s      0:00:06           0       17         5%

.
.
.

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Running        10 of     10        1.472      0,69 s      0,66 s      0:01:37          17       65        96%
TAddToCart      Running        10 of     10        1.412      0,66 s      0,68 s      0:01:37           0        0        96%
TCreateUser     Running        10 of     10        1.525      0,91 s      0,63 s      0:01:37           0      316        96%

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Finished        0 of     10        1.533      1,16 s      0,65 s      0:01:41          17       65       100%
TAddToCart      Finished        0 of     10        1.476      1,17 s      0,68 s      0:01:40           0        0       100%
TCreateUser     Finished        0 of     10        1.590      0,79 s      0,63 s      0:01:41           0      325       100%

Downloading test results ...
 -> Agent Controller ac1 <https://localhost:8443> ... OK

To abort the test prematurely, press CTRL-C to terminate the master controller. This terminates all running agents as well and triggers the download of all test results generated so far. This also means that it is not possible to disconnect the master controller from the test cluster while keeping the load test running.

Create the Test Report

Once you have downloaded the results of the load test to your local disk, you may want to go ahead and create the test report. The test report contains the following pieces of information:

All this information is rendered to an HTML page that can be viewed with a standard web browser. The page is enhanced with JavaScript to improve the navigability and the organization of information. To start the report generator, enter the following command in your shell:


cd <XLT>/bin
./create_report.sh ../results/<timestamp> [-o <outputDir>]

where <timestamp> is to be replaced with the appropriate value.

This tells the report generator to take the specified results directory as input for the report. By default, the report is generated to a directory beneath <XLT>/reports. This sub directory is named the same as the respective results directory. Once the report is generated (which may take a while depending on the amount of data gathered during the load test), you will find the file testreport.html in the root of the appropriate test report directory. Open this file in your web browser to view the report.

With the -o option, you can specify a different output directory. Please keep in mind, that you have to specify it including the final directory for your report, because with the -o option, the directory name is not set automatically.

Evaluate the Test Report

In the top-right corner of your test report, you find the navigation area. It contains quick-access links for each section in the report. Every section contains

The description paragraph contains some tips about the displayed information and how it can be evaluated. There are usually multiple charts stacked on top of each other. To switch to another chart, use the tab bar.

Terminology

Action

An Action represents a self-contained logical step in a test case. Typically, Actions are re-usable building blocks which can be used across multiple test cases. Actions themselves may issue one or more requests.

Agent

An Agent simulates a group of virtual users, which repeatedly execute certain test cases against the system under test.

Agent Controller

An Agent Controller controls one Agent.

Master Controller

The Master Controller controls all Agent Controllers and gets the results, the logs and the pages generated by the agents from the storage of the Agent Controller machines.

Property File

Property Files are text files that contain Java properties. A Java property represents a mapping from a symbol (property name) to a value (property value).

Request

Typically, a Request represents one call to a (remote) server. It does not matter what protocol is used to contact the server.

System Under Test

The System Under Test is the application being tested. In the XLT demo, for example, this is the Pebble blog application.

Test Case

A Test Case is the program that models a transaction.

Transaction

A Transaction represents one execution of a certain test case. Typically, a transaction uses multiple actions to model the test scenario.

User

A User is one incarnation of a client that interacts with a server. During a load test, multiple Users will operate in parallel to simulate a multitude of human or technical users interacting with the target system at the same time. Typically, Users are configured to execute a certain test case repeatedly.

Virtual User

See User.

Acknowledgments

This product includes software developed by Andy Clark.

This product includes software developed by The Apache Software Foundation (http://www.apache.org/).

This product includes software developed by the Caucho Technology (http://www.caucho.com/).

This product includes software developed by Gargoyle Software Inc. (http://www.GargoyleSoftware.com/).

This product includes software developed by Harald Kirschner (mail@digitarald.de).

This product includes software developed by The Jquery Team (http://www.jquery.com/).

This product includes software developed by Alex Gorbatchev (http://alexgorbatchev.com/wiki/SyntaxHighlighter).