ANSTE User Manual

  1. Installation
    1. Selenium
  2. Testing with Anste
  3. Writing scenarios
    1. Base images
    2. Scenarios
    3. Setup scripts
    4. Package profiles
  4. Writing tests
    1. Test suites
    2. Script-based tests
    3. Web application tests
  5. Running ANSTE
    1. Commandline options
    2. Configuration file

Installation

You can get the last snapshot of anste from the Subversion repository with:

svn co svn://pubsvn.warp.es/anste/trunk/ anste/

That repository is under continuous development, so use it at your own risk.

After you have downloaded the code you can execute the following comands to install the different ANSTE components: 

make install-anste
make install-anste-manager
make install-anste-job

Note that this installation method doesn't copy the config files to /etc/anste, you'll have to copy and modify them manually. There are example config files at anste/src/data/conf/*.conf.

The syntax of these configuration files is explained in the 'Configuration file' sections of this manual.

You can also build your own Debian packages from the source by doing the following: 

make deb

Selenium

In order to run web UI tests, you will need Selenium, download IDE and RC from http://seleniumhq.org/download/

  • Selenium IDE is a Firefox extension distributed as a autoinstalable .xpi file.
  • Selenium RC is a package with the connectors for different languages and platforms. Only the file selenium-server.jar is needed for anste, copy it to any location and specify the full path in the rc-jar option of the selenium section in /etc/anste/anste.conf.

Testing with Anste

Anste needs for testing two main things: the definition of the running enviroments (scenarios) and the tests. The next sections will explain how to write them.

In the last sections you will find information about anste and about its configuration file.

Once you have installed Anste, if you want to create a collection of test suites for your project, a basic directories structure is needed to organize all the files:

  • files/: Files that will be included in the deployed testing environment. This is thought for small files that tests could need during their execution.
  • images/: XML files with the definition of the base images.
  • profiles/: Plain text files with lists of packages that will be included in some of the base images or in the testing scenarios.
  • scenarios/: Definition of the hosts and network elements implied in each scenario and their network interfaces. Various scenarios can be stored in the same directory inside this one.
  • scripts/: Scripts used during the configuration of the base images or the virtual hosts. As we'll see, those scripts can be executed before (pre-install) or after (post-install) the installation of packages and profiles.
  • tests/: Specification of the Test suites. Currently, script-based and web application tests are supported.

A common/ directory is also recommended to store the files shared between tests.

Writing scenarios

The test scenarios are written in XML. Each scenario representation consists of its own scenario file and another base image description file for each of its hosts.

Base images

A base image is a machine snapshot with certain parameters such as disk size, RAM memory, operating system, package configuration, etc.

Once a base image is created, it can be copied several times for all the machines sharing the same configuration.

These machines can be set up after the copy with different newtork configurations, aditional package installations and so on.

The following example shows a base image description file: 

<?xml version="1.0" encoding="UTF-8"?>
<image>
    <name>baseimage-ebox</name>
    <desc>eBox base image</desc>

    <memory>256</memory>

    <size>800M</size>

    <swap>256M</swap>

    <install method="debootstrap">
        <dist>sarge</dist>
    </install>

    <packages>
        <package>postfix</package>
        <profile>ebox</profile>
    </packages>

    <pre-install>
        <script>write-ebox-sources.sh</script>
    </pre-install>

    <post-install>
        <script>add-services-to-ebox-firewall</script>
        <script>ebox-non-ssl.sh</script>
    </post-install>
</image>

The base image description files are saved in the images subdirectory of the ANSTE data path. By default this path is /usr/share/anste but it can be set to any other location. The files can have any name with the only restriction that must end with the .xml extension.

The elements contained in the file are described below:

name
Contains the name of the base image, it will be used for the hostname of the system. This names will be changed later in each copy of the image.
desc
Contains a description of the image. Only significant for the information messages.
memory
Contains an integer that specifies the amount of RAM memory reserved for this image.
size
Contains the capacity of the disk. The number is followed by a M (MegaBytes) or a G (GigaBytes).
swap
Contains the size of the swap partition (optional).
install
Contains the installation method in the method attribute. The available methods (for Xen) are: debootstrap, rpmstrap, rinse, copy or tar. In case of the copy and tar methods the source subelement specifies the path of the directory or tar file to copy. The debootstrap method contains the dist subelement with the name of the distribution to be installed.
packages
This element can contain two different subelements. The package subelements contains the name of a package to be installed. The profile subelements contains the name of a profile with a list of packages to be installed. This will be explained later.
pre-install
Contains a set of script subelements with the scripts to be executed before the packages installation.
post-install
Contains a set of script subelements with the scripts to be executed after the packages installation.

Scenarios

A scenario is a set of machines, each of they with a given base image and a certain software and network configuration. Similar to the base image description files, the scenario ones are also .xml files stored in the scenarios subdirectory of the ANSTE data path.

As an example of a complete scenario description is way bigger than an image base, it's shown broken down below.

The general structure is: 

<?xml version="1.0" encoding="UTF-8"?>
<scenario>
    <name>eBox and client</name>
    <desc>
    One eBox machine and a user client machine
    </desc>

    <host>
    ...
    </host>

    <host>
    ...
    </host>
</scenario>
name
Contains the name of the scenario. Only significant for reporting purpose.
desc
Contains a detailed description of the scenario. Only significant for humans reading it.
host
This element contains all the information needed to deploy each host of the scenario on its virtual machine.

Here is an example of the host elements of the above scenario: 

<host>
    <name>ebox-server</name>
    <desc>eBox server</desc>
    <baseimage>sarge-ebox-base</baseimage>
    <memory>512</memory>

    <network>
    ...
    </network>

    <post-install>
        <script>ebox-import-network.sh</script>
        <script>ebox-wait-start.sh</script>
    </post-install>
</host>

<host type="router">
    <name>internet-router</name>
    <desc>Host that assumes router functions</desc>
    <baseimage>sarge-mini</baseimage>

    <network>
    ...
    </network>
</host>
type
This attribute can contain the "router" value in order to specify that the machine will act as a router, allowing packet forwarding between its interfaces. This kind of virtual machines are of special interest when you want Internet access from other virtual machine.
name
Contains the name of the image that will be copied phisically on the disk and also used for the hostname on the virtual machine.
desc
Describes the role of the host in the scenario. Only used for documentation purposes.
baseimage
Refers to the base image .xml file, located on the images directory of the ANSTE data path. The filename must be written without the .xml extension.
memory
Contains the ammount of RAM memory (in MegaBytes) for the virtual machine. This element is optional, if not present the ammount of RAM will be the same as defined for the base image.
network
Contains the network configuration for the host. Explained below in this section.
post-install
Contains a set of script subelements with the scripts to be executed after the packages installation. Same as in the base image definition. Also the host could contain a pre-install section.

Here is an example of the contents of a network element: 

<network>
    <interface type="static">
        <name>eth1</name>
        <address>192.168.2.1</address>
        <hw-addr>00:19:D2:62:5F:4D</hw-addr>
        <netmask>255.255.255.0</netmask>
    </interface>
    <interface type="static">
        <name>eth2</name>
        <address>192.168.3.1</address>
        <netmask>255.255.255.0</netmask>
        <gateway>192.168.3.128</gateway>
    </interface>
</network>

Each network element is made of one or several interface elements. These are the attributes and subelements of an interface:

type
This attribute can be set to "static" or "dhcp" to specify if the interface is configured statically or dynamically. In the case of a dynamic interface, the address, netmask and gateway elements should not appear.
name
Contains the name of the interface in the system. Tipically ethX.
address
Contains the IP address assigned to the interface.
hw-addr
Contains the MAC address assigned to the interface.
netmask
Contains the network mask.
gateway
Contains the IP address of the default gateway, only one of the interfaces inside the network block must include this element.

The 'network' element can also contain routes for more complex network configurations, as it can be seen in the following example: 

<network>
    <interface>
    ...
    </interface>
    <route>
        <destination>192.168.45.0</destination>
        <gateway>*</gateway>
        <netmask>255.255.255.0</netmask>
        <iface>eth1</iface>
    </route>
    <route>
        <destination>default</destination>
        <gateway>jimbo</gateway>
        <netmask>0.0.0.0</netmask>
        <iface>eth1</iface>
    </route>
</network>

The route elements are:

destination
Contains the destination of the route, the value can be a network address, a host address, or default in case of the default route.
gateway
Contains the gateway address that must be used to reach the destination. This element is optional and has a default value of '*'.
netmask
Contains the network mask for the route.
iface
Contains the interface that must be used in the route to send the packets through.

The following example is the configuration for the another host in the scenario: 

    <host>
        <name>test-client</name>
        <desc>simple client host</desc>
        <baseimage>sarge-mini</baseimage>

        <network>
         <interface type="static">
            <name>eth1</name>
            <address>192.168.2.2</address>
            <netmask>255.255.255.0</netmask>
            <!-- ebox is the gateway -->
            <gateway>192.168.2.1</gateway>
         </interface>
        </network>

        <packages>
            <package>wget</package>
        </packages>
    </host>

As you can see, besides the pre-install and post-install blocks we have a packages block that can contain package names or package profiles in the same way of the base image definition.

Setup scripts

The setup scripts referenced in the pre-install and post-install blocks can be written in any language as long as the target machine has the software needed to run them.

The scripts are stored in the scripts directory inside the ANSTE data path. They don't need to have execution permission, it will be granted once transferred to the target machine.

As seen in the previous sections, this scripts are used before and after the package installation in both base images and hosts.

Package profiles

The package profiles are simply plaintext files in which each line corresponds to a package name to be installed.

They are stored in the profiles directory inside ANSTE data path and are referred from a profile element inside a packages block in both base images and hosts definition files.

The following is a example of the profiles/ebox file that contains the list of packages of eBox Platform: 

ebox
libebox
ebox-network
ebox-objects
ebox-mail
ebox-mailfilter
ebox-firewall
...

Writing tests

A test is a set of operations to be executed in a virtual machine of a given scenario. Checking the results of these operations we know if the system is working as expected. Despite of a test is runned on a single machine, usually these test use to create network connections to other machines of the scenario, or even to external machines.

Test suites

A test suite is a set of related tests that are executed in the same scenario, in an specific order.

Each suite have a separate directory, with a suite.xml file inside and one subdirectory for each of its tests.

All the test suites are contained in the tests directory of the ANSTE data path.

We can also have sets of suites recursively. If a directory contains a file called suites.list instead of the suite.xml file it means that is a set of suites, each line of the file will be the name of a subdirectory containing a suite or another set of suites.

Here is an example of a suite.xml file: 

<?xml version="1.0" encoding="UTF-8"?>
<suite>
    <name>eBox Proxy tests</name>
    <desc>
    Contains a set of tests to check
    that the eBox HTTP Proxy module works properly.
    </desc>

    <scenario>ebox/proxy.xml</scenario>

    <test type="selenium">
        ...
    </test>
...
</suite>

The elements that appear inside the suite are the following:

name
Contains the name of the suite as will appear in the reports.
desc
Contains a more detailed description of the purpose of the tests.
scenario
Refers to the description file of the target scenario that will be deployed before starting the tests execution.
test
Each test element contains the data of a test and may contain an special type attribute which currently only can be set to selenium when dealing with Selenium web interface tests. The rest of subelements are the same in both normal and Selenium tests.

Here are two examples of test elements: 

    <test type="selenium">
        <name>ConfigNetworkSetExternal</name>
        <desc>
        Sets the interface as external.
        </desc>

        <host>ebox-server</host>
        <dir>set-external</dir>
        <var name="IFACE" value="eth2"/>
    </test>

    <test>
        <name>FilterAllowDownload</name>
        <desc>Try to download a file of an allowed host</desc>

        <host>test-client</host>
        <dir>test-allow</dir>
        <var name="HOST" value="www.google.com"/>
    </test>

The content of each test subelement is the following:

name
Contains the name of the test as will appear in the results report. It needs to be a valid filename (e.g., not contain slashes).
desc
Contains a detailed description of what the test does.
host
Contains the hostname of the virtual machine that will execute the test.
dir
Contains the name of the subdirectory inside the suite's directory that contains the test itself. The content of this directory and the differences between script-based and selenium tests is explained on the two next sections.
var
Used to pass variables to the test, the attribute name contains the name of the variable and the value one contains the string that will be used in the variable substitution.

Script-based tests

The content of the directory of a script-based test, must be a script file named test and optionally there could be two pre and post scripts to be executed before and after the test one. The scripts can be written in any language supported by the target machine. The only requirment is that the test file must return a valid exit code (0 if the test was sucessful and another value if test failed).

The following example is one of the test scripts of the ebox/proxy suite partially shown above. The path of the file ebox/proxy/test-deny/test relative to the ANSTE data path. 

#!/bin/bash

echo "Setting the proxy"
export http_proxy="192.168.2.1:3128"

for i in `seq 1 10`
do
    [ -f output.html ] && rm output.html

    echo "HTTP request to $HOST"
    wget -r -t 3 $HOST -O output.html

    echo "Checking if access was denied"
    grep "Access has been Denied" output.html

    if [ $? = 0 ]
    then
        echo "Access denied."
        exit 0
    else
        sleep 2
    fi
done

echo "Test failed (timeout)."
exit 1

As you can see, we're using the HOST environment variable which value is set in the suite.xml for each test, so we can have a generic test script that is used in several different tests.

Web application tests

The directory content of a web-based test consts of a mandatory suite.html file, and a set of .html files with any name referenced by suite.html.

These files are respectively a Selenium test suite file and Selenium test case files. The first is an HTML table with a index of the test cases contained in the suite, the rest are HTML tables that contain Selenium instructions used for automagically controlling of a web browser without user interaction (e.g., open a webpage, type some text in field 'foo' of the page and then click on button 'bar'). Selenium also allows including assertion instructions that are really useful for our testing purposes (e.g., after clicking the button and received the server response, verify that the text 'foo' is present on it).

Complete reference of the Selenium test commands

These files can be written manually but there is a Firefox extension called Selenium IDE that can be used to automatically record the tests while you normally use the web application.

If you want to use web-based tests with ANSTE you need to download Selenium Remote Control.

Running ANSTE

The execution of anste program requieres root privileges. This is due to the virtualization tools (only Xen at the moment) that ANSTE relies on.

The three main use cases of the anste program are:

  • Creating a base image.
  • Deploying a scenario.
  • Running a test suite.

Running a test suite is the most common use of ANSTE, this option depends of the two others: first the scenario of the suite must be deployed in order to run the tests, and all the base images of the scenario have to be created if not already exist.

Sometimes it is also useful to only deploy a scenario and manually run tests on it, even only create a image with a given configuration and start it directly with the virtualizer tools.

Commandline options

If you run anste with incorrect commandline arguments or passing -help option, you'll get this brief help: 

Usage:
    anste [options] [file ...] [var1=value1 var2=value2 ...]

     Options:
       -help              show this help
       -create            create a base image
       -remove            delete a base image
       -deploy            deploy a scenario
       -shutdown          finish and delete all the images of a scenario
       -test              test a suite
       -path <dir>        where to find the data  
       -out  <dir>        report output directory
       -video             save video recording of selenium tests
       -format <fmt>      select the output format for the reports
       -wait              wait before shutting down after testing complete
       -wait-fail         wait after a test fails
       -step              run tests step by step, waiting after each one
       -break <t1,t2,...> set breakpoints after the given test names
       -quiet             do not show output of commands
       -reuse             reuse the created image if exists (with -create)
                          reuse a deployed scenario (with -test)

Some options are also present in the configuration file explained in the next section, in the case of not overriding a configuration option with its equivalent commandline option, the config or the default one will prevail.

Now we're going to see some examples of use of the anste command for different purposes.

Create the debian-base.xml and ebox-base.xml base images: 

anste -c debian-base.xml ebox-base.xml

Remove one of the recently created image bases: 

anste -r debian-base.xml

Deploy the ebox/proxy.xml scenario hiding the output of external commands: 

anste -q -d ebox/proxy.xml

Undeploy the scenario deployed above, destroying the machines and removing the copied images: 

anste -s ebox/proxy.xml

Run the my-tests test suite located on /home/foo/anste/tests saving the results to /tmp/results: 

anste -t my-tests --path /home/foo/anste --out /tmp/results

The results will be generated in HTML by default, to view then you have to browse the /tmp/my-results/index.html file.

Run the ebox/samba test suite writing the results in XML format and stopping the process until user input if a test fails: 

anste -t ebox/samba -wait-fail -o /tmp/xml-results --format XML

Configuration file

The configuration file for the anste program is located on /etc/anste/anste.conf and contains different options grouped by category that are explained one by one in this section.

First, we're going to see an example configuration file that contains all possible options. Furthermore, in this example the options have assigned its default values. In the case of a option doesn't appear in the file, its default value that is shown here will be taken. If the option doesn't have a default value it will be set as --MANDATORY-- in this example.

In some cases, the mandatory options have an equivalent commandline option, so if this option is passed there is no need to set it in the configuration file.

Some other options aren't mandatory and don't have a default value, so in this example they'll have --NONE-- as value. 

[global]
system = Debian
virtualizer = Xen
verbose = 1

[paths]
images = /home/xen
logs = --MANDATORY--

[ansted]
port = 8000

[master]
port = 8001
[comm]
first-address = 10.6.7.10
gateway = 10.6.7.1
nat-iface = eth1

[deploy]
auto-create-images = 1

[selenium]
rc-jar = --MANDATORY--
browser = *firefox
video = 0
record-all = 0

[xen-options]
modules = --NONE--
size = 2Gb
memory = 512Mb
fs = ext3
image = full
kernel = /boot/vmlinuz-‘uname -r‘
initrd = /boot/initrd.img-‘uname -r‘
mirror = http://ftp.debian.org/debian

The following shows a detailed description of all above options, grouped by category:

[global]

This section contains general options.

system
Contains the name of the system backend to use. At the moment Debian is the unique option supported.
virtualizer
Contains the name of the virtualizer backend to use. At the moment Xen is the unique option supported.
verbose
Contains a boolean value (0 or 1) if we want the output of the external programs ran by ANSTE to be shown. This option can be set to 0 with the -quiet commandline option.

[paths]

This section contains options about the paths that ANSTE uses to locate or write files.

images
Absolute path where the disk images will be stored and searched.
logs
Path where the test reports generated by ANSTE will be stored. This option can be overriden with the -out commandline option of the anste program.

[ansted]

In this section appear the options related with the daemon that runs on the virtual machines and listen the requests sent by the ANSTE master program.

port
Contains the TCP port on which the 'ansted' daemon will listen.

[master]

This section contains the options related with the way as ANSTE acts as a master that sends commands to the virtual machines and receives its responses.

port
Contains the TCP port on which ANSTE will listen the messages coming from the virtual machines. Change it to another value if the default port is already used by another service.

[comm]

This section contains the options relative to the communication of ANSTE with the virtual machines and the one between the virtual machines and external hosts.

first-address
Contains the IP address to assign to the first virtual machine created. The following will have the address that results from increasing by one the last octet of this value.
gateway
Contains the IP address of the real machine that will be used as gateway for the virtual machines that want external access. This value must be an address in the same network as first-address.
nat-iface
Contains the name of the interface on the real machine that gives Internet access. Needed for masquerading to allow external access to the virtual machines.

[deploy]

This section contains the options used in the process of deploy an scenario.

auto-create-images
This boolean option must be set to 1 if you want the inexistent base image to be automatically created before deploying a scenario that requires them. If the option is set to 0, ANSTE will stop the deployment warning the user that are images missing.

[selenium]

This section contains the option required for the correct execution and report of the web-based tests. Some of this options correspond to options of the Selenium Remote Control program.

selenium-rc
Contains the full path of the Selenium Remote Control .jar file.
browser
Contains the value for the browser option of Selenium Remote Control used for the web-based tests.
video
Setting this value to 1 causes that web-based tests will be recorded in video for later viewing. Setting it to 0 will disable video recording. The -video commandline option sets this value to 1. Note that you have to install the RecordMyDesktop application in order to get this feature working.
record-all
Setting this value to 1 causes that all the recorded videos will be stored. Setting it to 0 will cause that videos of successful tests will be removed, storing only the failed ones. This option is recommended when you want to save disk space and are only interested on how a web-based test has failed.

[xen-options]

This section contains specific options for the Xen virtualizer, all of them are equivalent to commandline parameters of the xen-create-image tool.

modules
Contains the Linux kernel modules path that will be used for the virtual machines.
size
Size of the disk image. The format is a string with the ammount followed by the unit, being the valid units "Mb" for MegaBytes and "Gb" for GigaBytes.
memory
Ammount of RAM memory to be used by default for the virtual machines if they don't redefine it in its description. The format is the same of the size option.
fs
Name of the filesystem for the partition of the image (e.g., ext3, reiserfs).
image
Type of disk image to create. The values are full for maximum space occupation or sparse for only used-space occupation.
kernel
Path of the Linux kernel used to boot the virtual machines.
initrd
Path of the initial RAM disk file used to boot the virtual machines.
mirror
Package repository that debootstrap will use to install the distribution set in the dist option of the base images.