Smurfweb 99 is the name of the testing suite developed as a free replacement to the Specweb99 testing suite. The name Smurfweb itself has a history that is probably only funny to it's creators, therefore there's no need to explain the name further.
The smurfweb testing suite is a set of tests run off of the JMeter testing application. JMeter was designed to load test behaviour and performance of applications. The smurfweb testing suite is a set of rules and tests run by JMeter to give a similar testing environment to that of the Specweb99 test suite.
TestsSmurfweb tests the maximum number of connections against a host running a defined set of tests against that host. Using access patterns gained from the Specweb testing system Smurfweb was designed to test against those same patterns through two types of testing.
Dynamic TestingThere is dynamic testing aspect were the following breakdown of http request are run against the host machine. CGI gets do a small amount of processing to work the CPU on the system, while the other GETs are purely download and upload testing system I/O.
There is also a standard workload test where generated files of certain distributions below are accessed. This distribution follows the Specweb99 distribution pattern and is explained in their FAQ
Each system configuration was tested with the smurfweb test suite, Linux on a single processor machine, Linux on a SMP machine and Xen. Our test files can be found in test_suites.tar.bz2. This file contains a test_suites directory with a results subdirectory. There are a number of directories inside this results directory, each directory name specifies what type of test was run. In the linux-1 directory 1 client was set to run tests against a single cpu Linux box. The linux-2 directory was 2 clients running against a single cpu target box running Linux, and so on as the numbers increase.
Running The TestsThe jmx files in each sub-directory can be loaded into the JMeter test program. To rerun the tests, you will need to change some information like the IP address of target machines, port numbers and location of the scripts to execute. You can change these using the Jmeter GUI or by editing the JMX file (JMX files are actually XML files). You can also change some other aspects of the test such as the number of tests, where the results are stored or even the workload mix. JMeter provides excellent tutorials on how to get your system up and running, once you have JMeter setup properly you should load the proper jmx files and you will be ready to run your tests.
When the tests are run, Jmeter has nice facilities to display the results. It also allows you to save the results to jlt files. However, be ware - not all the information displayed by Jmeter is saved to the JLT files. Reproducing all the information displayed by Jmeter after the test is run requires some additional processing. We did not find an option in Jmeter to reload JLT and redisplay all the information produced after the original test run.
Getting ResultsWith each test run, the results need to then be analyzed in a spreadsheet. JMeter allows you to dump your results into either and XML file type or a CSV type. We found it easiest to work with the CSV type as it imports easily into most spreadsheet applications.
For a quick analysis of our run we created a perl script to analyze and give a breakdown of the test run immediately. The analysis script will also remove any data that is useless to the aggregate set, like error codes and timeouts.
If your data is dropped into XML form, we have another simple python script which can convert the XML files into CSV form for transfering into a spreadsheet or being run against our perl script.