You are visitor number
to one of my pages
WHAT IS THIS?
This program is designed to test system responsiveness by running kernel
compilation under a number of different load conditions. It is designed to
compare different kernels, not different machines. It uses real workloads
you'd expect to find for short periods in every day machines but sustains
them for the duration of a kernel compile to increase the signal to noise
ratio.
Some of the load conditions are applications originally sourced by
B.Matthews' irman tool also licensed under GPL and modified to suit
Null load - No load
Cache Run - A no load run directly after a previous run without memory
flushing
Process load - Fork and exec N (N is 4*num_cpus) processes, connected in a
unidirectional ring by pipes. Insert M << N chunks of data into
the ring and pass them around (nice IPC/switch test)
Memory load - Repeatedly reference 110% of RAM in a pattern
designed to cause cache misses
IO Load - copies /dev/zero continually to a file the size of
the physical memory.
IO Other - same as IO load at a different location
Read Load - Reads a file the size of the physical memory
List Load - Lists the entire file system (ls -lRa /)
CTar Load - Repeatedly creates a tar of the kernel tree
XTar Load - Repeatedly extracts a tar of the kernel tree
Dbench Load - Runs dbench N (where N is 16*num_cpus) repeatedly
HOW DO I INSTALL IT?
Extract the tarball.
cd to the contest directory and type
make
make install
To override install dir, simply pass the INSTPATH variable
make INSTPATH=/path/to/install/dir install
(installs in /usr/bin by default)
HOW DO I RUN IT?
Download a linux 2.4.19 kernel source tree, and extract it where you
wish to be running your benchmarks. Copy the contest.config file in
this tarball to that directory as .config. Download dbench and compile
it (optional). Copy dbench somewhere into your PATH and copy the .txt
files from the dbench source into your kernel test tree. Then:
make oldconfig
make dep
Reboot into the kernel you wish to test, disabling any apm in the bios.
At lilo preferably boot into single user mode:
linux single vga=normal apm=off
Then cd to the kernel source directory and type (from the man page):
contest [-cdr] [-k name] [-t file] -o file [-n nrruns] [load...]
OPTIONS
These are the commonly used options:
-b Print a progress bar if information from previous runs is avail-
able. Do not use this for remote logins / testing as it will
taint the results.
-k NAME
Use the kernel name NAME in the report generated by contest
(default is the internal kernel name from uname.)
load
The name of the load you wish to run - can specify multiple loads
Available loads are: no_load, cacherun, pro-cess_load, ctar_load,
xtar_load, io_load, read_load, list_load, mem_load and
dbench_load (default all loads.)
-n N Number of times to run each load in contest to generate useful
averages (default 3.)
-r Generate report on all the cumulative log files in the current
directory.
-t FILE
Use the file FILE as the temporary file to use for io loads - can
specify a full path such as /tmp/tmpfile (default ./dump.)
-o FILE
target file for io_other (required unless io_other is not specified).
OTHER OPTIONS
You shouldn't need to touch these.
-c Assume a cold cache. This bypasses the memory flushing routines
that occur between different loads in contest that are normally
used to minimise the effects of caching of data from previous
compiles. It is of use only to test contest functionality and
will invalidate any testing you do.
-d Print debugging information if contest was compiled with debug-
ging enabled (will flood the console).
-p Dont cleanup after each load is run - speeds up the running of
contest but can use up massive amounts of disk space.
NOTES
* contest must be run in the top directory of a 2.4.19 linux kernel tree,
and the contest.config file from the contest source should be copied to
that directory as .config.
* contest assumes the existence of the common utilities, cc and dbench in
your PATH
* The version of cc used must not vary between benchmarks.
* dbench_load requires both the existence of dbench in your PATH and the
presence of the .txt files from the dbench source in your testbed tree.
* contest should be run by itself in single user mode or as the sole init
process to exclude the effects of any other loads on the system.
* contest requires large amounts of spare disk space to perform
dbench_load, the tar loads (ctar_load and xtar_load), read_load,
io_load and io_other.
* io_load requires as much spare disk space as the physical RAM the test
machine has.
* io_other needs the -o parameter to tell it where to do the other
io load. Running the io load on a separate hard disk tests different
aspects of the kernel.
* cacherun is only meaningful if run in combination with no_load.
* must be run with sufficient permissions to execute swapon/swapoff.
* The contest name is a play on words due to the Author's name being Con.
* Never do make dep again after running contest as it will change the
results of subsequent runs.
*Changing the vga from framebuffer mode to non-fb and vice versa changes
results.
There are four numbers generated for each load; the time taken to compile
the kernel, the average cpu percentage to do the compile, the amount of
times the load performed its task and the cpu% the load used while running.
The kernelname.log file contains more detailed information.
A report generated with -r gives the most useful information. It sorts the
kernels, lays out the information and creates a new result called
ratio which is simply the ratio of the time result compared to the noload
result for the kernel.
FAQ
WHAT DO THE NUMBERS MEAN
The lower the time (and ratio) the better, and the higher the cpu percentage the
better. The absolute number is not important, but the differences are.
When the difference between noload and the loads is small it shows that
on that kernel, the system is able to respond to requests for normal
tasks (ie responsiveness). When the cpu percentage is high it shows that
although the other loads are high, a cpu intensive application (kernel
compile) can still use as much as it needs. Ideally the load should be able
to perform some reasonable amount of work and the kernel compile take slightly
longer. If the balance shifts substantially in one direction or the other it
is indicating a large change in the fairness of scheduling (which is really
what contest is testing.)
From version 0.50 information about how much work the load itself has done
is also included. An internal number is generated by each work load to show
how much work is done called Loads in the results. Also included is the
average cpu% the load used while running (LCPU%). The best performance by
a kernel will be some compromise in minimising the time the kernel takes to
compile while not greatly decreasing the loads' work done - the kernel hackers
are the best people to interpret this information.
Do NOT change the hardware you test it on, the kernel tree you use for the
testing, gcc version or the hardware settings (eg with hdparm) as this will render
the results useless. Testing it on different hardware will be interesting
only to compare how the kernels compare on that piece of hardware.
Comparing results from different hardware is meaningless.
WHY DO MY NUMBERS LOOK WORSE NOW THAT MY HARDWARE IS BETTER?
This is of concern to the kernel design! If faster hard disks mean they
are so busy writing that other tasks wont get adequate attention then this
needs to be addressed.
CAN I TEST THE RESPONSIVENESS WHILE WRITING TO ANOTHER HARD DISK?
You can specify where the file written by io_loads is located by specifying the
io_other tempfile as a file on a different hard disk.
eg: if you have two hard disks mounted at / and /mnt/disk
and have your test kernel tree in /usr/src/linux you can specify
contest -o /mnt/disk/dump
SMP?
The cpu% may read >100%; it should be divided by the number of cpus.
HOW MANY TIMES SHOULD I RUN CONTEST?
The more the merrier to get the best results. However, good results are
obtained with the default 3 runs.
TODO
Lots I haven't thought of yet.
HOW DO I CONTACT YOU
Feel free to contact me with questions, suggestions, comments or patches
email me at: contest at kolivas dot org
Changelog:
v0.60 to 0.61
Bugfixes. Simpler command line options. Implementation of io_other.
Improved resolution of results. Logs are incompatible again with
previous versions. Dbench corrected back to 16*num_cpus. New URL &
contact details.
v0.51 to 0.60
Massive change with a complete rewrite of all elements in c by Aggelos
Economopoulos (the unofficial maintainer of the actual program now.)
Implementation of the internally used cacherun and dbench_loads.
Progress bar, cold cache, debug and no cleanup options added. Change
default to 3 runs of contest and all loads. Rewrite of process_load
by Rene Herman. First version of manual page.
v0.50 to 0.51
Added list_load, read_load, xtar_load, ctar_load. Further cleanups and compiler
issues.
v0.42 to 0.50
Included work done and cpu% by loads. io_load converted from using head
to using dd (to work with high memory machines and to present greater
io_load). Lots of internal changes to support the changes. Results while
similar to 0.4x have different output so are not compatible with previous
versions.
v0.41 to 0.42
Added averaging of results during parsing. Added precision option to ratio
(to allow integer only bash to do all sorts of stuff). Older gcc fix (A. Morton)
v0.40 to 0.41
Major restructure of priming for each test gives much greater resolution
in results. Removed priming compile and streamlined startup for faster runtimes.
v0.37 to 0.40
Modified process_load to be more of a process_load than a cpu load.
Changes results dramatically hence major version upgrade (R. Herman).
Better SMP support
v0.36 to 0.37
Dropped IO_halfmem (no useful info) and changed name of IO_fullmem to
IO_load. Performed sync,swapoff,swapon b/w tests. Added ratio app
to create ratios for results. Intrinsic results parser added. Fixed PATH in
contest. Added number of runs option. (lots)
v0.35 to 0.36
Added command line arguments. Choose tests to perform at runtime.
Optional install path (R. Maureira)
v0.34 to 0.35
Moved everything to /usr/bin. Extra kernel compile performed
prior to each benchmark to ablate effects of prior benchmark. Load started sooner.
mem_load modified to accept argument -p (percentage load) for future version.
v0.33 to 0.34
Better messages, more information, ability to specify io load file, removed
chance of loads overlapping. Removed accidentally commented out lines.
v0.32 to 0.33
Improved messages, names corrected, noload script removed
v0.30 to 0.32
Signal trapping, kernel name specifying and code cleanup (lots of people)
v0.22 to 0.30
Code cleanup (R.v. Riel) (now -Wall compat.)
v0.21 to 0.22
mem_load corrected for low mem systems by (R.v. Riel)
v0.20 to 0.21
Minor change to meminfo for high mem systems (R. Hron)
Change time to $time
v0.12 to 0.20
Changed IO load (idea by A. Morton)
v0.11 to 0.12
Modified mem_load to work with 2.5.x kernels (R.v. Riel)
v0.1 to 0.11
Internal nonsense
THANKS
Thanks Aggelos Economopoulos, Rik van Riel, Andrew Morton, Randy Hron,
Rene Herman, Cliff White, Robinson Maureira, Paolo Ciarrocchi and more!
Written by Con Kolivas (official maintainer).
Last update: Tue Feb 18 22:30:23 2003
You are visitor number
to one of my pages