WHAT IS THIS? This program is designed to test system responsiveness by running kernel compilation under a number of different load conditions. It is designed to compare different kernels, not different machines. It uses real workloads you'd expect to find for short periods in every day machines but sustains them for the duration of a kernel compile to increase the signal to noise ratio. Some of the load conditions are applications originally sourced by B.Matthews' irman tool also licensed under GPL and modified to suit Null load - No load Cache Run - A no load run directly after a previous run without memory flushing Process load - Fork and exec N (N is 4*num_cpus) processes, connected in a unidirectional ring by pipes. Insert M << N chunks of data into the ring and pass them around (nice IPC/switch test) Memory load - Repeatedly reference 110% of RAM in a pattern designed to cause cache misses IO Load - copies /dev/zero continually to a file the size of the physical memory. IO Other - same as IO load at a different location Read Load - Reads a file the size of the physical memory List Load - Lists the entire file system (ls -lRa /) CTar Load - Repeatedly creates a tar of the kernel tree XTar Load - Repeatedly extracts a tar of the kernel tree Dbench Load - Runs dbench N (where N is 16*num_cpus) repeatedly HOW DO I INSTALL IT? Extract the tarball. cd to the contest directory and type make make install To override install dir, simply pass the INSTPATH variable make INSTPATH=/path/to/install/dir install (installs in /usr/bin by default) HOW DO I RUN IT? Download a linux 2.4.19 kernel source tree, and extract it where you wish to be running your benchmarks. Copy the contest.config file in this tarball to that directory as .config. Download dbench and compile it (optional). Copy dbench somewhere into your PATH and copy the .txt files from the dbench source into your kernel test tree. Then: make oldconfig make dep Reboot into the kernel you wish to test, disabling any apm in the bios. At lilo preferably boot into single user mode: linux single vga=normal apm=off Then cd to the kernel source directory and type (from the man page): contest [-cdr] [-k name] [-t file] -o file [-n nrruns] [load...] OPTIONS These are the commonly used options: -b Print a progress bar if information from previous runs is avail- able. Do not use this for remote logins / testing as it will taint the results. -k NAME Use the kernel name NAME in the report generated by contest (default is the internal kernel name from uname.) load The name of the load you wish to run - can specify multiple loads Available loads are: no_load, cacherun, pro-cess_load, ctar_load, xtar_load, io_load, read_load, list_load, mem_load and dbench_load (default all loads.) -n N Number of times to run each load in contest to generate useful averages (default 3.) -r Generate report on all the cumulative log files in the current directory. -t FILE Use the file FILE as the temporary file to use for io loads - can specify a full path such as /tmp/tmpfile (default ./dump.) -o FILE target file for io_other (required unless io_other is not specified). OTHER OPTIONS You shouldn't need to touch these. -c Assume a cold cache. This bypasses the memory flushing routines that occur between different loads in contest that are normally used to minimise the effects of caching of data from previous compiles. It is of use only to test contest functionality and will invalidate any testing you do. -d Print debugging information if contest was compiled with debug- ging enabled (will flood the console). -p Dont cleanup after each load is run - speeds up the running of contest but can use up massive amounts of disk space. NOTES * contest must be run in the top directory of a 2.4.19 linux kernel tree, and the contest.config file from the contest source should be copied to that directory as .config. * contest assumes the existence of the common utilities, cc and dbench in your PATH * The version of cc used must not vary between benchmarks. * dbench_load requires both the existence of dbench in your PATH and the presence of the .txt files from the dbench source in your testbed tree. * contest should be run by itself in single user mode or as the sole init process to exclude the effects of any other loads on the system. * contest requires large amounts of spare disk space to perform dbench_load, the tar loads (ctar_load and xtar_load), read_load, io_load and io_other. * io_load requires as much spare disk space as the physical RAM the test machine has. * io_other needs the -o parameter to tell it where to do the other io load. Running the io load on a separate hard disk tests different aspects of the kernel. * cacherun is only meaningful if run in combination with no_load. * must be run with sufficient permissions to execute swapon/swapoff. * The contest name is a play on words due to the Author's name being Con. * Never do make dep again after running contest as it will change the results of subsequent runs. *Changing the vga from framebuffer mode to non-fb and vice versa changes results. There are four numbers generated for each load; the time taken to compile the kernel, the average cpu percentage to do the compile, the amount of times the load performed its task and the cpu% the load used while running. The kernelname.log file contains more detailed information. A report generated with -r gives the most useful information. It sorts the kernels, lays out the information and creates a new result called ratio which is simply the ratio of the time result compared to the noload result for the kernel. FAQ WHAT DO THE NUMBERS MEAN The lower the time (and ratio) the better, and the higher the cpu percentage the better. The absolute number is not important, but the differences are. When the difference between noload and the loads is small it shows that on that kernel, the system is able to respond to requests for normal tasks (ie responsiveness). When the cpu percentage is high it shows that although the other loads are high, a cpu intensive application (kernel compile) can still use as much as it needs. Ideally the load should be able to perform some reasonable amount of work and the kernel compile take slightly longer. If the balance shifts substantially in one direction or the other it is indicating a large change in the fairness of scheduling (which is really what contest is testing.) From version 0.50 information about how much work the load itself has done is also included. An internal number is generated by each work load to show how much work is done called Loads in the results. Also included is the average cpu% the load used while running (LCPU%). The best performance by a kernel will be some compromise in minimising the time the kernel takes to compile while not greatly decreasing the loads' work done - the kernel hackers are the best people to interpret this information. Do NOT change the hardware you test it on, the kernel tree you use for the testing, gcc version or the hardware settings (eg with hdparm) as this will render the results useless. Testing it on different hardware will be interesting only to compare how the kernels compare on that piece of hardware. Comparing results from different hardware is meaningless. WHY DO MY NUMBERS LOOK WORSE NOW THAT MY HARDWARE IS BETTER? This is of concern to the kernel design! If faster hard disks mean they are so busy writing that other tasks wont get adequate attention then this needs to be addressed. CAN I TEST THE RESPONSIVENESS WHILE WRITING TO ANOTHER HARD DISK? You can specify where the file written by io_loads is located by specifying the io_other tempfile as a file on a different hard disk. eg: if you have two hard disks mounted at / and /mnt/disk and have your test kernel tree in /usr/src/linux you can specify contest -o /mnt/disk/dump SMP? The cpu% may read >100%; it should be divided by the number of cpus. HOW MANY TIMES SHOULD I RUN CONTEST? The more the merrier to get the best results. However, good results are obtained with the default 3 runs. TODO Lots I haven't thought of yet. HOW DO I CONTACT YOU Feel free to contact me with questions, suggestions, comments or patches email me at: contest at kolivas dot org Changelog: v0.60 to 0.61 Bugfixes. Simpler command line options. Implementation of io_other. Improved resolution of results. Logs are incompatible again with previous versions. Dbench corrected back to 16*num_cpus. New URL & contact details. v0.51 to 0.60 Massive change with a complete rewrite of all elements in c by Aggelos Economopoulos (the unofficial maintainer of the actual program now.) Implementation of the internally used cacherun and dbench_loads. Progress bar, cold cache, debug and no cleanup options added. Change default to 3 runs of contest and all loads. Rewrite of process_load by Rene Herman. First version of manual page. v0.50 to 0.51 Added list_load, read_load, xtar_load, ctar_load. Further cleanups and compiler issues. v0.42 to 0.50 Included work done and cpu% by loads. io_load converted from using head to using dd (to work with high memory machines and to present greater io_load). Lots of internal changes to support the changes. Results while similar to 0.4x have different output so are not compatible with previous versions. v0.41 to 0.42 Added averaging of results during parsing. Added precision option to ratio (to allow integer only bash to do all sorts of stuff). Older gcc fix (A. Morton) v0.40 to 0.41 Major restructure of priming for each test gives much greater resolution in results. Removed priming compile and streamlined startup for faster runtimes. v0.37 to 0.40 Modified process_load to be more of a process_load than a cpu load. Changes results dramatically hence major version upgrade (R. Herman). Better SMP support v0.36 to 0.37 Dropped IO_halfmem (no useful info) and changed name of IO_fullmem to IO_load. Performed sync,swapoff,swapon b/w tests. Added ratio app to create ratios for results. Intrinsic results parser added. Fixed PATH in contest. Added number of runs option. (lots) v0.35 to 0.36 Added command line arguments. Choose tests to perform at runtime. Optional install path (R. Maureira) v0.34 to 0.35 Moved everything to /usr/bin. Extra kernel compile performed prior to each benchmark to ablate effects of prior benchmark. Load started sooner. mem_load modified to accept argument -p (percentage load) for future version. v0.33 to 0.34 Better messages, more information, ability to specify io load file, removed chance of loads overlapping. Removed accidentally commented out lines. v0.32 to 0.33 Improved messages, names corrected, noload script removed v0.30 to 0.32 Signal trapping, kernel name specifying and code cleanup (lots of people) v0.22 to 0.30 Code cleanup (R.v. Riel) (now -Wall compat.) v0.21 to 0.22 mem_load corrected for low mem systems by (R.v. Riel) v0.20 to 0.21 Minor change to meminfo for high mem systems (R. Hron) Change time to $time v0.12 to 0.20 Changed IO load (idea by A. Morton) v0.11 to 0.12 Modified mem_load to work with 2.5.x kernels (R.v. Riel) v0.1 to 0.11 Internal nonsense THANKS Thanks Aggelos Economopoulos, Rik van Riel, Andrew Morton, Randy Hron, Rene Herman, Cliff White, Robinson Maureira, Paolo Ciarrocchi and more! Written by Con Kolivas (official maintainer). Last update: Tue Feb 18 22:30:23 2003
You are visitor number
to one of my pages