Preventing the Unauthorised Binary

Brett Lymn
Copyright 2000-2002. All rights reserved.
January 19, 2000

Updated June 18, 2002 by Jay Fink


This paper will demonstrate a method of ensuring the binaries running on a Unix system have not been subverted or modified and that unauthorised binaries can be prevented from executing even if the root account is being used. This method also allows sophisticated shell interpreters to be safely installed by preventing the interpreter being run directly by exec. This paper is a description of work in progress and some features described may change.


At the time of writing, the current trend of system attack is gain control of many victim machines and use the resources of these victims to disrupt the services on a third party system. Sophisticated distributed Denial of Service (DoS) tools such as Stacheldraht[1], Trinoo[2], Tribal Flood Network[3][4] and others[5] that are being deployed by attackers to perform DoS attacks against various targets.

Part of the reason as to why these tools are widely deployed is the poor set up and maintenance of machines connected to the Internet. Basic precautions for securing machines has not been done and auditing of the machines is not performed. Part of the reason, also, is that a lot of these machines are UNIX systems and once the attacker has gained root privileges they have total control of the victim system. This means that they can install and run any binaries they choose or modify current system binaries to perform extra functions - such modified binaries are called Trojan Horses and have been commonly used to log user passwords to a file or provide a root shell on demand.

Poor system setup is a matter of educating administrators about the correct manner in which to secure a system. It seems amazing that this still needs to happen given the sheer volume of information available both commercially and freely but evidence shows that the education process needs to continue.

A method of preventing unauthorised binaries running and detecting Trojan horses will allow the conscientious administrators to more easily audit their systems and have confidence in the binaries they run. There are already methods of doing this but they do have some difficulties and this paper attempts to show a new approach to the problem.

Prior Art

Probably the most well known tool for auditing a running system is a tool called tripwire[6]. This tool is quite old now but it is still useful in auditing a system. Tripwire validates the signatures of files in it's control file against the files on the running system and flags any files that do not match. Multiple fingerprinting methods can be employed depending on the administrators tastes as well as checking the file modification times. Due to the nature of tripwire there are aspects of it that are difficult to secure properly so that the results can be trusted. This difficulty may be part of the reason why tripwire is not more widely used. Some of the problems with tripwire are:

  1. Tripwire runs as a user level process and as such there is nowhere that it can securely cache data between runs. It also means that an attacker may be able to subvert the running of tripwire to either prevent it running or run it with a subverted configuration.
  2. It is difficult to assure that the tripwire binary being run is the true, correct version and that the binary has not been manipulated by an attacker.
  3. Tripwire can be computationally expensive to run. The degree of expense depends on the number of differing fingerprinting methods configured and the fingerprint algorithms used. Normally, there is a trade off between the number of times a day tripwire is run and the processor load. A machine that continually runs tripwire may be too slow to perform real work. This trade off means that there is a window of opportunity for an attacker to break into a machine and use that machine for whatever she wants before tripwire runs again and, possibly, flags a problem.
  4. Depending on the configuration, reports from tripwire can be very verbose about changes to the system it is run on. Normally, parts of the machine are deliberately not scanned or only file status is checked. These blind spots may be used by an attacker to store their toolkit on unnoticed.

Some of these points can be addressed by very careful configuration and strict adherence to procedures for maintaining the tripwire configurations and databases.

Other Methodologies

In the release of 4.4BSD there is a kernel securelevel variable that affects the behaviour of the kernel when it is set to various values. In some 4.4BSD derived systems there is also the concept of file flags that extend the normal file permission semantics to include the ability to make a file immutable (unable to be changed), append only, opaque, archived or nodump. At securelevel 0 normally file permissions apply to device files and the immutable and append only flags can be turned off. At securelevel 1 immutable and append only flags cannot be changed, /dev/kmem, /dev/mem and raw devices of mounted file systems are made read only. Securelevel 2 is the same as securelevel 1 but raw devices are always read only.

With these features it is possible, in combination with judicious use of the read-only and no-exec mount options to build a system that has all the binaries made immutable and mounted on read only partitions. The log files and other temporary files on partitions that are mounted no-exec. A machine such as this is not impossible to build but requires some painstaking setup. One problem with this setup is that not all software respects the clear separation of writable files from executable binaries. Also some programs, for example, the ISC dhcp client runs programmatically written shell scripts which, in itself, is not a bad thing but it does mean that there needs to be some writable place that also allows execution if you want dhcp to work. This forces a breach in the system that can be used by an attacker. DHCP is not the only case of this, some programs such as syslog expect to be able to rewrite /dev to place a listening socket there. Syslog is can be told to put this socket elsewhere but some other programs may not be as cooperative.

To cater for the exceptions there is the temptation to leave the partitions writable and just make the binaries immutable. This can lead to a false sense of security because unless the locking process is done extremely carefully the attacker will be able to move aside the immutable binary and place their own Trojan horse in it's place. Unless a tool like tripwire is used in this situation there is no assurance that the binaries have not been manipulated due to exploitation of a set up error. There is also no method of preventing the execution of binaries loaded onto the system.

A Different Way

After some discussions on the Bugtraq[7] mailing list an analysis of the NetBSD[8] kernel sources was performed to see if the checking of executables could be made more stringent by performing a cryptographically strong fingerprinting of the executable. It was found that, ultimately, all exec calls go through a single execve call and that in this call there was a function called check_exec that it is used to validate the file to ensure that it can be executed. The check_exec function can be extended to include more checks and these checks will apply to any exec made on the system.

The NetBSD check_exec routine was modified to perform a md5 fingerprint of the executable that was going to be exec'ed. The md5 fingerprint was chosen because the code for evaluating the fingerprint was already in the kernel and it was simply a matter of modifying check_exec to make the appropriate calls. A pseudo-device driver was added that allowed a user level program to load a list of fingerprints into the kernel memory space, the driver resolved the filename string passed into a device node and i-node and stored the information in a simple linked list in the kernel memory space. Loading of fingerprints into kernel space is denied when securelevel is above 0.

With the modifications, when an exec was performed the check_exec routine evaluated the md5 fingerprint of the given file and then searched the stored list of fingerprints. If the device node and i-node of the executable was found in the stored list then the evaluated fingerprint was compared against the stored one. If the fingerprints match then check_exec returns success. If the file cannot be found on the stored list or the fingerprint does not match then the behaviour of check_exec depends on the securelevel. A new securelevel was introduced, securelevel 3. At this securelevel the check_exec call will return a permission denied error for any file that does not pass the md5 fingerprint check. For securelevels below 3 an error was logged to the console for any file that fails the md5 check but check_exec returned success. This allows the system to startup without a fingerprint list loaded, this needed to be done otherwise there would be no way of booting the system. Once the fingerprints were loaded the securelevel could be raised to prevent an attacker installing more signatures.

Performing the fingerprint calculation every time had a severe impact on the performance of the machine. A full make of a NetBSD kernel took 1.7 times longer with the fingerprint enabled than it did without. This was a noticeable slow down. There appeared to be three things that could be done to address this slow down:

  1. Do nothing. Hardly technically satisfying but it could be argued that the extra security was worth the slow down. Besides which the intention was that fingerprinting would only be done on firewall type machines where most processes are long running.
  2. Buy a faster machine. Making the problem go away by throwing money at it is valid but not usually popular.
  3. Implement some caching scheme so the fingerprints need not be evaluated every time. Because the fingerprinting code is operating inside the kernel it is safe to assume the cached data cannot be modified.

Option 3 was chosen and research started on how best to cache the data. The fingerprint cache needs to be fast to look up, keep the associated path of the binary, have a high hit rate and be flushed when the associated file is modified or deleted. As it turned out there was already a cache that did all of these things in the NetBSD kernel. It was found that the Directory Name Lookup Cache (DNLC) already caches a lot of useful information, including the v-node, about a file and that the cache was already doing all the functions required for the caching of the fingerprint data.

The v-node kernel structure was modified to include a single byte fingerprint status, because the NetBSD kernel attempts to keep v-node data as long as possible in case the v-node is used again this operates as an effective cache for the fingerprint status of the file. A single byte is used to keep the fingerprint status rather than keeping the entire fingerprint, this saves memory and means that a single byte only needs to be checked to verify the fingerprint status of a file. The fingerprint is validated against the loaded list value just after it is calculated and the status byte set to indicate the results of the comparison between the calculated and stored values. This fingerprint status is only invalidated when the v-node is cleaned up for use by another file. So, as long as there are no activities, such as a find on a filesystem, that cause a run of the v-node cache the v-node fingerprint status will be available for later invocations of the same executable. This results in a large reduction in the amount of time spent evaluating the fingerprints on executables by the kernel. Measurements of a make of the NetBSD kernel show that with caching the fingerprints the time to run the make took about 5% longer with the signed executables than it did without them. From this it can be seen that caching the fingerprint results makes a large positive impact on the performance of the system.

Handling Shell Interpreters

The way that a shell script exec is handled by the NetBSD kernel is different to a normal executable. When exec is called on a shell script, check_exec is called to verify the file is a candidate for execution. If check_exec returns success then the appropriate exec handler for the binary type is selected and executed. In the case of a script this exec handler is exec_script. The exec_script handler parses the header of the script and extracts the script interpreter and then calls check_exec to verify the interpreter is valid for execution. This two step process presents an interesting opportunity to treat the shell interpreter differently to a normal exec.

The NetBSD kernel check_exec routine was modified to pass an extra parameter to check_exec. This parameter was a flag that indicated whether or not check_exec was being called from execve itself or from an exec handler. The only exec handler that used check_exec was exec_script, this was modified to include the new parameter. The signed exec pseudo-device structure was modified to include an extra flag and the signature loader changed to support the flag. This flag is set to indicate when only indirect calls of the binary are allowed, that is, the binary is only allowed to be a script interpreter, any direct execution request of the binary will be denied.

It is important to note that both the shell script and it's interpreter are subjected to validation of their fingerprints in check_exec, regardless of the status of the indirect execution flag. This provides assurance that the script nor the interpreter have been tampered with. The indirect execution flag give the opportunity to install a powerful interpretive language, for example Perl, that can only be used to run verified scripts removing the risk that the languages capabilities will be used by an attacker.

An interesting application would be to make the binary /bin/sh a candidate for indirect execution. Careful checking would be needed to ensure no executables attempted to exec /bin/sh directly but if this was done then attempts to break into the machine by manipulating a buffer overflow to execute /bin/sh would be thwarted. This is not a complete solution as the attacker could attempt to exec another shell but this can be addressed which serves to "raise the bar" on the system security by preventing many scripted attacks from working and it does make it difficult to tell if a remote exploit failed to work because the buffer overflow failed or the shell exec was denied.

Problems with signed execution

The signed executable exec check does not protect against an attacker executing code on the processor stack via a buffer overflow or some sort of similar method. This type of exploit can be made more difficult by making the exec of the common shells indirect which means all scripts will still function but an attempt to exec a shell would fail. Login shells would need to be placed in an obscured location to allow logins to the system. This approach is a security by obscurity method and should not be trusted as a security measure in itself. If an attacker can find out where the executable shells are located then the buffer overflow can be modified to use the new location. So, this method is only secure as the information about the locations of the executable shells.

The signed exec fingerprints are, currently, stored in a file and loaded during boot in a rc script. The startup file, the fingerprints and the signed exec fingerprint loader all need to be protected as they now form the crux of the security of the system. These can be protected by making the files immutable or similar. Tripwire can also be used as a defense against manipulations of the system, since the tripwire binary can now be verified as the correct one this can be used to scan for tampering on the system. Note that now that the binaries are automatically checked by the kernel the work that tripwire has to do is much reduced giving the possibility that tripwire could be run more frequently than would otherwise be practical.

Possible Applications

Signed executables can be used in any application that has a defined set of executables that need to be run and these executables do not change often. Some possible scenarios are firewalls, routers and secure workstations where only approved binaries are to be run. The signed executable exec is not suited to being run in an environment where users need to execute code that is constantly changing, a software development environment would be such a case.

The signed executable fingerprint file can also be used as a method of securely distributing the operating system to end users. Once the user has installed the operating system they could request an encrypted copy of the fingerprint file. This fingerprint file could then be decrypted using a public key decryption and the resulting file used to protect the new system. This provides a secure method of ensuring the original distribution was not tampered with.


The signed executable exec kernel modification provides a visible method of verifying the executable that is being run is the correct one and has not been tampered with. This provides a level of trust in the executables that is difficult to attain by other means. Due to the operation at the kernel level important speed gains can be made by using the kernel memory protection mechanisms to protect cached data from tampering. This kernel modification also gives a fine level of control over what can be executed on the system, even by root, a capability that was not available before. Also, the addition of the indirect execution check means that powerful interpreters can safely be installed in security critical situations with the knowledge that the interpreter cannot be used to run unsigned scripts. This kernel modification does not provide a total security solution but should be treated as another tool in the chest that can be used to increase the security of an operational system.

Work Done Since the First Draft

Some work has been done since the first draft of this document:

Further Work

Work on the signed executable exec kernel modification is still in progress to address some issues. Some things that can be looked at are:

[1] Dittrich, David. The "Stacheldraht" distributed denial of service
    attack tool
[2] Dittrich, David. The DoS Project's "trinoo" distributed denial of
    service attack tool
[3] Dittrich, David. The "Tribe Flood Network" distributed denial of
    service attack tool
[4] Barlow, Jason and Woody Thrower. TFN2K - An Analysis
[5] Links to more information on DDoS tools can be found at
[6] Sources for Tripwire can be found at
[7] For and introduction to the Bugtraq mailing list, see
[8] The NetBSD home page is at