Security is 99% about containment and 1% about bugs

This is a little article about the forgotten art of containment. In the past I have worked for some years as system administrator with some different ISP's, administrating and securing *NIX systems, routers and fire-walls.
I have left this field of work 3 years ago. Last year I visited a conference on computer security and hacking, and I was completely shocked by the apparent ignorance to the nature of the problem and the possible security measures relating to the current impact of DDOS networks and worms like Code-Red after and talking and listening to some administrators and security specialists.
All the people that I talked to about the subject, and most speakers that spoke about this and related subject appeared completely ignorant of the underlying structure of the problem.
For this reason I try to explain the true nature of a large portion of the apparently misunderstood Worm and DDOS agent problems in the idle hope that at least some people will read it and come to understand the true nature of the problems.

Its not about bugs, its about containment

The last few years it seems that the focus of the security experts has shifted toward bugs and bug-fixes. This is done to such an extend that the number one security issue has been pushed to the background. Sure, if a program has a bug this should be fixed, and be fixed asap, but this is only the final maintenance phase in security, without containment measures in the implementation phase your security will be hopelessly incomplete.
There is one thing we must face up to in current computer science, almost any program with a complexity exceeding 'hello world' is and will remain broken. Making non broken programs is hard, and the prevailing notion amongst programmers and their superiors is 'If the average user doesn't notice its broke, don't fix it', and to be realistic, who could blame them. Anyone who has ever done quality assurance on code knows that any block of code that was written in a day would take a few days to completely verify. There is absolutely no force on earth that could bring programmers and/or software companies to spend 300% to 500% more resources in order to roll out bug free code. Knowing this, and knowing all the software we are using is most likely infested with thousands of unknown bugs of what tens or hundreds could be exploitable, we should realize that just depending on frequently patched software is insufficient.
So what could we do that goes beyond this? We could and should use containment to limit the possibilities of the broken programs that we are using. So what do I mean by containment? The most important thing is that any network,system,service or program that appears to need protection must be seen as not only something to protect, but more importantly something to protect against. So next to saying 'I need to protect this host and the software and services run on it from the Internet' you might also need to say : This is absolutely nothing new, but apparently a large part of the security community seems to have forgotten about it, and only seem to focus on bugs and patches. In order of importance these are 5 issues that should be addressed with current network security:
  1. Network Containment of hosts
  2. Anti Spoofing (strictly speaking this is also a form of network containment)
  3. Host based containment of applications
  4. The realization that the number of bugs present has a more than linear relation to the amount of 'features'.
  5. Bugs in applications.

A server isn't a client

This seems to be the core of the problem that everyone seems to have forgotten. In order to do basic containment, a server should not be used as a client and vice-versa, it is understandable that a small amount of home users would want to run a server on their client system, but it is inexcusable if the same is done in any other context. (Recent development in the field of p2p are virtual nightmares from a containment point of view, barn, one of my personal development projects tried to focus on this particular problem and failed).
Based on this policy a server could be protected with 'inclusive' access filters by its own fire-walling software, and also by all its surrounding router systems. There might be some exceptions on this general rule, for example for name-server or proxy usage, but basically this rule should be applied.

Inclusive access filters

On both router systems, and the server systems themselves a strict implementation of 'inclusive' access filters can be used both for protection and for containment. Using such filters, a web-server would for example basically only need to be allowed the following traffic: If someone would inadvertently leave a broken imapd or rpc.statd installed on the system this would not matter as the filters would contain its bug to local exploit-ability. If the http daemon would be broken by for example the bug that is exploited by the Code-Red worm, the system would be infected by the worm, but the inclusive access filters would contain the worm in such a way that it would not be able to spread. If the worm was to try to spread itself, it would be blocked by the filter rules, and this could if combined with logging result in early detection of the worm.
Thus can be concluded that these simple filter rules would facilitate in:

Simple OS containment

Basically any operating system will grant certain permissions to any process running. Server software should in all cases be contained within a small portion of rights that will grant it as little 'additional' rights outside the space it needs to do its job as server. Unfortunately it appears that this issue has not been addressed much in the mainstream operating systems. Some mainstream operating systems and server software hardly have any containment for these 'services'. Other mainstream operating like the current Unix systems have a minimal form of containment and are true multi-user systems.
This means that these systems can run different servers as different users with different rights on the system. The unix confinement mechanism is based on user/group ownership and rights for resources. This simple form of containment should be considered the absolute minimum for server application containment and should always be used for all server software (no server software should run as superuser/root), it will not be a complete waterproof containment by itself, but when combined with chrooted environments or jails, it could raise the level of containment to an acceptable level.
It can further be useful to limit every distinct functionality that can be identified as a separate user with separate access rights.
Some non mainstream operating systems are completely based upon the mechanism of containment (actually they are based on the mechanism of 'capability' what is the boolean inverse of containment with maximum containment being the same as minimum capability), we can only hope these mechanisms will one day make it into the mainstream OSses.

Chrooted servers

A more advanced and also complex form of containment that is only available on *nix based systems is the use of so called 'chrooted environments' or 'jails' for server software. Using this technique on a server it is possible to contain the exploitation of bugs in server software (provided the server is not running as root) to this limited part of the system.
The implementation of this can be used for servers like bind that are highly complex servers that have a large history of exploitable bugs. For the use of for example a multi user installation of a web-server server like apache, the use of a chrooted environment is also possible, but is most likely to complex for most administrators to fit into their already complex multi-user infrastructure. The implementation of this containment technique relies on a deep understanding of the underlying operating system. It will however contain the attacker in this environment where the parts of the system that might have local exploitable bugs (sgid and suid stuff) are not accessible, and thus the attacker could not gain root access and nest himself invisible into the system.

Host based network containment

Next to running network applications in a jail as a separate user, the current generation of host based fire-walling software allows the use of 'application based', or better yet 'user based' fire-walling. An excellent example of this are the firewall functions in the iptables or ipfw fire-walling software, where it is possible to sharpen the inclusive access lists that are present on a network level by adding user based filtering. This can be specially useful for shielding off localhost services that could otherwise be accessible to a compromised network server.

Sharp objects

One further issue of containment is the availability of resources. If an attacker breaks through a layer of containment this attacker might be able to use any resource that happens to be laying around as a way to brake through the next layer of containment. For this there is one golden rule: This rule should be used for server and software configuration, but might also be useful for client systems, especially where network-enabled software is concerned.
If there is an exploitable bug in unused software, than this software should not reside on your system, and as we assumed that all software has load of unknown bugs its the smartest thing to get rid of everything unused. Just think of it as follows:
If you walk into your living room at night into a raving mad unarmed burglar, you will be glad you did not leave any sharp objects laying around in the room. Just think of any unused piece of software, especially those that use raised privileges or give access to forms of network or ipc as sharp objects.

The realization that the number of bugs present has a more than linear relation to the amount of 'features'

A perfect designed piece of software could have a linear relationship between the amount of 'features' and the amount of 'bugs', a badly designed piece of software could have a close to exponential relationship between the number of features and the amount of bugs. For this reason one should always try to minimize the amount of available features in order to get the minimum amount of bugs.
This rule should already apply in the system selection phase. A big and feature rich application will have much more bugs and holes than a application that is build with a minimal-config approach. If you make use of the feature-rich software but don't use the features, then the only feature that remains is the possible exploit-ability of bugs in the code of these features. For this reason you should always select the most minimal-config solution that contains all the required features and as little as possible non-used features.

(I am not getting into the dynamic-modular vs statical discussion here, a dynamic module interface can be a gaping hole, but so can loads of statical features, just try to think of a module interface as a 'rather big' feature, and try to determine if this interface would be a bigger hole than having all the otherwise unused features linked into the system.)

Open source code and containment by creating dead code

The rule that anything not used should not be there could also apply to the compilation process of the software. With open-source software you are in the happy circumstance that you are able to disable any piece of code that you are not going to use. For example, if you install bind, and you don't need ipv6 support etc, you just set aside a few hours, look at the code and ad 4 or 5 shortcut lines to the code in the right places in order to cut-out all the unused functionality from the code. With this action you will contain any bug in this code, as this code is now dead code and any bug in it won't be able to do any harm. It is good development practice for developers to already create such dead-code shortcuts for you, so all you might need to do is add some compile flags. unfortunately not all open-source developers think of this in such a way, so many software will have to be done by you by hand.

Don't hesitate to pull the plug

One extreme form of containment is the one you might need to use if you (or your IDS system) has reason to believe that your system has been compromised. The combination of an unsubstantiated fear for false positives, and the false assumption that availability is more important than security. We must realize that the availability problems that could arise if we hesitate to pull the plug on a (sub)system will sustain a greater damage than the damage done by pulling the plug. As long as an IDS detects 'intrusions', and not intrusion-attempts, than pulling the plug on a (sub)system will contain the escalation of the problem, and will even eventually limit the damage to the overall availability.

Vendor responsibility

Software vendors and open-source programmers, in such a big section of the field that I've never seen the positives, tend to think of containment as something that is the responsibility of the user and/or administrator of their software.

I would like to suggest that this approach is flawed, and leads to a high degree of insecurity.
The basic point is that many administrators, and most home-users don't have the skill to set up a decent sandbox for software, and most programmers of things like server software do (or at least should) have this knowledge.
The only way to make the Internet a more secure place where containment is properly implemented is to implement part of the containment in the installation software and procedures of software. Containment measures are at this point very OS Dependant, but than again so is most software. For open source software I would suggest that all network software, and all server or p2p software in particular comes with a 'containment' section in the makefile.
My latest personal project c-duck combines some of the containment features mentioned in this article in a script that is called in the 'containment' section of the makefile. The installation script will create its own chroot environment, adds different users for the different sub systems, creates its own (additional user based ) firewall rules, and even includes its own customized 'pull-the-plug' trivial intrusion detection system. Although the functionality of c-duck itself will be of use only to a limited public, the fact that its setup scripts can be seen as a proof of concept that software vendors can successfully implement containment into their products might be helpful in creating vendor awareness of their responsibility on this point.
Feel free to use and improve this in your own programming, and please keep me updated of your improvements and of the software you create using a 'containment' section.
Rob J Meijer 08/2001