5.9.1 Learn What the Normal Usage Profile Is

If you're going to monitor the bastion host, looking for abnormalities that might indicate break-ins or other types of system compromise, you will need to first develop an understanding of what the "normal" usage profile of the bastion host is. Ask these questions, and others like them:

• How many jobs tend to be running at any one time?

• How much CPU time do these jobs consume relative to each other?

• What is the typical load at different times throughout the day?

Your goal is to develop an almost intuitive grasp of what your system normally runs like, so you'll be able to recognize - and investigate - anomalous activity very quickly.

5.9.2 Consider Writing Software to Automate Monitoring

Doing a thorough job of system monitoring is tough. Although the logs produced by your system provide lots of useful information, it's easy to get overwhelmed by the sheer volume of logging data. The important information may often be buried. Too often, the logs end up being used only after a break-in, when, in fact, they could be used to detect - and thus perhaps stop - a break-in while it is occurring.

Because each operating system and site is different, each bastion host is configured differently, and each site has different ideas about what the response of a monitoring system should be. For example, some want email; some want the output fed to an existing SNMP-based management system, some want the systems to trip the pagers of the system administrators, and so on. Monitoring tends to be very site- and host-specific in the details. However, there are some useful tools out there that you should be able to configure and adapt for your own use. The SWATCH (Simple WATCHer) package is a good example.

SWATCH, developed by Stephen E. Hansen and E. Todd Atkins, automates the monitoring of UNIX systems. SWATCH enhances the standard syslog facility in various ways. (See the discussion of syslog in "Setting Up System Logs" earlier in this chapter.) It sifts through the logs as they're created by syslog, and takes certain actions when certain types of log messages are found, e.g., sounding an alert when repeated unsuccessful login attempts are made to the same account, or a "file system full" message is encountered. SWATCH also includes modifications for a number of daemons to make their logging more useful; these include fingerd, ftpd, ruserok, rshd, and login. For example, login has been modified so that it allows only three login atempts; it reports to syslog on any "Incomplete Login Attempt", "Repeated Login Attempt", and "Root Login Refused" events; and it includes the account names attempted and the originating host. SWATCH can also watch files other than ones generated bysyslog. Appendix B gives you information on where to get SWATCH.[4]

[4] The 1993 and 1994 USENIX/SAGE LISA conferences (see Appendix A for information about USENIX, SAGE, and the LISA conferences) have produced a number of papers on other automated monitoring tools that were originally intended for system administration use, but that might be adapted to use in monitoring system security.

SWATCH is written in Perl, which is an unfortunately powerful tool to have sitting on a bastion host; it provides almost everything an intruder could get through having a compiler except the ability to build new kernels. You will probably want to run SWATCH on the machine that the bastion host is logging to, rather than on the bastion host itself.