Another type of logging that can help you with security is not done by the computer at all; it is done by you and your staff. Keep a log book that records your day's activities. Log books should be kept on paper in a physically secure location. Because you keep them on paper, they cannot be altered by someone hacking into your computer even as superuser. They will provide a nearly tamper-proof record of important information.
Handwritten logs have several advantages over online logs:
They can record many different kinds of information. For example, your computer will not record a suspicious telephone call or a bomb threat, but you can (and should) record these occurrences in your log book.
If the systems are down, you can still access your paper logs. (Thus, this is a good place to keep a copy of account numbers and important phone numbers for field service, service contacts, and your own key personnel.)
If disaster befalls your disks, you can recreate some vital information from paper, if it is in the log book.
If you keep the log book as a matter of course, and you enter into it printed copies of your exception logs, such information might be more likely to be accepted into court proceedings as business records. This advantage is important if you are in a situation where you need to pursue criminal or civil legal action.
Juries are more easily convinced that paper logs are authentic, as opposed to computer logs.
Having copies of significant information in the log book keeps you from having to search all the disks on all your workstations for some selected information.
If all your other tools fail or might have been compromised, holding an old printout and a new printout of the same file together and up to a bright light, may be a quick way to reveal changes.
Think of your log book as a laboratory notebook, except the laboratory is your own computer center. Each page should be numbered. You should not rip pages out of your book. Write in ink, not pencil. If you need to cross something out, draw a single line, but do not make the text that you are scratching out unreadable. Keep your old log books.
The biggest problem with log books is the amount of time you need to keep them up to date. These are not items that can be automated with a shell script. Unfortunately, this time requirement is the biggest reason why many administrators are reluctant to keep logs - especially at a site with hundreds (or thousands) of machines, each of which might require its own log book. We suggest you try to be creative and think of some way to balance the need for good records against the drudgery of keeping multiple books up to date. Compressing information, and keeping logs for each cluster of machines is one way to reduce the overhead while receiving (nearly) the same benefit.
There are basically two kinds of log books: per-site logs and per-machine logs. We'll outline the kinds of material you might want to keep in each type. Be creative, though, and don't limit yourself to what we suggest here.
In a per-site log book, you want to keep information that would be of use across all your machines and throughout your operations. The information can be further divided into exception and activity reports, and informational material.
These reports hold such information as the following:
Time/date/duration of power outages; over time, this may help you justify uninterruptible power supplies, or to trace a cause of frequent problems
Triggering of alarm systems
Servicing and testing of fire suppression systems
Visits by service personnel, including the phone company
Dates of employment and termination of employees with privileged access (or with any access)
This material contains such information as the following:
Contact information for important personnel, including corporate counsel, law enforcement, field service, and others who might be involved in any form of incident
Copies of purchase orders, receipts, and licenses for all software installed on your systems (invaluable if you are one of the targets of a Software Publishers Association audit)
Serial numbers for all significant equipment on the premises
All machine MAC-level addresses (e.g., Ethernet addresses) with corresponding IP (or other protocol) numbers
Time and circumstances of formal bug reports made to the vendor
Phone numbers connected to your computers for dial-in/dial-out
Paper copy of the configuration of any routers, firewalls, or other network devices not associated with a single machine
Paper copy of a list of disk configurations, SCSI geometries, and partition tables and information.
Each machine should also have a log book associated with it. Information in these logs, too, can be divided into exception and activity reports, and informational material:
These reports hold such information as the following:
Times and dates of any halts or crashes, including information on any special measures for system recovery
Data associated with any unusual occurrence, such as network behavior out of the ordinary, or a disk filling up without obvious cause
Time and UID of any accounts created, disabled, or deleted, including the account owner, the user name, and the reason for the action.
Times and levels of backups and restores along with a count of how many times each backup tape has been used
Times, dates, and circumstances of software installation or upgrades
Times and circumstances of any maintenance activity
This material contains such information as the following:
Copy of current configuration files, including passwd, group, and inetd.conf. (update these copies periodically, or as the files change)
List of patches applied from the vendor, software revision numbers, and other identifying information
Configuration information for any third-party software installed on the machine
"ls -l " listing of any setuid/setgid files on the system, and of all device files