sendmail

sendmailSearch this book
Previous: 23.1 Overview of the QueueChapter 23
The Queue
Next: 23.3 A Bogus qf File (V8 only): Qf
 

23.2 Parts of a Queued Message

When a message is stored in the queue, it is split into pieces. Each of those pieces is stored as a separate file in the queue directory. That is, the header and other information about the message are stored in one file, while the body (the data) is stored in another. All told, six different types of files may appear in the queue directory. The type of each is denoted by the first two letters of the filenames. Each filename begins with a single letter followed by an f character. The complete list is shown in Table 23.1.

Table 23.1: Queue File Types
FileDescription
dfSection 23.2.2, "The Data (Message Body) File: df"Data (message body)
lfSection 23.2.3, "The Lock File (obsolete as of V5.62): lf"Lock file (obsolete as of V5.62)
nfSection 23.2.4, "The ID Creation File (obsolete as of V5.62): nf"ID creation file (obsolete as of V5.62)
tfSection 23.2.6, "The Temporary qf Rewrite Image: tf"Temporary qf rewrite image
xfSection 23.2.7, "The Transcript File: xf"Transcript file
qfSection 23.2.5, "The Queue Control File: qf"Queue control file (and headers)

The complete form for each filename is

Xfident

The X is one of the leading letters shown in Table 23.1. The f is the constant letter f. The ident is a unique queue identifier associated with each mail message.

In the following sections we first describe the identifier that is common to all the queue file parts, then describe each file type in alphabetical order. The internal details of the qf file can vary depending on the version of sendmail, so it is discussed separately at the end of this chapter.

23.2.1 The Queue Identifier

To ensure that new filenames are not the same as the names of files that may already be in the queue, sendmail uses the following pattern for each new ident:

AApid<- prior to V8.6
hourAApid<- beginning with V8.6

Here, pid is the process identification number of the incarnation of sendmail that is trying to create the file. Because sendmail often fork(2)'s to process the queue, that pid is likely to be unique, resulting in a unique ident. For V8 sendmail an extra letter prefixes the AA. Shown as hour, it is an uppercase letter that corresponds to the hour (in a 24-hour clock) that the identifier was created. For example, a file created in hour three of the day will have a D prefixed (the hour begins at midnight with A). [2]

[2] Programs should not depend on the lead letter actually encoding the hour. It is intended only to ensure that all identifiers be unique within any 24-hour period and as an aid to scripts that need to extract information from log files.

If sendmail cannot create an exclusive filename (because a file with that identifier already exists), it clocks the second A of the AA to a B and tries again. It continues this process, clocking the right-hand letter from A to Z and the left-hand letter from A to ~ until it succeeds:

AA            <- start
AB            <- second try
AC            <- third try
 ... and so on
~W
~X
~Y            <- last try
~Z            <- failure

If it never succeeds, the ident ultimately looks like the following and sendmail has failed:

hour~Zpid

This ident is unlikely to ever appear, because the clocking provides for over 1600 possibilities.

All the files associated with a given mail message share the same ident as a part of their filenames. The individual files associated with a single mail message differ only in the first letter of their names.

23.2.2 The Data (Message Body) File: df

All mail messages are composed of a header and a body. When queued, the body is stored in the df file.

Traditionally, the message body could contain only characters that had the high (most significant) bit turned off (cleared, set to 0). But under V8 sendmail, with a version 2 or higher configuration file (see Section 27.5, "The V Configuration Command"), the high bit is left as is until delivery (whereupon the F=7 delivery-agent flag, see Section 30.8.4, F=7, determines whether or not that bit will be stripped during delivery).

Because the message body can contain sensitive or personal information, the df file should be protected from reading by ordinary users. If the queue directory is world readable, then the TempFileMode (F) option (see Section 34.8.68, TempFileMode (F)) should specify minimum permissions (such as 0600) for queued files. But if the queue directory is protected by both narrow permissions and a secure machine, the TempFileMode (F) option may be relaxed for easier administration.

There is currently no plan to provide for encryption of df files. If you are concerned about the privacy of your message, you should use an end-to-end encryption package (not discussed in this book).

23.2.3 The Lock File (obsolete as of V5.62): lf

When old versions of sendmail process a queued message (attempt to redeliver it) they create an empty lock file. That lock file is necessary to signal to other running sendmail processes that the mail message is busy so that they won't try to deliver the message too. Current versions simply flock(2) or fcntl(2) the qf file.

23.2.3.1 Current style file locking

The method that sendmail uses to initially create an exclusive lock when first queueing a file is twofold. First it attempts to creat(2) the file with the argument

O_CREAT|O_WRONLY|O_EXCL

If that succeeds, it then attempts to lock the file. If HASFLOCK (see Section 18.8.9, HAS...) is defined when sendmail is compiled, the file is locked with flock(2). Otherwise, it is locked with a fcntl(2) F_SETLK argument.

23.2.3.2 Locks shown when printing the queue

When mailq is run (or the -bp command-line switch is given to sendmail), the contents of the queue are listed. In that listing, an asterisk that appears to the right of an identifier indicates that a lock exists on the message:

                Mail Queue (1 request)
-Q-ID- -Size- --Q-Time--- ------Sender/Recipient------
MAA17445*   126 Fri Apr 17 10:17 <gw@wash.dc.gov>
        -^                        <ben@franklin.edu>
       note

23.2.3.3 Locks can get stuck

Occasionally, a file will become locked and remain that way for a long time. One indication of a stuck lock is a series of syslog messages about a given identifier:

Apr 12 00:33:38 ourhost sendmail[641]: AA00614: locked
Apr 12 01:22:14 ourhost sendmail[976]: AA00614: locked
Apr 12 02:49:23 ourhost sendmail[3251]: AA00976: locked
Apr 12 02:49:51 ourhost sendmail[5977]: AA00614: locked
Apr 12 03:53:05 ourhost sendmail[9839]: AA00614: locked

An occasional lock message, such as AA00976 in the third line above, is normal. But when an identifier is continually reporting as locked (like the AA00614 lines), an orphaned lock may exist and should be investigated. Use ps(1) to look for lines that list queue file identifiers:

root      5338 160  -AA00614 To wash.dc.gov (sendmail)

This shows that the queued mail message, whose identifier is AA00614, is currently being processed. It the lock on that file is stuck, consider killing the sendmail that is processing it.

23.2.4 The ID Creation File (obsolete as of V5.62): nf

Old versions of sendmail used an nf file when creating a message identifier to avoid race conditions. [3] But contemporary versions of sendmail create the queue identifier when first creating the qf file. The nf file is obsolete.

[3] Historical footnote: This stems from the days when the only atomic file-system call was link(2).

23.2.5 The Queue Control File: qf

A queued mail message is composed of two primary parts. The df file contains the message body. The qf file contains the message header.

In addition to the header, the qf file also contains all the information necessary to:

The qf file is line-oriented, with one item of information per line. Each line begins with a single uppercase character (the code letter), which specifies the contents of the line. Each code letter is then followed by the information appropriate to the letter. The code letters and their meanings are shown in Table 23.2 of Section 23.9, "The qf File Internals".

Here is an example of a version 1 (for V8.8 sendmail) qf file:

V1
T826845694
K0
N0
P30016
I7/4/20
$_you@localhost
MDeferred: Host wash.dc.gov is down
Syou@your.domain
RPFD:george@wash.dc.gov
RPFD:jefferson
H?P?Return-Path: you@your.domain
HReceived: (from you@your.domain) by your.domain (8.8.4/8.8.4)
        id QAA06571 for george@wash.dc.gov; Thu, 14 Mar 1996 16:21:34 -0700 (MST)
H?D?Date: Thu, 14 Mar 1996 16:21:34 -0700 (MST)
H?F?From: Your Name <you@your.domain>
H?x?Full-Name: Your Name
H?M?Message-Id: <199603142321.QAA06571@your.domain>
HSubject: foo
.

This fictional qf file shows the information that will be used to send a mail message from you@your.domain (the S line) to two recipients: george@wash.dc.gov and jefferson (the R lines). It also shows the various headers that appear in that message (the H lines). We discuss the individual lines of the qf file in at the end of this chapter.

23.2.6 The Temporary qf Rewrite Image: tf

When processing a queued message, it is often necessary for sendmail to modify the contents of the qf file. This usually occurs if delivery has failed or if delivery for only a part of the recipient list succeeded. In either event, at least the message priority needs to be incremented.

To prevent damage to the original qf file, sendmail makes changes to a temporary copy of that file. The temporary copy has the same queue identifier as the original, but its name begins with a t.

After the tf file has been successfully written and closed, sendmail calls rename(2) to replace the original with the copy. If the renaming fails, sendmail syslog(3)'s at LOG_CRIT a message like the following:

cannot rename(tfAA00000, qfAA00000), df=dfAA00000

Failure to rename is an unusual, but serious, problem: A queued message has been processed, but its qf file contains old and incorrect information. This failure may, for example, indicate a hardware error, a corrupted queue directory, or that the system administrator accidentally removed the queue directory.

23.2.7 The Transcript File: xf

A given mail message may be destined for many recipients, requiring different delivery agents. During the process of delivery, error messages (such as "User unknown" and "Permission denied") can be printed back to sendmail by each delivery agent.

While calling the necessary delivery agents, sendmail saves all the error messages it receives in a temporary file. The name of that temporary file begins with the letters xf. After all delivery agents have been called, sendmail returns any collected error messages to the sender and deletes the temporary xf file. If there are no errors, the empty xf file is silently deleted. The -d51.104 debugging switch (see Section 37.5.171, -d51.104) can be used to prevent deletion of the xf file.


Previous: 23.1 Overview of the QueuesendmailNext: 23.3 A Bogus qf File (V8 only): Qf
23.1 Overview of the QueueBook Index23.3 A Bogus qf File (V8 only): Qf