4.5.1 - Function

The Deliver Task is a NAFlet which delivers stored messages onwards to their next-hop destination. It is one of the big two Mailismus naflets.

The Deliver task is composed of an organising agent called the Sender (or Forwarder), which obtains pending messages from the queue (see §4.3), and hands them to SMTP Clients.
The SMTP Clients are pre-allocated software objects, which connect to the remote servers and perform the necessary SMTP protocol operations to deliver the messages.

The Deliver task starts by fetching a large batch of messages from the queue, and it allocates this batch optimally to a set of client objects. This batch functions as a temporary in-memory copy of part (or all) of the queue, and is therefore called the queue cache.
Optimal allocation means that if the same message has multiple recipients in the same domain, they will be allocated to the same client, so that the message can be sent once with multiple recipients, rather than being sent multiple times, in multiple connections, with single recipients. When a client has finished delivering a message, it sieves the cache to find any more messages for the same destination, and sends them within the same TCP connection. The client doesn't disconnect until it can't find any more messages for that server in the cache, and the cache is ordered to optimise the process of scanning for related entries.
Once all the clients in the current batch have completed their work and disconnected, the cache is flushed back to the persistent queue. It is now that successfully delivered messages can be removed from the queue, while failed messages get scheduled for their next retry.
The Deliver task will retrieve a new batch of messages after a short pause, and the cycle begins again.

Note that a single cached batch is not necessarily consumed in full. No individual SMTP client will connect more than once per batch, so if the batch contains a preponderance of messages to the same destination (as in more messages than the clients are configured to send in one connection), some of those messages will be left untouched. However, they will just get picked up again in the next batch (and will presumably be nearer the front of the queue).

It is important to stress that this task's activity happens not only within one process, but also within one thread, which is why the Deliver task scales up so well to loadings of thousands of simultaneous connections.

4.5.2 - Task Config

<deliver>
    <audit rot="daily">%DIRLOG%/audit/Y%Y%/M%M%/delivered.log</audit>
    <interval_low>1s</interval_low>
    <interval_high>30s</interval_high>
    <interval_error>3m</interval_error>
    <delay_start>60s</delay_start>
    <maxconnections>1000</maxconnections>
    <maxdomainconnections>5</maxdomainconnections>
    <maxmessages>100</maxmessages>
    <maxrecips>50</maxrecips>
    <queuecache>5000</queuecache>
    <client>
        ...
    </client>
</deliver>

audit
This specifies an audit log, reporting the delivery of each message to each recipient.
It is meant to serve as a formal record of activity, suitable for archiving (as opposed to the more debug-oriented trace logs). It records the delivery timestamp, the sender and recipient email addresses, and the IP addresses the message was received from and delivered to.
This config item supports the same attributes as the GreyLog loggers (rotation, bufffering, etc - see GreyLog Guide).
This config item has no default and auditing will be turned off if it is omitted.

interval_low
If any entries were found when we polled the queue to build the latest cached batch, this is the pause before polling again.
Defaults to 10 seconds.
NB: The queue polling frequency is restricted when in demo-licence mode, so see chapter §8 for the impact it has on this setting.

interval_high
If no entries were found when we polled the queue to build the latest cached batch, this is the pause before polling again.
It defaults to 45 seconds, and this low/high split delay allows us to avoid thrashing the queue when it has no pending messages.

interval_error
If we encountered an error when we polled the queue to build the latest cached batch, this is the pause before polling again.
Defaults to 3 minutes.

delay_start
This specifies the pause on initial system startup, before we first poll the queue.
Defaults to 1 minute.

maxconnections
This specifies the max number of simultaneous outgoing SMTP connections.
Defaults to 1,000, unless in smarthost mode in which case it defaults to 10.

maxdomainconnections
This specifies the max number of simultaneous outgoing SMTP connections to any one domain.
Defaults to 5, and is irrelevant in smarthost mode, since all connections are made to the smarthost.

maxmessages
This specifies the max number of messages we will batch into a single SMTP connection.
This setting is the one that mainly controls how long it can take to process a single batch, and it defaults to 100. This limit is based on ensuring that a batch completes in a reasonable time, to minise the latency before we flush it back to the queue.

maxrecips
This specifies the max number of common recipients we will batch into a single message.
Defaults to 50 (well below RFC-5321 minimum of 100, just to be conservative).

queuecache
This specifies the max size of the delivery batch, measured as the number of cached queue entries (each queue entry constitutes a unique recipient/message combination).
This defaults to 5 x maxconnections which is 5,000 for the default settings.
It is subject to the limit imposed by the queue manager itself, with its maxmemoryqueue setting (see §4.3.1).

NB: The max batch size is restricted when in demo-licence mode, so see chapter §8 for the impact it has on this setting.

4.5.3 - SMTP Client

The client's config block is listed below, with largely default values.

There is one config setting which is of major relevance to the client but is specified outside this block, and that is the served_domains block. See §4.2.
The client loads the entries for which an ip is specified, and ignores any domains with no route specified.

<client>
    <transcript>DIRLOG%transcripts/smtpclient.log</transcript>
    <stats_interval>1h</stats_interval>
    <relay_server>relayhost:8825</relay_server>
    <announcehost>hostname</announcehost>
    <maxpipeline>25</maxpipeline>
    <timeout>2m</timeout>
    <recvbufsize>128</recvbufsize>
    <xmitbufsize>96</xmitbufsize>
    <filebufsize>1M</filebufsize>
    <sayHELO>N</sayHELO>
    <fallbackHELO>N</fallbackHELO>
    <sendQUIT>Y</sendQUIT>
    <waitQUIT>Y</waitQUIT>
    <remotenets>
        <remotenet ip="%SYSNAME%/24">
            ...
        </remotenet>
    </remotenets>
</client>

transcript
This config item supports the same attributes as the GreyLog loggers (rotation, bufffering, etc - see GreyLog Guide).
This config item has no default and transcripts will be turned off if it is omitted.
See §4.4.7 for a description of the transcript contents.
The only difference with the server transcripts, is that in this case, we're the ones initiating the connections and sending the commands.

stats_interval
The Client prints various stats to the logfile at a recurring interval, which is 1 hour by default.
Set to zero to disable stats.
Here is a sample of the output:

14:00:00.197 INFO Stats since 13:00:00 - Conns=182 (peak=25, failed=15), Msgs=190, Recips=455 (ok=455)
15:00:00.199 INFO Stats since 14:00:00 - Conns=110 (peak=7), Msgs=102, Recips=102 (ok=102)
16:00:00.197 INFO Stats since 15:00:00 - Conns=150 (peak=13, failed=11), Msgs=199, Recips=400 (ok=400)

The first line tells us that 182 connections were initiated in that period, of which 15 resulted in a failure to connect (this includes failed DNS lookups, where we never even got as attempting an SMTP connection) and simultaneous connections peaked at 25. We tried sending a total of 190 messages to 455 recipients (as many messages contain multiple recipients), which resulted in all 455 message-recipient pairs being successfully accepted.
The second line illustrates a case where all connections succeeded.

relay_server
This specifies a default server to which all emails should be relayed. We say "default" because matching entries in the served_domains block take precedence, and this relay server only comes into play for domains not specified in there.
If there are no served domains, then this relay server becomes a full-blown smarthost, ie. we forward all our emails straight on to it.
Obviously this entry is blank by default, and the above example demonstrates how we can relay to a non-default TCP port.
One effect of specifying a relay_server (whether or not we also have a served-domains block) is that the SMTP client will never have to perform any DNS lookups.

announcehost
This specifies how the client will announce itself in its greeting message (SMTP: HELO or EHLO).
If absent it defaults to the setting in the system block - see §4.1
Unlike its namesake in the SMTP-Server config, it is critical for clients that this hostname is a valid DNS entry which matches the IP address we call out on, and ideally has a reverse entry as well (ie. the in-addr.arpa DNS domain).

maxpipeline
This controls how aggressively the client pipelines its outgoing requests, when the remote Server advertises its support for the ESMTP pipelining extension.
The default is 25, meaning we will have at most 25 outstanding requests, before we pause to wait for responses.
Setting this to 1 effectively disables pipelining, regardless of what the Server supports. A value of 0 is also interpreted as being 1.

timeout
This specifies the max idle period, after which the client will declare failure and abandon the connection.
The default is 2 minutes, and zero would mean no timeout, which is highly inadvisable.

recvbufsize
This specifies the size of the I/O buffer in which the client holds the incoming SMTP responses.
The default is 128 bytes, and there is no reason to change that.

xmitbufsize
This specifies the size of the I/O buffer in which the client dynamically constructs outgoing SMTP requests.
The default is 96 bytes, and there is no reason to change that.

filebufsize
This specifies the size of the I/O buffer the client uses for the data phase of the transaction, ie. sending the message body.
It is distinct from xmitbufsize because the Client uses an optimised Java interface to transmit the message body directly from the fileystem buffers. Unlike the receive and transmit buffer sizes above, this setting does not represent a per-client memory cost.
The default is 1 MB.

sayHELO
This tells the Client to use plain SMTP rather than ESMTP, by issuing the SMTP HELO greeting rather than EHLO.
The default is to announce our ESMTP capability, by saying EHLO.

fallbackHELO
This tells the Client to fall back to issuing a HELO command if the EHLO. is rejected.
The SMTP standard recommends falling back to HELO, but also mandates that servers must support EHLO, and you're very unlikely to come across any servers that don't support EHLO nowadays.
For that reason, this setting defaults to No (don't fall back) as falling back merely delays recognition of the far more likely possibility that our EHLO was rejected because of who we are, rather than because the server doesn't support EHLO.

sendQUIT
Setting this to No enables the client to accelerate the standard SMTP end-of-connection dialogue, by disconnecting abruptly without sending the SMTP QUIT command.
This default to Yes (ie. send QUIT), as it is not generally considered polite behaviour to take this shortcut. It does not affect the message delivery either way, as the message transfer phase of the connection is already over at this stage.

waitQUIT
If we issue an SMTP QUIT command, this controls whether we wait for the server's response, or accelerate the process by disconnecting immediately after sending the QUIT.
Clearly the message transfer phase is over at this, and we've already issued our final SMTP squawk, but abruptly disconnecting before the server has had a chance to respond to the QUIT may result in spurious errors in its logfiles, so for the sake of orthodoxy and politeness, the default is Yes, ie. we do wait for the QUIT response.

remotenets
This serves the same purpose as in the Server config. It is an optional block containing one or more nested remotenet blocks. Each remotenet block has an ip attribute which specifies an IP address block in CIDR notation, but you may specify a hostname in place of the IP part (as illustrated above).
The purpose of this is that each remotenet block identifies a family of remote servers (ie. those whose IP matches the CIDR block) to which you want to apply alternative config settings. There are no new settings defined for the remotenet block, but it may override most of the settings defined in the common client config.
The full list of settings that may be overridden within a remotenet block is:
• announcehost
• maxpipeline
• timeout
• sayHELO
• fallbackHELO
• sendQUIT
• waitQUIT