Integrating amavisd-new into the Postfix delivery process requires that Postfix is able to delegate messages to external content filters. The minimum version that provides content filtering is Postfix release-20010228.

Chances are that configuration errors during implementation cause Postfix to bounce legitimate messages. Setting the soft_bounce parameter during integration and reloading the Postfix configuration afterwards prevents Postfix from bouncing legitimate mail during that time:

# postconf -e "soft_bounce = yes"
# postfix reload

As soon as soft_bounce has been activated Postfix will treat all delivery errors as temporary errors - any client that wants to send messages to Postfix will keep mail in the mailqueue and it will suspend delivery until the soft_bounce parameter has been removed or set to no.

Once the integration of amavisd-new into the Postfix delivery process has been completed successfully soft_bounce must be removed or Postfix will not generate bounce messages for legitimate mail.

Configuring amavisd-new to work with Postfix answers the following two questions:

$inet_socket_port = 10024;

$notify_method  = 'smtp:[192.0.2.1]:20025';
$forward_method = 'smtp:[192.0.2.1]:20025';

Both, amavisd-new and Postfix, are able to use either SMTP- or LMTP-communication to transport a message from Postfix to amavisd-new. Both variants will be described in this section.

Why configure a dedicated service?

Theoretically it's possible to transport messages from Postfix to amavisd-new using the existing smtp-, lmtp, or even the relay-service in /etc/postfix/master.cf.

In practice transporting messages to amavisd-new requires imposing transport limits on the transporting service. Imposing such limits on a globally available service would impose these limits on the complete Postfix mail system - it would slow down the system significantly and should be avoided.

Note

The number of Postfix clients that may connect simultaneously to amavisd-new instances must be limited to the maximum number of daemon child processes amavisd-new starts. If the Postfix transport client was allowed to open more connections amavisd-new can handle, amavisd-new would start to queue incoming Postfix connections. Postfix in turn would interpret such behaviour as “unresponsive remote MTA” and would itself begin to queue mail that should be filtered. All this would possibly throttle down the complete system and all further filtering attempts would suffer.

2.2.1. Configuring a dedicated lmtp-client

The following example creates a new, dedicated lmtp-transport named amavisfeed in /etc/postfix/master.cf. Its configuration details are explained following the listing:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================

...

amavisfeed unix    -       -       n        -      2     lmtp
    -o lmtp_data_done_timeout=1200
    -o lmtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

	Important
	A noteworthy quote from the Postfix documentation: “...do not specify whitespace around the ‘=’. In parameter values, either avoid whitespace altogether, ...”. Further details on master.cf configuration syntax can be found in master.cf or master(5).

Here's a quick rundown on the settings that differ from other services defaults:

maxproc

The maximum number of concurrent Postfix amavis-service processes has been limited to 2 (default: default_process_limit = 100). This value reflects the default of 2 amavisd-daemon children processes and is a good setting to start from. The value may be raised later, when the system works stable and still can take a higher load. It should not exceed the number of simultaneous amavisd child processes.

lmtp_data_done_timeout

Setting lmtp_data_done_timeout to 1200 (seconds) doubles the default time span a regular Postfix client waits after message delivery for the server to reply DONE to claim successful delivery. It must be larger than amavisd setting $child_timeout (default 8*60 seconds) and should add a sufficient safety margin, for example to cater for periods of automatic database maintenance (e.g. bayes database on non-SQL database types) which can take a long time in some cases.

If the server does not reply within the configured time span, the Postfix client will quit the connection, put the message into the deferred queue, log a delivery failure and retry later to transport the message to amavisd-new.

Note

Raising this value serves a trick amavisd uses to avoid message loss in case of power outage etc. The trick consists in keeping the incoming connection as long open as it takes to filter the message and take appropriate action (reinjection, notification, quarantine, etc.). Only when the message (or notifications etc.) has been reinjected amavisd will send DONE to the client and the client will close the connection. This way Postfix will always keep the message in its own mail queue, where it can be reactivated after a system failure.

lmtp_send_xforward_command

Enabling lmtp_send_xforward_command configures the Postfix lmtp-client to forward the original clients HELO name and IP address to amavisd-new. amavisd-new in turn can use these informations for

• logging and notifications (macro %a)

• switching policy banks (MYNETS, @mynetworks_maps)

• pen pals functionality

• p0f fingerprinting

disable_dns_lookups

The transport route from Postfix to amavisd-new, it will be configured later in Section 3, “Message filtering examples”, will probably never change. It will - probably - only change when the whole mail system is being reconfigured. The target host may therefore be specified as IP address instead of using a DNS hostname. This saves “expensive” DNS-request (3 lookups) and improves performance.

max_use

By default Postfix reuses a service instance 100 times (max_use = 100), before the instance terminates. The master daemon will reinvoke such a service if required. There's no need for the amavisfeed-service to have such a long life-span. Best practice has it to set max_use to 20.

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================

...

amavisfeed unix    -       -       n       -       2     smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

The second service that needs to be added to the Postfix mail system is a dedicated SMTP-server. It will exist only to accept filtered messages and notifications from amavisd-new to transported them closer to their final destination.

This dedicated smtpd server will differ in many aspects from the default smtpd daemon. The most important difference is that it configures an empty content_filter parameter, thus overriding any global external content filtering settings in Postfix.

	Note
	Delegating messages to an external content filter in Postfix is done using the content_filter parameter. If the dedicated smtpd-daemon would not override any global content_filter settings, the reinjected message would be sent of to the external content filter again - the mail would end in an endless loop.

The following Postfix example uses amavisd-new default settings taken from the $forward_method and $notify_method parameters. These settings configure amavisd-new to forward filtered messages and notifications to 127.0.0.1 on port 10025; the Postfix smtpd daemon will be configured to bind to that IP address and listen on the specified port for incoming connections:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================

...

127.0.0.1:10025 inet n    -       n       -       -     smtpd
    -o content_filter=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_end_of_data_restrictions=
    -o smtpd_restriction_classes=
    -o mynetworks=127.0.0.0/8
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
    -o local_header_rewrite_clients=

Here's a quick rundown on the settings that differ from smtpd defaults:

Running the postfix reload will activate the new transports (Postfix will not yet send regular mail to amavisd). Combined with the tail command problems can easily be detected:

# postfix reload && tail -f /var/log/maillog

If there are no problems reported, basic configuration can be tested.

Testing basic configuration consists of three separate tests, starting at the end of the new delivery chain and working to it's beginning. Their goal is to answer the following questions:

$ telnet localhost 10024
220 [127.0.0.1] ESMTP amavisd-new service ready
EHLO localhost
250-[127.0.0.1]
250-VRFY
250-PIPELINING
250-SIZE
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250 XFORWARD NAME ADDR PROTO HELO
QUIT
221 2.0.0 [127.0.0.1] amavisd-new closing transmission channel

$ telnet 127.0.0.1 10025
220 mail.example.com ESMTP Postfix (2.3.2)
EHLO localhost
250-mail.example.com
250-PIPELINING
250-SIZE 40960000
250-ETRN
250-STARTTLS
250-AUTH PLAIN CRAM-MD5 LOGIN DIGEST-MD5
250-AUTH=PLAIN CRAM-MD5 LOGIN DIGEST-MD5
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
QUIT
221 2.0.0 Bye

2.4.3. Testing the new transport chain

This test proves amavisd accepts e-mail as specified in Section 2.1, “Configuring amavisd-new for Postfix”, filters it and finally hands it over to Postfix' dedicated smtpd-daemon as specified in Section 2.3, “Configuring a dedicated SMTP-server for message reinjection”.

The following example uses the content of test-messages/sample-nonspam.txt from the amavisd test-messages to send an e-mail:

$ telnet localhost 10024
220 [127.0.0.1] ESMTP amavisd-new service ready
HELO localhost
250 [127.0.0.1]
MAIL FROM: <>
250 2.1.0 Sender  OK
RCPT TO: <postmaster>
250 2.1.5 Recipient postmaster OK
DATA
354 End data with <CR><LF>.<CR><LF>
From: virus-tester
To: undisclosed-recipients:;
Subject: amavisd test - simple - no spam test pattern

This is a simple test message from the amavisd-new test-messages.
.
250 2.6.0 Ok, id=30897-02, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 079474CE44
QUIT
221 2.0.0 [127.0.0.1] amavisd-new closing transmission channel

The maillog shows the delivery path. Here's an excerpt from a successful delivery process:

Nov  1 11:28:10 mail postfix/smtpd[30986]: connect from localhost[127.0.0.1] 
Nov  1 11:28:10 mail postfix/smtpd[30986]: 079474CE44: client=localhost[127.0.0.1]
Nov  1 11:28:10 mail postfix/cleanup[30980]: 079474CE44: message-id=<20061101102810.079474CE44@mail.example.com>
Nov  1 11:28:10 mail postfix/qmgr[20432]: 079474CE44: from=<>, size=822, nrcpt=1 (queue active)
Nov  1 11:28:10 mail amavis[30897]: (30897-02) Passed BAD-HEADER, <> -> <postmaster>, quarantine: badh-le5gjszxowBk, mail_id: le5gjszxowBk, Hits: -1.76, queued_as: 079474CE44, 39505 ms 
Nov  1 11:28:10 mail postfix/smtpd[30986]: disconnect from localhost[127.0.0.1]
Nov  1 11:28:10 mail postfix/local[30987]: 079474CE44: to=<postmaster@example.com>, relay=local, delay=0.27, delays=0.14/0.05/0/0.08, dsn=2.0.0, status=sent (delivered to mailbox: postmaster) 
Nov  1 11:28:10 mail postfix/qmgr[20432]: 079474CE44: removed

	amavisd connects with Postfix dedicated smtpd-daemon and hands over the e-mail that had been sent during the telnet session. smtpd gives a queue-id of 079474CE44 that can be tracked throughout the maillog.
	amavisd notices it has checked and sent an e-mail to <postmaster>.
	Postfix' local-service logs it successfully delivered an e-mail with queue-id 079474CE44 to the mailbox of postmaster.

If the test fails, the following questions may help to debug the problem:

• Does amavisd-new log errors?

• Does running amavisd-new in debug-mode report errors?

In most cases email policies require global filtering - every inbound and every outbound e-mail must be filtered by amavisd-new - before it may be sent closer to its final destination.

Why check outgoing mail traffic?

Some reasons for checking mail coming from internal networks or from authenticated roaming users are: detect an internal infected PC which is sending viruses detect an internal zombiized PC (or an internal open relay or proxy) which is sending or relaying spam let the SpamAssassin Bayes autolearning feature see a balanced view of all mail, including useful samples of non-spam originating from inside make it possible for pen pals feature to function (if enabled)

In Postfix global settings for its services are written to main.cf. The content_filter parameter, the parameter configuring that messages are sent to amavisd-new, must therefore be placed in main.cf.

The content_filter parameter requires a triplet, consisting of the transport service's name (here: amavisfeed, given in Section 2.2.1, “Configuring a dedicated lmtp-client”), the target hosts IP address and the port where amavisd-new listens for incoming connections. Following the values used in this documents examples the content_filter configuration results in this:

content_filter=amavisfeed:[127.0.0.1]:10024

The new external content filter will be activated once Postfix has been reloaded. Sending a test-mail verifies the system works.

Postfix is able to filter messages per service. Such configuration requires the content_filter not to be applied globally to all services in main.cf (see: Section 3.1, “Filtering E-mail globally”), but selectively, per service in master.cf.

The following example presumes Postfix runs on a system offering three IP addresses. In this example these are: 192.0.2.1 (WAN), 127.0.0.1 (localhost) and 10.0.0.254 (LAN). The goal is to filter only e-mail that enters from the WAN interface.

This requires to create three dedicated smtpd-daemon instances, each binding to one of the given IP addresses and deactivating the global smtp service calling the smtpd command.

Additionally the WAN interface (here: 192.0.2.1:25) is configured to use content_filter =amavisfeed:[127.0.0.1]:10024 - it will delegate any message that enters the Postfix mail system at this service to the external amavisd content filter.

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
# smtp      inet  n       -       n       -       -       smtpd

...

192.0.2.1:25 inet n    -       n       -       -     smtpd
    -o content_filter=amavisfeed:[127.0.0.1]:10024
    -o receive_override_options=no_address_mappings

10.0.0.254:25   inet n    -       n       -       -     smtpd

127.0.0.1:10025 inet n    -       n       -       -     smtpd
    -o content_filter=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_end_of_data_restrictions=
    -o smtpd_restriction_classes=
    -o mynetworks=127.0.0.0/8
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
    -o local_header_rewrite_clients=

Postfix is able to filter e-mails per recipient domain. In order to do this the content_filter parameter must not be set globally (see: Section 3.1, “Filtering E-mail globally”). Instead the content_filter parameter has to be associated with one or more recipient domains listed in a lookup table (map).

Caution

This filter method is not selective! It will send any mail with a recipient domain listed in the lookup table to amavis even if the mail contains another recipient that should not be examined by the amavis framework. If fully selective rules are required all mail should be sent to amavis and amavis' own rule sets should be configured to decide whether a message for a given recipient should be examined or not.

When Postfix searches the lookup table and finds the recipients domain listed as key, it will take the action associated with that domain. The action will send the message to a FILTER amavisfeed:[127.0.0.1]:10024.

The following map /etc/postfix/filter_recipient_domains specifies to send messages to the FILTER amavisfeed whenever a message for any recipient at example.com enters the Postfix mailqueues:

example.com               FILTER amavisfeed:[127.0.0.1]:10024

Once the table has been created the postmap command must be used to create an indexed map Postfix can read:

# postmap /etc/postfix/filter_recipient_domains

Once the map has been indexed, the postmap command is used to test the map. In the following example the postmap command queries for the domain example.com and returns the associated action:

# postmap -q "example.com" /etc/postfix/filter_recipient_domains
FILTER amavisfeed:[127.0.0.1]:10024

The tested map must be added to main.cf, before Postfix can make use of the new filter policy. Setting the check_recipient_access parameter in the list of smtpd_recipient_restrictions triggers evaluation of entries in the map - check_recipient_access is triggered by the envelope-recipient(s) given by a SMTP-client in a SMTP-session with Postfix.

The following example puts the check_recipient_access rule before permit_mynetworks - all clients envelope-recipient(s) will be filtered:

smtpd_recipient_restrictions =
    ...
    check_recipient_access hash:/etc/postfix/filter_recipient_domains
    ...
    permit_mynetworks
    reject_unauth_destination
    ...

Filtering E-Mails per Recipient Domain only from External Clients

This example puts the check_recipient_access rule after permit_mynetworks - only messages sent from clients that are not in Postfix $mynetworks list (external or untrusted clients) will be filtered:

smtpd_recipient_restrictions =
    ...
    permit_mynetworks
    reject_unauth_destination
    check_recipient_access hash:/etc/postfix/filter_recipient_domains
    ...

In general it doesn't make sense to filter e-mails by sender-domain, as anyone can fake a sender-domain during a SMTP-session. Filtering by sender-domain will probably only make sense, if messages are not filtered globally, but e-mails from ones own domain should be checked for spam or viruses before they leave the network.

Most of the configuration steps are identical with the ones noted in Section 3.3, “Filtering E-Mails per Recipient Domain”, except for the parameter that triggers evaluation of the indexed map. In this scenario envelope-senders should trigger map evaluation. The map, named /etc/postfix/filter_sender_domains this time, contains the sender domain (example.com) and associates it with the required FILTER:

example.com               FILTER amavisfeed:[127.0.0.1]:10024

Once the map has been converted and tested with the postmap command (see: Section 3.3, “Filtering E-Mails per Recipient Domain”) it must be added to the list of smtpd_recipient_restrictions using the check_sender_access parameter:

smtpd_recipient_restrictions =
    ...
    check_sender_access hash:/etc/postfix/filter_sender_domains
    ...
    permit_mynetworks
    reject_unauth_destination
    ...

	Important
	The map must be listed before permit_mynetworks, because only then it will be applied to all clients - even the ones Postfix trusts, which are very likely the ones from example.com.

Postfix is able - with deliberate limitations (see: BUILTIN_FILTER_README) - to search for strings in headers, the body and MIME-headers. If a string matches, Postfix may call appropriate action.

The following example configures Postfix to look for the string offer in Subject:-headers and delegate the message to an external content filter if if finds a matching string.

A map, consisting of the search string noted as regexp-expression, associates the search pattern with a FILTER action:

/^Subject:.*offer/   FILTER amavisfeed:[127.0.0.1]:10024

	Indexing regexp- or pcre-maps?
	regexp- or pcre-maps are and must be plaintext files. They must not and cannot be converted to an indexed map using the postmap command. They can be tested using the postmap command using the -q command line option.

Once the map has been created, Postfix must be configured to use it. The following example uses the header_checks parameter (not body_checks or mime_header_checks as they apply to other message parts) to implement the map into the Postfix delivery process:

header_checks = regexp:/etc/postfix/filter_header

Once Postfix has been reloaded it will send every e-mail that contains the word offer in the Subject:-header off to the external amavisd content filter.

To gain more control over a cleanup service than offered by receive_override_options, two (or more) cleanup services, each with its own set of options, must be run. A Postfix setup with more than one cleanup service is possible either with two separate Postfix instances, or through a specification of services and their options in master.cf file of a single Postfix instance.

The following diagram illustrates a setup with two cleanup services in a single Postfix instance:

      .......................................
      :                Postfix              :
   ----->smtpd \                            :
      :         -pre-cleanup-\       /local---->
   ---->pickup /              -queue-       :
      :             -cleanup-/   |   \smtp----->
      :     bounces/    ^        v          :
      : and locally     |        v          :
      :   forwarded   smtpd  amavisfeed     :
      :    messages   10025      |          :
      ...........................|...........
                        ^        |
                        |        v
            ............|...............................
            :           |   $inet_socket_port=10024    :
            :           |                              :
            : $forward_method='smtp:[127.0.0.1]:10025' :
            : $notify_method ='smtp:[127.0.0.1]:10025' :
            :                                          :
            :    amavisd-new                           :
            ............................................

Configuring Postfix smtpd services to use two separate, dedicated cleanup services requires the following steps:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
# smtp      inet  n       -       n       -       -       smtpd

...

pre-cleanup unix    n       -       n       -       0       cleanup
    -o virtual_alias_maps= 

4.2.2. Modifying the existing cleanup service

The already existing cleanup service - having the service name cleanup - will be used to process messages that re-enter the Postfix mail system (also for delivery notifications and forwarding as generated internally by Postfix).

Cleanup jobs that already have been performed by the pre-cleanup service should not be run again. The following example disables typical checks that have been run before or are not needed for internally generated notifications:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
# smtp      inet  n       -       n       -       -       smtpd

...

cleanup unix    n       -       n       -       0       cleanup
    -o mime_header_checks= 
    -o nested_header_checks= 
    -o body_checks= 
    -o header_checks= 

The specified options disable header and body checks as these would already be performed by a pre-cleanup service.

	always_bcc
	This cleanup service would also be the appropriate one for specifying always_bcc option - doing it globally would apply to both cleanup services and would result in two copies of each message to be sent to the specified address.

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
# smtp      inet  n       -       n       -       -       smtpd

...

pickup    fifo  n       -       n       60      1       pickup
    -o cleanup_service_name=pre-cleanup
smtp      inet  n       -       n       -       -       smtpd
    -o cleanup_service_name=pre-cleanup
submission inet n       -       n       -       -       smtpd
    -o cleanup_service_name=pre-cleanup

The most important settings to tune and optimize in Postfix and amavisd workflow are the maximum number of concurrent processes. The maximum number of concurrent processes on both sides must be chosen with care.

If the number is too low, hardware resources aren't used efficiently and delivery time will be unnecessarily prolonged. Experience tells that raising the number of processes a little, will not raise the overall throughput in the same proportion.

As the system resources are nearing saturation with each increase of the number of processes, an increase in throughput becomes marginal, and eventually even negative when the number of processes exceeds its near-optimum value. E-mail throughput will decrease, because processes need to wait for each other. At worst e-mail delivery stalls.

Best practice is to start with a (conservative) maximum number of 2 concurrent processes. Everyday use has shown that this value may be raised to a value between 10 and 30 concurrent Postfix client and amavisd server processes. This also depends on the overall resources the system may provide, how amavisd has been integrated into the Postfix delivery process and on the anti-virus and anti-spam software being loaded and used by amavisd-new.

Regardless of the maximum number of concurrent processes, both sides - Postfix and amavisd - should be synchronized. To synchronize both sides edit, the $max_servers parameter for amavisd-new (see: amavisd.conf) and the number of processes in master.cf listed in the dedicated transports maxproc column for Postfix.

Both values should be identical for two reasons: If amavisd-new offers more processes than Postfix will ever use, amavisd-new wastes resources. On the other hand, if Postfix starts more dedicated transports than amavisd can handle simultaneously, e-mail transport will be refused and logged as error.

Controlling the maximum number of concurrent processes in main.cf

Instead of controlling the maximum number of concurrent processes of Postfix' dedicated transport in master.cf it is also possible to keep the default setting - in master.cf and set the following parameter and option in main.cf: amavisfeed_destination_concurrency_limit = 2 The name of the parameter starts with the service in master.cf (here: amavisfeed) that should be controlled and goes on with the suffix _destination_concurrency_limit. Here also 2 is set as initial (conservative) value.

Further Tuning-Tips can be found in README.performance and the slides from amavisd-new, advanced configuration and management.