amavisd-new-20021116 release notes

- provide a mechanism to load only required code sections
  (anti-virus scanning, anti-spam scanning, SMTP/LMTP server module,
  traditional amavis client & milter server module),
  resulting in reduced memory usage and less installation dependencies;

- introduce the per-recipient status handling and make possible some
  per-recipient functionality that was not possible before, such as
  handling the per-recipient spam thresholds. This required a major
  rewrite and cleanup of some sections, hopefully providing cleaner
  mechanism for possible future advances in this area.

  Consider for example a previously incorrectly handled situation
  where there is more than one recipient, and some (but not all) of them
  get a reject from the (outgoing) relay MTA - it is not possible for a
  single SMTP response or status code to describe the situation;

- generate proper (non-)delivery status notifications (DSN), compliant
  with rfc1892 and rfc1894. This form supersedes the simple virus/spam
  sender notifications, but also covers the area of genuine delivery
  failures, such as selective recipient rejects by the outgoing relay MTA,
  which could previously lead to multiple deliveries;

  NOTE: to avoid sender getting two non-delivery messages (one from
  MTA, and another from amavisd), do not set $warnsender and
  $final_virus_destiny=-1 (REJECT) at the same time (and equivalently
  for spam settings);

- enhanced mail system status codes (rfc1893, rfc2034) are now included
  with all SMTP responses and DSN notifications;

- added checking for banned MIME types and names. If any mail part
  matches, the whole mail is rejected, much like the way viruses are
  handled. A list @banned_filename_patterns can be defined to provide
  a list of Perl regular expressions to be matched against
  each part's:

  * Content-Type value (both declared and effective mime-type),
    including the possible security risk content types
    message/partial and message/external-body, as specified by rfc2046;

  * declared (recommended) file names as specified by MIME subfields
    Content-Disposition.filename and Content-Type.name, both in their
    raw (encoded) form and in rfc2047-decoded form if applicable;

  * file content type as guessed by 'file' utility and classified into
    short type names such as .asc, .txt, .html, .doc, .jpg, .pdf,
    .zip, .exe, ... - see subroutine determine_file_types().
    This step is done only if $bypass_decode_parts is not set.

  NOTE: by default the $final_banned_destiny is set to 1 (pass),
  so detected banned file names only cause a header line to be added,
  quarantining, and added address extension - but the mail is delivered
  nevertheless. This default is set so that we can get initial experience
  without being too obtrusive. Change as you prefer..

- besides SMTP/ESMTP protocol, the server now also accepts LMTP protocol
  (rfc2033). This now explains why I had to implement PIPELINING,
  ENHANCEDSTATUSCODES and 8BITMIME SMTP extensions, as these are required
  by rfc2033.

  One advantage of using LMTP to feed content filter is that LMTP uses
  per-recipient status response, as opposed to 'one-size-fits-all'
  SMTP status response, which require that the SMTP client (e.g. content
  filter) is capable of generating DSN.

  Another advantage is specific to the Postfix setup, as Postfix
  LMTP client supports multiple transactions per session, saving on
  connection teardown/reconnect for every message being checked.

  To enable LMTP feed from Postfix, add 'max_use=10' to main.cf, and
  replace last argument 'smtp' with 'lmtp' in the master.cf line:
  'smtp-amavis unix - - n - 2 lmtp'. No changes are needed in amavisd-new,
  both protocols can coexist, the distinction is based on HELO/EHLO
  vs. LHLO command.

  Btw, older versions of Postfix lmtp client inappropriately lowercased
  the envelope addresses. This is fixed in Postfix Snapshot 1.1.11-20021015,
  and in the regular Postfix release 1.2 (when available). Lowercasing
  can be a problem for addresses where local part is case-sensitive,
  although such setups are rare.

- improved per-recipient error condition handling when sending mail via SMTP;

- added command line option '-c config-file' so one can override the
  default location of the configuration file (/etc/amavisd.conf);

- explicitly set PATH and HOME environment variables
  (settings: $path and $helpers_home)

- added another form of lookups: Perl regular expression matching.
  See README.lookups for details. Corresponding new variables in amavisd.conf
  are: $virus_lovers_re, $spam_lovers_re, $bypass_virus_checks_re,
  $bypass_spam_checks_re, $local_domains_re, as well as $banned_filename_re,
  $viruses_that_fake_sender_re, $keep_decoded_original_re,
  $whitelist_sender_re, $blacklist_sender_re.

- besides whitelist_sender* lookup tables, there is now also
  a blacklist_sender* set of tables, which causes mail to be declared spam
  and to skip remaining spam checks. See examples in amavisd.conf.

- provide a configurable lookup table $keep_decoded_original_re
  of file types, for cases where unpacker is not very trustworthy.
  The lookup key is what 'file' utility returned. If the file contents
  match the lookup table, we keep both the original and the unpacked file
  for virus scanner to check;

- provide a configurable list of regular expressions
  $viruses_that_fake_sender_re, which recognizes viruses that usually fake
  envelope sender address, and don't send sender notification in such case.
  For syntax of the new regexp lookup tables see README.lookups;

- specifying per-recipient quarantine address or location is now possible
  by setting $virus_quarantine_to and $spam_quarantine_to be a ref to a
  hash lookup table. Thanks to Vivek Khera for the idea;

- no 'configure' and 'make' for the daemon; all configuration is done
  via amavisd.conf configuration file at the daemon startup time;
  Also the ./mta, ./av and ./notify configuration subdirectories
  are now gone;

- provided a sample init shell script amavisd-init.sh (edit to will,
  and move it to /etc/init.d/amavisd if you want). Based on the script
  from Wil Cooley;

- oneshot debugging mechanism: if envelope sender matches @debug_sender_acl
  lookup table, turn debugging fully up just for this one message and causes
  temp file and directories not to be cleaned. This facilitates debugging
  a particular problem even in the presence of heavy regular traffic;

- cleaner debug log entries for multi-transaction SMTP/LMTP protocol sessions;

- cleaner log entries - avoid misleading and incomplete text
  when quarantine is disabled; thanks to Michael Leone;

- avoid using (nonstandard) field width in the %e format specifier
  when calling strftime, which lead to bad date syntax on some systems;

- local time zone offset is now automatically computed, no more '-0000'
  in rfc2822 dates;

- put Message-ID field value in angle brackets as required by rfc2822
  when generating notifications;

- add a In-Reply-To field to notifications when original Message-ID is known;

- don't bounce a virus (or spam) back to a mailing lists even if
  $final_virus_destiny (or $final_spam_destiny) is set to REJECT;
  a patch by Brian May from the Debian support crew, thanks;

- split lookup tables bypass_checks* into bypass_virus_checks
  and bypass_spam_checks to allow for finer control;

- because some external module may play games with STDIN and STDOUT
  (like SpamAssassin seems to do when local_tests_only=>0)
  run the input protocol directly on the Net::Server's socket,
  not on STDIN and STDOUT, which are aliases to the socket;

- a patch to amavis-milter.c by Didi Rieder to support REJECTing mail,
  by instructing sendmail to return non-delivery notification to sender;

- a new macro %l is available for use in notification messages (via
  'expand'); it evaluates to true (1) if the sender matches @local_domain,
  and returns empty otherwise; by default it is now used in creating
  a 'Subject:' line, inserting word 'LOCAL' before the sender name
  when appropriate;

- new macros %D and %N expand to lists of recipients that got the mail
  delivered (%D), or not-delivered (%N).  Union of both sets gives %R,
  i.e. a list of all recipients as specified in the envelope;

- improved parsing of e-mail addresses according to rfc2821
  (full address literals syntax, etc.);

- proper line folding for generated rfc2822 header fields;

- proper SMTP response wrapping (for very long responses) according
  to rfc2821;

- allow for obsolete rfc822 syntax of permitting whitespace before
  colon in header field name;

- added optional spam-sender nondelivery notifications, based on patch
  from Lazslo E. Miranda (lazslo@dcc.ufmg.br) and Fernando F. Morais
  (frota@cecom.ufmg.br);

- dynamically change process name (Perl variable $0) to reflect
  the process state; suggested by Chip Paswater;

- determine location of external programs (or their absence) at
  startup time; in amavisd.conf one may specify absolute path or just
  rely on PATH. This mechanism is also used to determine absolute path
  of the daemon itself, making reload (after HUP) more predictable;

- explicitly specify lock file for serialization to be used by the
  Net::Server::PreForkSimple module, instead of relying on default
  provided by POSIX::tmpnam. The default approach has a possibility
  for a minor security problem, because the lockfile is created with
  open()..., so it will follow symlinks. Observed, and a patch
  provided, by Jarno Huuskonen;

- new defense against mail bombs: for the cumulative total of all
  decoded mail parts we set max storage size. The formula is:
    quota = max($MIN_EXPANSION_QUOTA,
                $mail_size*$MIN_EXPANSION_FACTOR,
                min($MAX_EXPANSION_QUOTA, $mail_size*$MAX_EXPANSION_FACTOR))
  In plain words (later condition overrules previous ones):
    allow MAX_EXPANSION_FACTOR times initial mail size,
    but not more than MAX_EXPANSION_QUOTA,
    but not less than MIN_EXPANSION_FACTOR times initial mail size,
    but never less than MIN_EXPANSION_QUOTA

- tested with razor-agents-2.20 and SpamAssassin-2.43 (Razor2 is called
  by SpamAssassin). New configuration variable $helpers_home, which
  defaults to $TEMPBASE. Thanks for hints by Chris Hastie and John Stewart;

- to avoid CPU loop in SpamAssassin-2.43 and earlier, my patch needs
  to applied to SA - see amavisd-new web page;

- to avoid taint problem in Razor 2.20 (if SpamAssassin-2.43 is configured
  to call it), my patch needs to be applied to it - see amavisd-new web page;

- chroot available (but not well tested): $daemon_chroot_dir = '/var/amavis'

- provide a failover mechanism for SQL database connect - given a list
  of SQL servers/databases, pick the first that is available.
  Thanks to Ken McKittrick for making available the patch,
  and to Ben Ransford <ben@terakeet.net> for writing it.

- remove existing virus-related and spam-related headers (some of them
  optionally) if we'll be providing our own; suggested by Borut Mrak;

- avoid 'insecure dependency' in lookup_sql when calling DBI::execute
  (thanks to ric* at mpc.com.br)

- fixed macro %H to provide original header lines, not the ones
  stored internally by MIME::Entity, which may have been modified;
  (thanks to Chris Hastie for noticing the problem);


INCOMPATIBLE CHANGES:

- when specifying boolean values to variables in the amavisd.conf file,
  please specify 1 (or old style "yes") for true, and 0 or "" or undef
  for false. The old style "no" yields true for Perl, and is only still
  supported (converted to 0) for some traditional variables for
  compatibility with amavisd(-snapshot);

- removed variables $sendmail_wrapper*, and changed the syntax for
  specifying $forward_methor/$notify_method, with the intention to
  do all mail sending settings at one place. See amavisd.conf for examples;

- variable $mailto is now deprecated (but still works as a fallback default
  for compatibility with previous version). Use %virus_admin and %spam_admin
  lookup tables instead, they also offer a mechanism to specify per-domain
  administrator address;

- variable $warnadmin is no longer used. Use %virus_admin and %spam_admin
  lookup tables instead. Not specifying administrator address
  (e.g. leaving %virus_admin, %spam_admin and $mailto undefined)
  turns off admin warnings.

- dropped variable $LOGDIR, the variable $LOGFILE now specifies the full path;

- dropped old variable $localhost_ip, use $relayhost instead;

- dropped old variable $smtp_port,    use $relayhost_port instead;

- renamed: $warnsender -> $warnvirussender, $warnrecip -> $warnvirusrecip

- dropped variables $enable_relay, $sendmail_cf_orig, $QMAILDIR;
  they were never used in amavisd-new;

- dropped $MAX_ARCHIVE_NESTING, which is replaced by new storage limitations:
  $MIN_EXPANSION_QUOTA, $MIN_EXPANSION_FACTOR
  $MAX_EXPANSION_QUOTA, $MAX_EXPANSION_FACTOR

- SQL database is now specified differently (@lookup_sql_dsn)


MTA-SPECIFIC: sendmail/milter

- $notify_methos now specifies deferred delivery mode ('-odd') by defaut,
  when submitting notifications to sendmail. This is to avoid calling milter
  immediately during submission, which in turn calls amavisd-new, possibly
  leading to a deadlock situation when the number of amavisd-new child
  processes is small. Seems like this change is needed since Sendmail 8.12
  or so.

  The following recommendation is from mimedefang-filter man page:

  | You MUST run a client-submission queue processor if you use
  | Sendmail 8.12.  We recommend executing this command as part of the
  | Sendmail startup sequence:
  |     sendmail -Ac -qp1m


DECODERS-UNPACKERS:

- rewritten decode_parts() to allow for retaining source text if the
  unpacker sw is considered unreliable. This more cleanly resolves
  the problem reported on the amavis-user mailing list on 2002-06-06:
    "Amavisd passing through VBS/VBSWG.gen@MM"
  A nice side-product is that a directory search is avoided for
  each nested unpacking step;

- refined do_unzip to control and limit the size of decompressed
  members (among others it defends against the 42.zip-type bombs);

- rewritten external decompressors from the gzip/bzip2/compress family.
  Instead of using a 'system' call, they are now called through fh_copy,
  making possible to control and limit the size of the decompressed
  contents on the fly, avoiding denial-of-service attacks.
  Affected: b(un)zip2, g(un)zip, (un)compress;

- fh_copy now uses IO::Handle object to assure the forked process
  gets reclaimed even in case of aborted contents extraction;
  stdin gets redirected to /dev/null or to a specifield input file
  for the exec'd process;

- Convert::UUlib is called again (the amavisd-new-20020630 removed its
  usage due to problems with improper decoding). This time the originals
  are kept, so that virus checker sees both the original and the
  attempted-decoded part;

- completely rewritten do_ascii and its usage of Convert::UUlib
  to fix:
  * coding error (bitwise op treated as logical op and improperly negated):
      ... if (!$uu->state || !FILE_OK || -z $newpart);
  * it never reset the state, so if a successfully decoded ASCII file
    contained another ASCII file, each decoding level would decode all
    previously decoded parts again, plus add new ones at this level;
    recursion would not stop until the hard limit, resulting in TEMPFAIL;
  * it never checked nor reported errors that should have been detected
    (I/O errors, out of memory, trouble accessing or creating files);
  * more informative log entry;

- save MIME preambles and epilogues (if nonempty) as extra (pseudo)parts
  to be scanned. This also mitigates the problem of syntactically-incorrect
  MIME mail as produced by some user agents, which (rightfully) gets
  treated as one long preamble by MIME-Tools, and previously went by
  unchecked;

- supports unpacking arc archives using 'nomarch' (by Russell Marks,
  http://rus.members.beeb.net/nomarch.html), (thanks to David D. Kilzer
  for the initial code). Using 'nomatch' fixes a nasty habit of arc
  which gratuitously appends a form-feed at the end of file when using
  the 'p' (pipe) option, which might mislead a virus scanner.
  Besides, 'nomarch' is GPL licensed;

- fixed a 'broken pipe' problem when calling unrar,
  thanks to Ricardo Campos Passanezi and Rainer Link;


VIRUS SCANNERS:

- rewritten interfacing for most command-line virus scanners.
  All settings for them is now done in amavisd.conf. New ones may be
  added without having to modify the daemon source. More complex scanners
  (e.g. daemonized scanners) still need to have a corresponding interface
  routine in the daemon;

- rewritten Sophie and Trophie interfaces to be more resilient
  to Sophie/Trophie daemon restarts during virus database reloads,
  avoiding an unecessary retry (TEMPFAIL) - thanks to Cor Bosman
  for the suggestion and code, and to Dale Perkel for testing the
  Trophie interface;

- make sophos sweep tolerant to encrypted attachments: if all files are
  password protected, then the scanner failure is ignored and the message
  is allowed to pass. A patch by Radu Greab;

- updated nai uvscan interface to recognize the result
  'Found trojan or variant Exploit-CodeBase !!!', a patch by Anton Berezin;

- Clam Antivirus supported;
  
- fixed a problem in 'avp' scanner interface ($TEMPBASE not imported),
  thanks to Joshua E Warchol;

- avpdc (KasperskyLab AVPDaemonClient): recognize additional exit codes,
  based on patch from Christian Hammers;

- Panda new regexps, no TERM vt100 setting (thanks to Benjamin Zwittnig);

- Trend vscan exit code seems to be the number of infected parts.
  Updated the test to reflect that, based on observation from Stephane Lentz;

- MksVir scanner interface returns error code 2 if viruses are removed
  (if --clean option is passed) - a patch by Robert Litwiniec;


Mark

-- 
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  !!  Mark Martinec (system manager)     tel  +386 1 4773-575 !!
  !!  J. Stefan Institute, Jamova 39     fax  +386 1 2519-385 !!
  !!  SI-1000 Ljubljana, Slovenia        mark.martinec@ijs.si !!
  !!!!!!!!!!!!!!!!!!!!!!!!!! http://www.ijs.si/people/mark/ !!!!


---------------------------------------------------------------------------
Previous release notes
---------------------------------------------------------------------------
(the rest may differ from the original posting in small detail,
mostly to fix typos)
---------------------------------------------------------------------------

                                                               June 30, 2002
amavisd-new-20020630 release notes

Since it seems like several people are adapting amavisd-new
in details to their requirements, and certain improvement requests
have much in common, I'm making available the 20020630 release,
shortly before leaving on vacation. As this means my support
will be absent for the comming few weeks, and this release is
perhaps by few days premature, please consider it primarily a
development and new features release.

It is available at the usual location at:

  http://www.ijs.si/software/amavisd/
  (or ask Google about 'amavisd-new')

Having said that, it is still a fully functional and tested version,
and it is running in production at a couple of sites now. Also it is
my version of choice in view of dependability, having it running
at our site while I'm away. If you get into trouble, you can still go
back to amavisd-new-20020517 with which it is fully upwards compatible.

The main changes and features since amavisd-new-20020517 are:

- the code is thoroughly rearranged, interfaces cleaned, separated
  into namespaces (packages), several sections generalized (e.g. lookups,
  appending/editing header lines). The AV scanner and unpacking sections
  are still mostly the same and compatible with amavisd, so whatever
  improvements and new AV scanner support becomes available for amavisd,
  applies almost without a change to amavisd-new;

- SMTP on the input side (used with Postfix and Exim) now talks ESMTP
  (rfc2821) and not just rfc821, including some SMTP extensions:
  command pipelining (rfc2920), message size declarations (rfc1870), and
  8bit-MIME transport (rfc1652). The main reason for this was the change
  in recent Postfix varsions which can now do MIME transformations to support
  7bit transports (implied by SMTP). To keep the transparent 8bit path and
  avoid message transformation by MTA, amavisd-new needs to declare it does
  present an 8bit-clean path. A side benefit is a little speedup in passing
  chunks of mailing list addresses due to pipelining support;

- split certain previous amavisd.conf settings (variables)
  into several variables or lookup tables:

  * sender address for notifications:
    $mailfrom -> $mailfrom_notify_admin, $mailfrom_notify_sender,
                 $mailfrom_notify_recip, $mailfrom_notify_spamadmin
    (these may also be empty to specify null reverse path <>,
    which is most useful for sender notification);

  * administrator address for notifications:
    $mailto -> %virus_admin, %spam_admin (per-sender lookups)

  * %bypass_checks -> %bypass_virus_checks, %bypass_spam_checks

- supports SQL database lookups via Perl module DBI (interface to
  popular database types). Some examples are provided, if you need
  other SQL lookups just modify the code by analogy. SQL lookups
  are most useful for per-user settings of virus_lovers, bypass_virus_checks,
  bypass_spam_checks, spam thresholds etc. when the user base is large
  and subgroups can not be identified through their (sub)domains.
  Another use is for dynamically changing settings without having
  to restart amavisd-new;

- can optionally insert 'Received:' header if acting as a mail relay
  (not with milter); does loop detection as required by rfc2821 section 6.2;

- notification messages now contain 'Date:' and 'Message-id:' headers;

- quarantined viruses contain X-AMaViS-Alert header line with names
  of detected viruses;

- quarantined spam contains X-Spam-Status and X-Spam-Level header lines;

- optionally send spam admin notifications, which include the full
  SpamAssassin spam report and message header;

- when started as root, changes UID and GID to $daemon_user, $daemon_group;

- to facilitate startup scripts and debugging, supports few simple
  command line parameters:
    amavisd        ... standard run: changes uid/gid and daemonizes
    amavisd start  ... same thing
    amavisd debug  ... starts with full debug level, stays attached
    amavisd reload ... finds amavisd master process and sends it a HUP
    amavisd stop   ... finds amavisd master process and sends it a TERM

- some more (minor) configurable options:
    $daemonize, $pid_file,
    $replace_existing_extension, $localpart_is_case_sensitive

- no longer calls Convert::UUlib for uuencoded, xxencoded, and binhex
  attachments. The first two are handled by MIME::Parser, the remaining
  are likely to be handled by anti-virus scanners, especially when some
  virus would use such encoding. The use of Convert::UUlib is dropped
  because it was causing recent problems with garbling virus so that it
  could no longer be recognized by AV scanners, and because the underlying
  library does not seem very dependable. See the thread 'Amavisd passing
  through VBS/VBSWG.gen@MM' from the beginning of June 2002 in amavis-user
  mailing list archives;

- quarantine files now include internal amavis id in the file name,
  instead of the process number; previous naming scheme could stumble
  across a name contention on a busy system;

- only a recommendation: logging via syslog is now preferred to direct
  logging to a file. It serializes the logging, and avoids locking/unlocking
  and reopening a log file by amavisd* for every log entry.
  The syslog daemon does it more efficiently and reliably.

  NOTE for Linux users: make sure you prefix the file name in syslogd.conf
  with a '-' tag to disable fsync after every write; this is most necessary
  for heavy logging such as from MTA and/or amavisd-new.  The downside
  is that you may lose the last few log entries in case of machine crash.
  See man page of your syslogd for details.

- changed examples according to rfc2606 and recent complaints on the
  postfix-users list.


CAVEATS:

- header rewriting is only available in SMTP-in/SMTP-out setup,
  i.e. with Postfix and Exim, but not with sendmail milter setup;

- the SQL lookups are a very recent addition and not so well tested
  as the rest of the program. Also since SQL lookups are supposed
  to introduce per-user settings (e.g. spam thresholds), the code
  still does not accomodate it, and spam thresholds of the first
  recipient in a message affects the whole mesage;

- as Razor2 is still pretty much unstable, I left the spam_scan()
  routine much as it was in the May version. Contributions welcome;

- as I ran out of my time for this release, I didn't prepare a
  separate version without SpamAssassin and Razor 1.20 support,
  so you will need to strip it out if you do not need it.
  Similarly for the required DBI Perl module. I'm very sorry;

- the time stamp used in the 'Date:' header in notifications,
  and in an optional 'Received:' header, does not contain true time zone
  offset, but -0000 instead (standard meaning for unknown). I did not want
  to include a fat Perl package for handling time zones. For now just edit
  the subroutine rfc2822_timestamp() if you want to change that;

- amavisd.conf settings $daemon_user and $daemon_group are not
  automatically set by ./configure options. Please set them manually.

I would like to thank many people on the amavis-user list, on the
postfix-users list, and in private converations, who contributed
valuable ideas and improvements, and offered much appreciated
encouragements.

---------------------------------------------------------------------------
                                                               May 17, 2002
Available at:
  http://www.ijs.si/software/amavisd/

amavisd-new-20020517 is primarily a response to popular demand for
Mail::SpamAssassin support. If amavisd-new-20020224 meets your needs,
there is no urgent need to upgrade. There are no incompatible changes
between these two versions, except some new (optional) amavisd.conf
variables, so you may keep old amavisd.conf file if you wish.

A new file README.exim is now provided, so that Exim 4.x is now a
supported and tested configuration, besides the usual Postfix and
sendmail/milter MTAs. Also included is a brand new qmail amavis client
by Lars Hecking (untested, please try it), plus his updated version of
amavis.c (no longer needed in recommended Postfix and Exim configurations,
and Sendmail milter and qmail configurations use a different client).

Several files are unchanged from the base amavisd CVS release 2002-05-13
and do not reflect the amavisd-new state: FAQ, HINTS, INSTALL, BUGS,
NEWS, TODO, ChangeLog, tests/, doc/. Please start with this file
README.amavisd-new-RELNOTES and follow it. More recent instructions
and last-minute changes are available from the web page.


Changes since amavisd-new-20020224:

- supports Mail::SpamAssassin and Vipul's Razor (1.20 required)
  for spam checking (but not for modifying mail body - only add headers
  and/or address extension, or reject/discard/quarantine spam).

  NOTE: spam checks are off in the default amavisd setup.
        Copy file ./amavis/amavisd.in.all to ./amavis/amavisd.in
        before running ./configure and make, to use the
        spam-check-enabled version!

  Amavisd calls Mail::SpamAssassin directly, avoiding the need to set up
  spamc/spamd or to chain filters. This is more efficient, one daemon
  less to worry about, although maybe less flexible for some taste.
  At the moment the SA per-user database is not used. Feel free to
  experiment with it and let me know what you came up with.

  The usual SA config files are observed, but remote tests are disabled
  by default (Razor is handled directly, RBL lookups can/should be
  performed early by MTA which knows from what IP address mail came,
  and that information is (mostly) lost afterwards). If you feel otherwise,
  change the hard-wired settings in the call:

    $spamassasin_obj = Mail::SpamAssassin->new(
      { dont_copy_prefs => 1, local_tests_only => 1 } )

  (also the Razor score contribution is hard-wired,
  modify: '$razor_spam_found ? 3 : 0' to will).

  Amavisd-new handles Razor checks directly to be able to exercise
  more control over it than would otherwise be available through SA:
  timing, signatures are needed for insertion into header, skips one-liner
  body checks which are common Razor false-positives. To avoid SA calling
  Razor again, either keep the default setting 'local_tests_only=>1',
  or set 'score RAZOR_CHECK 0' in the SA configuration.

  Added are 'whitelist_sender' hash and ACL lookups (see README.lookups),
  which approves spam from specified SMTP originator addresses - SpamAssassin
  can only check and whitelist rfc822 headers, not the envelope addresses,
  and I see no way of passing envelope addresses to it - which is a pitty,
  as important information is lost.

  SpamAssassin checks are computationally quite expensive compared to other
  amavisd activities, and the time needed for SA check goes up significantly
  with the message size. SA check are skipped (but not the Razor check)
  if mail size exceeds 64k. My analysis shows that presently less than 1%
  of spam exceeds 64k characters, and this is probably well below the
  false-negative SA rate, so it is not worth wasting time to check large mail.

  
remaining changes:

- new file README.exim (thanks to Jochen Erwied, Patrice Fournier
  and Igor D'Astolfo);

- updated README.postfix to describe how to avoid running header_checks,
  body_checks and dns_lookups in Postfix twice;

- put back file README.customize, which was omitted from amavisd-new-20020224
  by mistake (but available with previous version and on the web);

- added introductory paragraph to README, the rest still needs to be updated;

- added missing last-step check for '.' in hash lookups to match examples
  and make it more useful;

- added timeouts to certain tasks to make it better suitable for unattended
  operation (less, but still subject to certain DoS attacks, similar to
  the official amavisd). The main reason for adding timeouts is that we
  don't want spam checking to slow down amavisd operation too much: if it
  takes too long, just skip it (assume not spam) and move on;

- change unmangle_sender() to believe sender address for Klez viruses;
  this is sometimes wrong, but seems like people prefer to sacrifice
  few false accusations in favour of some warranted sender notifications.
  Feel free to hack this routine at will (and publish good ideas),
  it is intended to be modified;

- use Perl module Errno instead of errno.ph to avoid using broken
  file errno.ph on some popular platforms;

- make possible to have per-sender-domain administrator e-mail address
  for admin notifications (hash lookup %mailto, sub warn_admin() )

- relax temporary file/dir protection to allow them to be readable
  by the group. This makes possible for anti-virus (daemon) checker
  to run under a different user (but in the same group). This should
  be safer as it makes impossible for virus-checking daemon to clobber
  files. This is only supported in the all-SMTP configuration. To use
  it with traditional amavis clients (e.g. milter), you will have to
  modify their sources to change umask and mods for file/dir they create;

- avoid changing sender address <> to <""> under certain circumstances
  (not strictly wrong, but still a bad idea);
  
- replaced header 'X-Razor-Warning:' with SA-compatible 'X-Spam-Status:' ;
  the 'X-Razor-id:' is still provided to facilitate user spam reporting;

- added '--' between options and argument when calling $sendmail_wrapper
  to be triply sure we avoid problems with some mailers (note that
  $sendmail_wrapper is only still needed in the sendmail setup,
  Postfix and Exim do not need it in the recommended configuration);

- tidy the %local_delivery_aliases mechanism for local delivery / quarantine;

- allow MIME::Parser to decode uuencoded parts, if it feels
  it can (and should) do it;

- write warnings from MIME::Parser to the log (at log level 1 or higher),
  instead of discarding them.


The package is available at the usual location at:

  http://www.ijs.si/software/amavisd/

where also the most up-to-date version or FAQ
and certain other fresh documentation files live.


---------------------------------------------------------------------------
                                                             April 24, 2002
amavisd-new-20020424 is primarily a maintenance release
to summarize one week's worth of experience with amavisd-new-20020418
and to implement some good ideas from the amavis-user mailing list.
It also brings one or two new features.

It is available at:
  http://www.ijs.si/software/amavisd/

or more specifically, at:
  http://www.ijs.si/software/amavisd/amavisd-new-20020424.tar.gz


Changes since amavisd-new-20020418:

- removed Perl 'my' declaration from configurable variables which prevented
  them from being changed in amavisd.conf (thanks to Sebastian Hagedorn
  and Wouter de Jong for reporting)

- introduced child timeouts to prevent bad amavis client from monopolizing
  a child forever (thanks to Sebastian Hagedorn for reporting the problem)

- supported and documented (in README.postfix) configuration where multiple
  remote or local SMTP-in/SMTP-out MTAs (e.g. Postfix) can use the same
  amavisd server, by making it deliver checked mail back to the same
  IP address it came from (see variable $relayhost_is_client in amavisd.conf)
  (thanks to Wouter de Jong for the splendid idea which fitted naturally
  into the overall scheme)

- in certain log messages include the SMTP-in and SMTP-out MTA IP address

- new access list checking for IP addresses - used to limit SMTP access
  to authorized MTAs only: @inet_acl; access control is now enabled by default

- slightly more sophisticated hash-based access list lookups, modeled after
  Postfix map lookups. The sequence now goes through the following steps:
    - hash lookup for user+foo@do.ma.in
    - hash lookup for user@do.ma.in  (only if $recipient_delimiter is nonempty)
    - hash lookup for do.ma.in
    - hash lookup for .do.ma.in
    - hash lookup for .ma.in
    - hash lookup for .in
    - hash lookup user+foo@
    - hash lookup user@  (only if $recipient_delimiter is nonempty)

- updated README.lookups

- cleaner quarantine code; new variable $mailfrom_quarantine allows
  to choose either the original envelope sender, or admin-specified
  fixed sender address; include a special per-user quarantine example
  (look for 'trouble-user-quarantine' in the amavisd)

- fixed problem with localized system error messages - use numeric errno
  instead of strings in Sophie and Trophie clients (thanks to Igor D'Astolfo
  for reporting the problem with italianized version of Linux and for a good
  suggestion)

- successfully tested with Exim
  (thanks to Igor D'Astolfo for testing and for reporting
  the <<doubly-quoted>> sender (but not recipient!) problem)

- wrong variable used in the spam section ($final_virus_destiny
  instead of $final_spam_destiny) (thanks to Wayne Smith for reporting
  and for testing Vipul's Razor 'plugin' patch)

- changed pattern match in the ./av/oav to support new version of
  OpenAntiVirus ScannerDaemon (thanks to Rainer Link)

- changed 'configure' to make it recognize Sophie 1.33rc1
  (thanks to Igor D'Astolfo and Lars Hecking)

- include acinclude.m4/acx_pthread.m4 macro
  (thanks to Rainer Link)

- updated README.postfix: includes instructions on how to avoid body_checks
  and header_checks for reinserted mail (contributed by Wayne Smith,
  works nicely, requires Postfix version 1.1.7-20020331 or later)

- clarified comments in amavisd.conf

- optionally keep existing X-Virus-Scanned: header lines, or remove them
  before adding our own header line - see $remove_existing_x_scanned_headers
  (requested by Darryl Harvey)

- avoid historical (misleading) parameter name $localhost_ip;
  use $relayhost and $relayhost_port instead, but take old variables into
  account for backwards compatibility with existing amavisd.conf files

- Here is an overall picture (sequence of events)
  of how pieces fit together:

    bypass_checks? ==> PASS
    no viruses?    ==> PASS
    log virus     if $log_templ is nonempty
    quarantine    if $virus_quarantine_to is nonempty
    notify admin  if $warnadmin
    notify sender if $warnsender
    notify recips if $warnrecip
    final_destiny==pass?  ==> PASS
    virus_lovers?         ==> PASS
    DISCARD or REJECT (depending on final_*_destiny)


---------------------------------------------------------------------------
                                                             April 18, 2002
This it to announce the second release of amavisd-new-20020418,
available at:

  http://www.ijs.si/software/amavisd-new-20020418.tar.gz

It is a version of amavisd (a daemonized AMaViS, which is an interface
between MTA and virus scanners), based on amavisd CVS from today (20020418)
(same configuration, amavis clients in C, instructions, AV client code),
while also being a successor of the initial release of amavisd-new-20020329,
which is a performance-enhanced pre-forked Net::Server -based amavisd
with SMTP-in/SMTP-out capability, written in Perl.

(I tried to make this release based on amavisd-snapshot-20020300, but failed,
as that version still has $errval semantics bug (the bitwise-operations
problem was discussed on the AMaViS-user list some time ago).
This was fixed in the CVS version and most AV clients were changed then.
As the CVS version contains AV client code for new AV scanners, this makes
it incompatible with config stuff from amavisd-snapshot-20020300.

If you have problems with ./configure or make, these problems would be
common to both versions. Also the unpackers and decompressors code is mostly
the same for all recent amavisd versions, so any problems in this arena
(like DoS mishandling) are most likely to be common for both the official
amavisd and the amavisd-new version.

Compared to the first release of amavisd-new-20020329, the second release
brings further significant performance improvements especially in the
SMTP-in/SMTP-out configuration, e.g. with Postfix, but also brings
some interesting new features and new configuration possibilities.

No important bug fixes were needed, so upgrade at your leisure if you are
running my initial version. The Postfix users would perhaps want to rush
a bit though, to put new performance improvements into use.

The summary of changes since the initial version:

- significant SMTP-in speedups (25% with fast AV scanner), file reuse
- pass reject reason to MTA on the input side
- more informative MTA log entries in the SMTP-in/SMTP-out setup
- amavis internal id (am_id) in log entries and passed to MTA in SMTP response
- ISP features: specify subgroups of users who want to receive viruses
- address extensions: e.g. user@domain -> user+virus@domain if virus detected
- can specify final_virus_destiny: reject, discard, pass
- quarantine new options: save to individual file, save to mailbox, pass to MTA
- new headers in quarantined viruses preserve envelope addresses
  and quarantine id (similar to the suggestion from Furio Ercolessi - see code)
- detailed timing breakdown report for each passed message
- anti-spam hooks and examples, example patch to integrate Vipul's Razor client
- body cache now always enabled - heavy speedup for mailing list bursts
- Sophie 1.33-ready
- rewritten Trophie client, based on new Sophie client code
- rewritten README.postfix, describing new setup possibilities
- new file README.lookups (to be used with virus_lovers and bypass_checks)
- new file README.customize (same as in the initial version)
- new file README.performance (unfinished)
- new amavisd.conf options, documented there
- code heavily commented, cleaned, generalized again
- does not accuse innocent users of sending viruses if we are suspicious
  of sender address (see FAQ below) (same as in the initial version)

CAVEATS:

- no test mode;

- only Postfix and sendmail milter are fully supported and tested;
  to integrate with other mailers one would need some understanding of their
  operation to set up properly; contributions are welcome;

(P.S. note: this is probably a non-issue, related to the file system problem:
  All amavisd versions seem to share one still unresolved problem, probably
  with amavis-milter.c client, its use of libmilter, or perhaps even in the
  libmilter code itself - see recent thread on the AMaViS-user list
  (subject: Leftover email.txt files with amavisd 'standard' also).
  The problem is most pronounced in burst of heavy traffic.
)

FAQ:

- Net::Server 0.82 triggers a Perl 5.005 bug (the problem is obvious:
  you get syntax errors). Either upgrade to Net::Server 0.83, or
  upgrade your Perl - 5.6.x should be ok.

- if you intend to play with customized notifications, it is wise
  to remove the '-t' option from $sendmail_wrapper_args in amavisd.conf.
  That way you are free to screw up notification mail headers any way you want,
  and the message would still be delivered to the correct recipients.
  Removing '-t' is now the recommended setting, but both variants should work.
  If you use SMTP-out method for notifications (e.g. Postfix recommended
  and default setting), this does not affect you.

- if you see virus notifications claiming the virus originator is <?>
  or <?@some.domain> and sender notifications are not sent, this is not a bug,
  but a feature - see comments at the subroutine unmangle_sender().
  The original idea comes from Furio Ercolessi: as some viruses tend to use
  forged or corrupted sender or 'From:' addresses, we try to determine
  the true virus sender, and if we can not do that, we avoid accusing
  innocent users of sending viruses.

- if you kill or HUP amavisd, temporary directories may be left undeleted;
  this is normal and mails are not lost;

- if amavisd does not restart after receiving HUP, a possible reason may be
  that amavisd can not be found in the path as set in the $ENV{PATH} variable
  (near the beginning of amavisd program). Another reason may be a syntax
  error if you changed the amavisd.conf file. Try to start it manually:
    $ su vscan -c amavisd
  If that does not make you wiser, set $DEBUG = "yes" and retry.
  This is also the recommended first-time start method.

- after changing $inet_socket_bind in amavisd.conf, you must stop amavisd
  and start it anew. The HUP method causes amavisd to stumble over its feet.

Below is my announcement notice for the initial version of amavisd-new,
just slightly edited to remove some mistakes.


---------------------------------------------------------------------------
Date: Sat, 30 Mar 2002 04:13:25 +0100
From: Mark Martinec <Mark.Martinec@ijs.si>
Subject: [AMaViS-user] ANNOUNCE: new amavisd - leaner and meaner
To: amavis-user@lists.sourceforge.net
Message-id: <01KFYI1DJ2O200AMKT@CATHY.IJS.SI>

I would like to announce an updated version of amavisd,
based on recent CVS code of amavisd (which is not far
from the February amavisd snapshot). It is a result of my
three weeks work on the code, caused by our needs for:
- better reliability;
- higher throughput (less overhead);
- versatility (e.g. separating amavis and MTA hosts, load sharing),

Available at:
  http://www.ijs.si/software/amavisd-new-20020329.tar.gz

This is now finally a version which I can recommend to friends :)
TODO: better DoS handling in unpackers.

Main features - in brief:

- pre-forked reusable children - saving on process creations;
- persistent connections to certain AV scanners, e.g. Sophie, saving on forks;
- both SMTP and pipe (sendmail wrapper) interfaces independently available
  and configurable on all three sides: input, output, notifications;
- sendmail Milter interface supported and tested; Postfix supported and
  thoroughly tested (Exim untested; classical sendmail untested, no qmail);
- customizable notification messages;
- compatibility with existing configurations;
- cleanups, generalizations, speedups, fixes, better code documentation;
- HUP signal causes restart with new configuration;
- ISP feature: certain recipients may be allowed to receive viruses
  (with alert header line added (not with milter), notifications are
  still generated);
- anti-spam hooks, caching-ready, Vipul's Razor interface in a form
  of a patch included - should make integration with SpamAssassin easier.

Install:

  Unpack the tar over the checked-out CVS version of amavisd branch
  (or February snapshot). It overlays some files, the rest is unchanged.
  For milter interface make sure you use amavis-milter.c from CVS,
  as the snapshot version contains a bug which can cause message loss
  if amavisd dies.

Details:

- child and socket handling is now delegated to Perl module Net::Server,
  which gives us pre-forked children which are able to do more than one
  mail-check during their lifetime, saving on process creations
  and giving better response time;

- Net::Server controls number of children, does signal handling,
  takes care of dead children, handles listening on multiple sockets
  (both Unix and TCP/UDP), delegating tasks and synchronizing 'accept's;

- as a consequence, certain AV-scanner interfaces (most notably Sophie,
  and soon to follow Trophie) can keep persistent connections to the
  AV checking daemon, saving on AV scanner process creations
  (forks on accept), and socket setups/teardowns;

- on the input side: both SMTP (TCP) and traditional amavis client
  protocol (Unix socket) are now accepted - even both at the same time
  (by default) to ease transition to SMTP interface;

- new input-side SMTP interface (SMTP server) is easier to set up
  (no need to worry about file ownerships, UIDs and GIDs),
  and more versatile (e.g. SMTP responses carry more information
  that sysexits.h-based status codes, SMTP can talk to remote host);
  It is based on Perl native I/O and it is quite fast - no OO overhead,
  _not_ SMTP::Server-based (which I do not consider production quality);

- a nice by-product of SMTP input interface is the extra information
  available in MTA logs, e.g. Postfix log:
  postfix/smtp[7656]: DA7B147FA6:
    to=<user@domain>, relay=localhost[127.0.0.1], delay=5,
    status=sent (250 Ok, discarded - VIRUS: EICAR-AV-Test)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- notifications messages can be delivered either via SMTP or
  as traditionally via pipe to sendmail wrapper / MTA pickup.
  This means that in the Postfix setup for example, one needs
  to worry only about one output interface (SMTP or pipe).
  This also saves unnecessary work of checking the just-generated
  notification messages for viruses;

- the output pipe interface is now more careful with status
  checking and forking;

- output SMTP interface error handling more compact;

- sender notifications can be sent to more than one contact address
  (not used at the moment, but the mechanism is available);

- sender address unmangling patch incorporated (but I'm willing
  to take it out if considered inappropriate for the base distribution);

- see file ./amavis/mta/postfix_init for comments describing
  how to select SMTP- or pipe-based output interface
  (for reinjection and notifications);

- customizable notification messages and log entry text
  (see README.customize);

- clean notion of when mail addresses are in their quoted and
  when in unquoted form (RFC2821). All internal handling uses unquoted
  form, addresses get quoted as required by the output interface,
  and quoting gets stripped away as required by each input interface;

- besides traditional choices where a mail can be forwarded/accepted,
  discarded, or temporary failure/retry indicated, there is now
  a fourth choice: REJECT. Depending on the MTA this requires a
  message rejection to be done by the input side MTA itself.
  This comes handy in cases when amavisd accepts a non-infected mail,
  but outgoing MTA does not want to take it back for final delivery,
  e.g. in case of some policy violation. Traditionally amavisd
  would indicate temporary failure on its input side, causing the
  message to be retried and re-scanned over and over again,
  without having a chance of ever being accepted;

- all existing virus scanners are still there (one little change
  in each: replaced a call to do_virus with return 1);
  Sophie client rewritten to take advantage of persistent connections;

- decoding sections are mostly unchanged. This area needs more work
  in the future;

- includes exit status codes from sysexits.ph
  instead or having them hard-wired;

- anti-spam code easier to integrate into amavisd due to some code
  rearrangements; a patch to integrate Vipul's Razor is included
  as an example, but it may be better to tie amavisd with SpamAssasin.
  Anti-spam code will NOT be integrated into amavisd, but the least
  we can do is making it easier for people to add their own code;

- cache-ready (example in the included Razor patch). This works by
  calculating a message digest (hash, signature) of the message body
  and keeping it in storage for a short while, e.g. for the lifetime
  of a child process (10 consecutive requests by default).
  If another message with the same body content arrives in the near future
  we can skip a virus check. This comes handy where mailing list traffic
  is frequent, especially if we have to deal with poorly done mailing
  list managers or heavy spam traffic;

  To support this concept the functions of MIME decoding and
  unpacking of archives are now separated.

- MIME parsing is now supplied with our own Filer subclass.
  This was necessary to avoid MIME parser complaints when it tried
  to reconstruct file name extensions from file names in strange
  character sets. Its work was completely unnecessary and harmful,
  since we want to supply our own file names and do not care
  for file name extensions.

- as a consequence of our own Filer subclass, we now avoid the first
  (or the only) directory traversal (reading) in the first decoding pass;
  Not much, but every little bit counts.

- when using sendmail wrapper it is no longer necessary to supply
  the sendmail -t option. If -t is not specified, sendmail wrapper
  receives addresses via command arguments (exec, no shell),
  which is more reliable than having to parse mail headers - which are
  now more error-prone due to user-customizable notifications.
  The -t option is still supported though, but not recommended.

- do_unzip no longer complains with multi-line backtrace when it dies;
  just a single message is issued, like with other decoders;

- HUP signal causes restart and re-reading of config file;

- many new comments, code unifications, supplied some missing
  error checks, code generalizations;


MTA support:

- thoroughly tested with Postfix in all combinations of input,
  output, and notifications interfaces; in production use;

- big thanks to Sebastian Hagedorn who helped to test the milter
  interface on his Solaris!

- Exim and traditional sendmail interface untested, but should
  work without much work.

- Qmail interface is still missing, like in current CVS amavisd.
  I believe the amavis client for qmail can be written as
  a few-dozen line Perl program. Anybody want to try?


A quick cookbook on how to set up Postfix / amavisd interface
using SMTP on both amavisd input and output side (including notifications).
Amavisd by default now accepts both Unix socket and SMTP on the loopback
interface, so the transition is easier: first install new amavisd,
then at some other time change Postfix configuration (if desired) to:

master.cf:
  # MTA -> amavisd
  smtp-amavisd      unix - - n - 2 smtp

  # amavisd -> MTA
  localhost:10025   inet n - n - - smtpd -o content_filter=

main.cf:
  # choose transport to amavisd
  content_filter = smtp-amavisd:localhost:10024

[see new file README.postfix for details]

The amavisd can now easily be located on a different host than MTA,
also Postfix load-balancing transport methods can be used (e.g. multiple
MX records). Although amavisd now talks SMTP and incorporates some rudimentary
defenses against malicious SMTP clients, do not expose its SMTP server
directly to the world - always front-end it with MTA. By default it
binds to the local interface only, but other access restrictions are
also available.

Happy amavising!  Experiences and comments are most welcome.
