ANNOUNCE: amavisd-new-2.10.0 has been released

[Mark Martinec]

The version 2.10.0 of amavisd-new has been released.
It is available at:

   http://www.ijs.si/software/amavisd/amavisd-new-2.10.0.tar.xz
   http://www.ijs.si/software/amavisd/amavisd-new-2.10.0.tar.bz2

Release notes are at:

   http://www.ijs.si/software/amavisd/release-notes.txt


amavisd-new-2.10.0 release notes

Contents:
   COMPATIBILITY
   BUG FIXES
   NEW FEATURES


COMPATIBILITY

- New requirement: perl module Net::LibIDN needs to be installed.

- Uses a perl module File::LibMagic if installed, instead of spawning
   a file(1) utility.

- Support for international email relies heavily on perl to do the
   right thing in its support of Unicode, so using a reasonably recent
   version of perl is recommended. Amavisd was tested with perl 5.18
   and 5.20.1. Versions of perl older than 5.12 may cause problems
   with handling, encoding, and decoding of Unicode characters.
   It is reasonable to expect that versions 5.14 and 5.16 are fine too,
   but have not been tested extensively.

- Default log templates and notification templates have changed
   in details (like in decoding of international e-mail addresses), so
   if locally customized templates are in use these will benefit from
   updating - otherwise expect some mojibake in log and notifications.

- International domain names (IDN) encoded in ASCII-compatible encoding
   found in e-mail addresses and in Message-ID header field will be 
decoded
   to Unicode for presentation purposes (syslog, JSON structured log,
   notifications). This decoding does not affect a mail message itself.

- Logging via syslog expects that syslogd (or equivalent) will not
   clobber UTF-8 octets. It may be necessary to tell syslogd to accept
   C1 control characters unchanged, e.g. by adding a command line option
   "-8" to syslogd. Failing to do so may leave logged entries (like
   sender and recipient address, From, Subject) in international mail
   garbled or poorly readable in syslog.
   On FreeBSD one should add:  syslogd_flags="-8"  to /etc/rc.conf.

- Third party log parsers may need updating to accept logs with Unicode
   characters in UTF-8 encoding.

- A SMTP response to an EHLO command will now announce SMTPUTF8 
capability
   by default.


BUG FIXES

- releasing a message from an SQL quarantine was broken in version 2.9.1
   due to introduction of parent_mail_id(); patches provided by Stef 
Simoens
   and Gionatan Danti;

- if checking of a message was aborted prematurely (like due to a 
timeout
   or some fatal error), JSON log could receive a copy of a previous
   log entry;

- prevent non-ASCII non-UTF-8 octets from reaching a JSON log/report
   (which produced an invalid JSON object and Elasticsearch complaining);

- allow SMTP commands MAIL FROM and RCPT TO to accept options without
   values, as allowed by the RFC 5321 syntax;

- in delivery status notification (DSN) the field Received-From-MTA
   specified 'smtp' as mta-name-type, instead of a 'dns' as prescribed
   in RFC 3464;

- releasing from a quarantine left envelope sender address as '<>'
   instead of using the address found in a Return-Path header field
   of a quarantined message, while also logging a warning:
     Quarantine release $QID: missing X-Envelope-From or Return-Path
   reported by Pascal Volk;

- avoid failure of os_fingerprint in certain cases where the
   $os_fingerprint_method uses an asterisk in place of a host IP address
   or port number; the reported error was:
     os_fingerprint FAILED: Insecure dependency in socket
       while running with -T switch
       at /usr/lib/perl/5.18/IO/Socket.pm line 80
   reported by -ben;

- files LDAP.ldif and LDAP.schema: added a missing attribute
   amavisDisclaimerOptions to objectClass; reported by Quanah 
Gibson-Mount;


NEW FEATURES

- added support for Internationalized Email:
   * RFC 6530 - Overview and Framework for Internationalized Email
   * RFC 6531 - SMTP Extension for Internationalized Email (SMTPUTF8)
   * RFC 6532 - Internationalized Email Headers
   * RFC 6533 - Internationalized Delivery Status Notifications

   This supports UTF-8 (EAI) in SMTP/LMTP sender addresses, recipient
   addresses, and message header section. Feature parity with Postfix
   version 2.12 (support introduced in development snapshot 20140715).

   The SMTPUTF8 extension is supported by Gmail since 2014-08-05:
     
http://googleblog.blogspot.com/2014/08/a-first-step-toward-more-global-email.html


- added support for Internationalized Domain Names (IDN) according
   to IDNA (RFC 5890, RFC 5891; RFC 3490);

   * A-labels in ASCII-compatible encoding of domain names are converted
     to U-labels for presentation/logging purposed;

   * U-labels are converted to A-labels when feeding a mail message
     to an MTA which does not announce support for SMTPUTF8 extension
     (instead of rejecting them as invalid mail address);

   * For lookup purposes an international domain name is converted to
     ASCII-compatible encoding when used as a query key in DNS lookups
     and in lookups into hash, list, SQL and LDAP lookup tables (but not
     in regexp table lookups). These tables are expected to contain 
domain
     names in their ASCII representation (ACE). For convenience of config
     files subroutines idn_to_ascii() and mail_idn_to_ascii() are 
available,
     which encode a Unicode domain name to ACE (like ToASCII in RFC 
3490);

   * Many configuration settings may have their domain names in UTF-8.
     These will be converted to ACE automatically where necessary
     (e.g. when creating a Received and Authentication-Results header
     fields, DKIM signatures, mail addresses in notifications, ...).
     These settings include:

       $myhostname, $localhost_name, $myauthservid, $mydomain,
       notification sender and recipient mail addresses
         ($mailfrom_notify_*, $hdrfrom_notify_*, @*_admin_maps),
       domain names and selectors in DKIM signing keys (in calls
         to dkim_key() );


- delivery notifications and admin notifications now show the following
   information encoded as UTF-8 (which is a default $bdy_encoding) in the
   plain text part of the message: IDN domain names in sender and 
recipient
   mail addresses and Message-ID are first decoded to Unicode, Subject 
and
   author display names are MIME-decoded;

- 'amavisd showkeys' and 'amavisd testkeys' can now deal with IDN
   (international domain names): domain names in DNS zone comments
   end up as UTF-8, DNS labels are in ASCII (A-labels); domain names in
   calls to dkim_key() may be specified either as UTF-8 or in ASCII 
(ACE);

- Received trace information in $log_verbose_templ and in notifications
   now include protocol information (the WITH clause from Received header
   fields);

- added a macro 'protocol' to the default $log_verbose_templ template.
   It evaluates to a protocol name by which a message was received by
   amavisd, according to RFC 3848 ("Transmission Types Registration") and
   "Mail Transmission Types" / "WITH protocol types" IANA registration
     
http://www.iana.org/assignments/mail-parameters/mail-parameters.xhtml
   e.g.: SMTP, ESMTP, ESMTPA, ESMTPS, ESMTPSA, LMTP, LMTPA, LMTPS, 
LMTPSA,
         UTF8SMTP, UTF8SMTPA, UTF8SMTPS, UTF8SMTPSA,
         UTF8LMTP, UTF8LMTPA, UTF8LMTPS, UTF8LMTPSA, ...

- new macros: mail_addr_decode, mail_addr_decode_octets, mime_decode,
   header_field_octets, ip_trace_all, ip_trace_public, protocol,
   ip_proto_trace_all, ip_proto_trace_public, client_protocol;
   (documented in README_FILES/README.customize);

- use a perl module File::LibMagic when available, instead of spawning
   a file(1) utility for classifying contents of mail parts.
   By using a direct interface to a libmagic library the startup cost
   of spawning an external process is avoided. Benchmarking shows that
   using libmagic is significantly faster especially for checking a small
   number of files - takes 4 ms for checking one file with libmagic
   vs. 27 ms with a spawned file(1); based on a patch by Markus Benning;


OTHER

- RFC 6533: recognize a MIME type 'message/global' as similar
   to 'message/rfc822', and 'message/global-headers' as similar
   to 'text/rfc822-headers' where appropriate (e.g. in bounce killer);

- header validity check now distinguishes 'non-ASCII and invalid UTF-8'
   from 'non-ASCII but valid UTF-8' characters in a mail header section.
   By default valid UTF-8 strings in a mail header section are not 
treated
   as error even if mail is not flagged as international mail (SMTPUTF8),
   as these are quite common in practice. To treat non- MIME-encoded 
UTF-8
   in a header section as error the test can be enabled by:
     $allowed_header_tests{'utf8'} = 1;

- ORCPT attribute in SMTP 'RCPT TO' command now accepts the original
   recipient mail address in any of these encodings: utf-8-address,
   utf-8-addr-unitext, utf-8-addr-xtext, or as a legacy xtext,
   as required by RFC 6533;

- updated do_cabextract (extraction of Microsoft cabinet .cab archives)
   to recognize a slightly changed output of cabextract version 1.2;
   patch by Thomas Jarosch;

- adjusted some timeouts to leave more reserve for later stages of
   mail processing and forwarding;

- prefer sanitizing/protecting control characters as hex code (like 
\x7F)
   instead of octal (like \177) (e.g. in logging and DSN);




Mark