ANNOUNCE: amavisd-new-2.11.0-rc1 release candidate is available

[Mark Martinec]

A release candidade RC1 of the coming version 2.11.0
of amavisd-new is available at:

   https://www.ijs.si/software/amavisd/amavisd-new-2.11.0-rc1.tar.xz

Release notes are at:

   https://www.ijs.si/software/amavisd/release-notes.txt

Please try it out, feedback is welcome.



amavisd-new-2.11.0 release notes

Contents:
   DEPRECATION NOTICE
   COMPATIBILITY
   BUG FIXES
   NEW FEATURES
   OTHER
   SUPERVISED PROCESS NOTES


DEPRECATION NOTICE

- the old DomainKey signatures (a predecessor to DKIM) has been 
published
   as a historic document RFC 4870 and obsoleted by RFC 4871 in May 2007;
   Support for DomainKey signatures is likely to be removed with a next
   version of amavisd.


COMPATIBILITY

There are some minor incompatibilities between versions 2.10.1 and 
2.11.0:

- During startup more detailed testing is performed for taint bugs of
   a module Encode and the function utf8::is_utf8(), which may produce
   warnings on old versions of perl with its old core module Encode,
   or may exit on detecting more sinister bugs in these modules.
   Note that the module Encode may be upgraded independently of perl,
   if desired;

- with MySQL: changed character set 'utf8' to 'utf8mb4' for fields
   msgs.subject and msgs.from_addr, as previously some of the UTF-8
   characters could not be stored in a database;

- when logging to stderr a timestamp prefix to each message is only
   still inserted if $DEBUG is true.  When $DEBUG is false each message
   is prefixed with a syslog log level in angle brackets, and a timestamp
   is omitted (for compatibility with systemd);

- a perl module Digest::SHA is now a required module. It is a perl core
   module since perl 5.10, so it shouldn't introduce a new dependency,
   and it was a de-facto required module even previously, as it was 
needed
   for DKIM processing;


BUG FIXES

- delivery method was undefined when always_bcc was used;
   reported by Marieke Janssen;

- amavisd: avoid warnings issued by perl 5.21.7 and later:
     Negative repeat count does nothing at ./amavisd line 16408
   and similarly in amavisd-status;

- remove a stale database file __db.nanny.db on a reload or restart,
   as it can prevent a successful start when a previous start failed
   for some reason; a patch by Trent Lloyd;


NEW FEATURES

- Polished rough corners to facilitate running amavisd as a 
non-daemonized
   supervised process, e.g. under systemd:

   * make it possible/easier to disable use of a pid_file;

   * send status notifications to systemd when a NOTIFY_SOCKET 
environment
     variable is provided;

   * improved logging to stderr when $do_syslog and $logfile are 
undefined
     (although logging through syslog might still be preferred, as 
writing
     to a shared pipe from multiple child processes only guarantees 
atomicity
     of writes shorter than PIPE_BUF, which is typically 512 bytes on 
*BSD,
     and 4096 bytes on Linux);

   See below for a sample amavisd.service file.

- A log template macro 'report_json' can now take arguments, which can
   include or exclude fields (key/values) from the JSON report object.
   Arguments to a macro are either field names (keys) to be included
   in a report, or are field names to be excluded, each prefixed with
   an exclamation mark, to produce a report with all but excluded fields.

   Field names are case-sensitive. The order of fields in a serialized
   JSON object is unaffected by the order of field names in a filter.
   Unknown or non-present field names in a filter are silently ignored.

   Example:
     [:report_json|mail_id|action|content_type|queued_as|mail_from|size]
   or:
     [:report_json|!recipients|!elapsed|!os_fp|!subject|!subject_rot13]

   For better clarity, instead of listing field names as individual
   arguments to a macro, it is also possible to provide a single argument
   to a macro, in which field names are separated by whitespace:
     [:report_json|mail_id action content_type queued_as mail_from size]
   or:
     [:report_json|!message !recipients !to_addr   !elapsed !os_fp
        !subject !subject_rot13 !user_agent !tests !tests_ham 
!tests_spam]

   As an example, a setting in a config file may look like:
     $log_templ = '[:report_json|mail_id action queued_as mail_from]';

   If at least one field name has an exclamation mark (i.e. is to be
   excluded), all but excluded fields are implied, so any field names
   without an exclamation mark are redundant.

   Currently this is a simple filter where subfields of a structured
   object cannot be selectively filtered (e.g. elapsed.SpamCheck).
   For finer control on JSON content use some external JSON-processing
   utility.  Based on a patch by Markus Benning.

- Two new configuration settings are added: %smtpd_tls_server_options
   and %smtp_tls_client_options. These two associative arrays are passed
   to IO::Socket::SSL->start_SSL when establishing a server-side or a
   client-side TLS session with an MTA, and provide more control over
   a TLS session - like providing certificates and restricting ciphers.
   See documentation of a perl module IO::Socket::SSL for a list of
   all options with their descriptions and their defaults.

   When TLS is in use, it is recommended to stick to fresh versions
   of the module module IO::Socket::SSL and the underlying ssl library,
   as it can provide a safer set of defaults (e.g. excluded SSLv2).

   Existing config options $smtpd_tls_cert_file and $smtpd_tls_key_file
   are now deprecated in favour of a more generic 
%smtpd_tls_server_options.
   Preferably set fields 'SSL_key_file' and 'SSL_cert_file' directly in
   %smtpd_tls_server_options instead. For compatibility with 2.10 the
   values of $smtpd_tls_cert_file and $smtpd_tls_key_file are fed into
   the associative array %smtpd_tls_server_options if fields 
'SSL_key_file'
   and 'SSL_cert_file' are not provided (do not exist) there.

   Example:

   %smtp_tls_client_options = (
     SSL_verifycn_scheme => 'smtp',
     SSL_version => '!SSLv2,!SSLv3',
     SSL_cipher_list => 'HIGH:!MD5:!DSS:!aNULL',
#   SSL_client_ca_file => ... ,
   );

   %smtpd_tls_server_options = (
     SSL_verifycn_scheme => 'smtp',
     SSL_session_cache => 2,
     SSL_key_file  => "$MYHOME/cert/amavisd-key.pem",
     SSL_cert_file => "$MYHOME/cert/amavisd-cert.pem",
     SSL_dh_file   => "$MYHOME/cert/amavisd-dh.dat",
   # SSL_ca_file   => ... ,
     SSL_version   => '!SSLv2,!SSLv3',
     SSL_cipher_list => 'HIGH:!MD5:!DSS:!aNULL',
   );

   Or just to change some field and leave the rest at their default:

     $smtp_tls_client_options{SSL_verify_mode} = 0;  # SSL_VERIFY_NONE

   Suggested by Marc Grooz and Patrick Ben Koetter, based on a patch
   by Markus Benning.

- Supports receiving SMTP/LMTP connections through a HAProxy,
   recognizing 'PROXY protocol Version 1' data on the first line read,
   after a connection from HAProxy to amavisd has been established.
   Connection data (IP addresses and ports) received via this protocol
   end up replacing such data in the the Amavis::In::Connection object
   ($conn).  Set configuration variable $haproxy_target_enabled (also
   a member of policy banks) to true in order to enable this protocol.

- redis: allow a scoped / link-local IP address specification
   (avoiding current limitation in IO::Socket::IP [rt.cpan.org #89608]);

- the Amavis::Unpackers::Part::digest method now holds a digest (SHA1,
   hex) of a decoded (base64 or quoted-printable) MIME part contents,
   followed by a colon and a lowercased Content-Type of the MIME part.
   Canonical line endings CRLF in decoded textual parts are normalized
   to a native newline (\n) before feeding them to a digest algorithm.

   These digests are passed to SpamAssassin through a 'mimepart_digests'
   supplementary attribute, and are available to custom hooks. As of
   version SpamAssassin 3.4.1, these are used as additional tokens in
   a Bayes plugin. Even though SpamAssassin is capable of computing
   the same or similar digests on its own, the advantage of computing
   them in amavisd is that they reflect all and completely unmodified
   and untruncated MIME parts of a mail message, including non-textual
   attachments.

   For debugging, search the log for "mimepart digest: ", logged at
   log level 5, and ".* Content-Type: .*, size:" at log level 2.
   Based on a suggestion by Andreas Schulze back in 2014.

   A configuration setting $mail_part_digest_algorithm was added, which
   chooses an algorithm name for generating digests of decoded MIME
   parts of a message. The value is an algorithm name as accepted by
   Digest::SHA->new(), e.g. 'sha1' or 'SHA-1' or 'SHA-256' or 'sha256',
   or a string 'MD5' (case-insensitive) which chooses the MD5 algorithm
   as implemented by a module Digest::MD5. An undefined value disables
   generating digests of MIME parts. The $mail_part_digest_algorithm
   setting is a dynamic setting, i.e. it is a member of policy banks.

   For compatibility with SpamAssassin the chosen algorithm should be
   SHA1 (which is a default), otherwise bayes tokens won't match those
   generated by sa-learn (which is typically used for off-line learning).
   Bayes auto-learning in SpamAssassin is unaffected by a mismatch of
   the algorithm, as it believes digests received from amavisd.

- Policy bank names in a @client_ipaddr_policy setting can now accept
   a comma-separated list of policy names to be loaded on a match
   (for loading of policy banks based on an IP address of a SMTP client).
   Whitespace around each policy name is allowed and is stripped.
   Previously only a single policy bank name was allowed in each entry
   of @client_ipaddr_policy.

   This makes it consistent with loading of policy banks based on a
   DKIM-based setting @author_to_policy_bank_maps, and on virus checker
   results via the @virus_name_to_policy_bank_maps setting.


- Experimental feature: IP lookups (as implemented by lookup_ip_acl()
   and used by @client_ipaddr_policy) can now also do DNS-based lookups,
   in addition to array- and hash- based lookups.
   Suggested by Patrick Ben Koetter.

   DNS lookups follow RFC 5782 conventions (DNS Blacklists and 
Whitelists:
   DNSBL, DNSWL, collectively known as DNSxL).  A DNS query of a type 'A'
   is performed on a reversed IP address prepended to a specified domain
   name (zone name).  RFC 5782 suggests that only type-A resource records
   of a DNS reply in an address range 127.0.0.0/8 may be considered.

   For example, given a zone name 'rbl.example.org' and a SMTP client's
   IP address 198.51.100.12, a DNS type-A query for a domain name
   "12.100.51.198.rbl.example.org" would be sent to a specified or to a
   default DNS resolver or server. Similarly, an IP address 2001:db8::2:f
   would produce a DNS type-A query for a domain name "f.0.0.0.2.0.0.0.0.
   0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.rbl.example.org" .

   The setting @client_ipaddr_policy contains a list of pairs, each pair
   consisting of a lookup object (arrayref or hashref, or now also an
   Amavis::Lookup::DNSxL object), followed by a policy bank name (which
   is a string: one or more policy bank names, comma-separated).

   The object constructor Amavis::Lookup::DNSxL->new accepts as its
   arguments: a dns zone name, expected result(s) for a match, and a
   resolver object. Only the first argument (a DNSxL zone name) is
   required, the remaining two arguments are optional. A default
   expected result is '127.0.0.2', and a default Net::DNS::Resolver
   persistent object is provided implicitly if not provided by a caller
   (it reads a DNS resolver's IP address from /etc/resolv.conf).

   The "expected result(s) for a match" argument (the second argument)
   is compared to the address found in a DNS reply (in a 127.0.0.0/8 
range).
   It can be:
   a) an integer between 0 and 255 (or a string representing such
     integer), which is used as a last byte on the 127.0.0.x quad;
   b) a string in a dotted-quad form of an IPv4 address in a 127.0.0.0/8
     range, where leading bytes may be omitted (e.g. '1.8' == 
'127.0.1.8');
   c) a reference to an array consisting of entries in an (a) or (b) 
form,
      where a match with any of the array elements suffices for a match;
   d) a perl regular expression object (e.g.  qr{^127\.[3-8]\.0\.\d*$} ).

   If the address in a DNS reply matches the provided "expected result"
   argument, the policy banks associated with that entry are loaded,
   and a search through a @client_ipaddr_policy list stops.

   As a shorthand a subroutine Amavis::Conf::q_dns_a() is provided,
   which is just a convenient wrapper for Amavis::Lookup::DNSxL->new().

   Example:

   @client_ipaddr_policy = (
     [qw( 0.0.0.0/8 127.0.0.0/8 [::] [::1] )] => 'MYNETS, LOCALHOST',
     [qw( 169.254.0.0/16 [fe80::]/10 )]       => 'MYNETS, LINKLOCAL',
     [qw( 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 )] => 'MYNETS, 
PRIVATENET',
     \@mynetworks => 'MYNETS',
     q_dns_a('rbl.example.org')           => 'MY-CUSTOMER-A',    # 
127.0.0.2
     q_dns_a('rbl.example.org', 3) => 'MYNETS,MY-CUSTOMER-B',    # 
127.0.0.3
     q_dns_a('rbl.example.org', '0.2.99') => 'MY-CUSTOMER-C',    # 
127.0.2.99
     q_dns_a('rbl.example.org', '127.0.0.7') => 'MY-CUSTOMER-D', # 
127.0.0.7
     q_dns_a('rbl.example.org', qr/^127\.1\.\d+\.2\d*\z/) =>'X', # 
127.1.*.2*
     q_dns_a('rbl.example.org', '192.0.2.0.2')=>'never matches', # not in 
127/8
   );

   Below is an example of an amavisd.conf section with an explicitly
   provided Net::DNS::Resolver object, which offers finer control over
   its settings:

   use Net::DNS;
   my $dnsxl_res = Net::DNS::Resolver->new(
     config_file => '/etc/resolv.conf',
     port => 5333, retry => 1, persistent_udp => 1,
     tcp_timeout => 2, udp_timeout => 2, retrans => 1,
   );
   $dnsxl_res or die "Module Net::DNS not available for DNSxL usage";
   $dnsxl_res->udppacketsize(1220);

   my $myrbl = 'rbl.example.org';

   @client_ipaddr_policy = (
     \@mynetworks => 'MYNETS',
     q_dns_a($myrbl, 2,       $dnsxl_res) => 'MY-CUSTOMER-A',  # 
127.0.0.2
     q_dns_a($myrbl, [3,4,5], $dnsxl_res) => 'MY-CUSTOMER-B',  # 
127.0.0.{3,4,5}
   );

   This DNS-lookup feature is considered experimental in a sense that
   its API may change in future versions. As it is currently implemented,
   each q_dns_a() entry in a @client_ipaddr_policy results in its own
   DNS query, which is quite inefficient with more that one or two such
   entries. It would make more sense to do a single DNS lookup and 
provide
   some mapping between results returned and policy bank names to be
   loaded. Note also that DNS lookups are performed synchronously and
   sequentially (one at a time, one after another), so a slowly 
responding
   DNS server combined with multisecond timeouts and retries could 
severely
   bog down the amavisd response time, easily to exceed the time a MTA or
   a SMTP client is willing to wait for a response. YOU HAVE BEEN WARNED!


OTHER

- Relax a check on a PID number found in a pid file, considering
   that amavisd may run as PID #1 under Docker; reported by Imre Rad.

- Relax a check on $pid_file being configured or provided by a command
   line option -P.  Amavisd can now run without checking or providing a
   PID file of a running master process, which is appropriate for running
   non-daemonized amavisd as a supervised process (e.g. under supervision
   suites such as systemd, s6, nosh, runit, launchd or similar).  Also,
   specifying a command line option -P '' (i.e. giving it an empty name
   of a pid_file) overrides a configuration option $pid_file and is a
   quick way to disable usage of a pid_file.

   A default value of $pid_file is now only provided if a global
   setting $daemonize is true (which is a default, unless running
   with 'foreground' or 'debug' command line options).
   A non-daemonized amavisd leaves $pid_file undefined as a default,
   which facilitates running amavisd as a supervised process, e.g.
     $daemonize = $pid_file = $daemon_user = undef;

   When a pid_file is disabled and running under systemd, amavisd obtains
   a PID of a master process from systemd through environment variable
   MAINPID, which allows operations like 'amavisd reload' and 'amavisd 
stop'
   from a .service file  (ExecReload and ExecStop in systemd.exec(5)).

   Btw, a command line option 'foreground' is a quick way to override
   a configuration setting $daemonize - it sets its value to 0.

   To let amavisd provide and use a PID file even when not daemonized,
   configure a PID file explicitly, e.g.: $pid_file = 
"$MYHOME/amavisd.pid";

- provide sensible diagnostics when $daemon_user is undefined and
   starting as root;

- 'sanitize_nul' function is now enabled by default (this is currently
   not configurable). Null octets found in a message are replaced by a
   pair of octets \xC0 \x80, which is a "Modified UTF-8" encoding of a
   NUL. This is done to avoid a mailbox server (like Cyrus) or a mail
   client on choking on such mail. The downside is that such sanitation
   can invalidate a DKIM signature - but non-encoded NUL octets are not
   allowed in mail anyway, so not much harm is done;

- overhauled a client side of the ClamAV clamd protocol;

- updated decoder for 7z archives to improve handling of encrypted
   content; based on a patch by Markus Benning;

- adjusted log levels of some log/debug messages;

- reject a message with an 8BITMIME body type if a back-end MTA does
   not announce 8bit-MIMEtransport capability in its EHLO response
     ( 550 5.6.3 Conversion to 7BIT required but not supported );

- replaced calls to Encode::is_utf8() by utf8::is_utf8() - less buggy
   in old versions of perl, but requires perl 5.8.1 or later;

- replaced calls to Encode::encode_utf8() by utf8::encode() - is
   much faster, and is less buggy in old versions of perl;

- more detailed testing for taint bugs of a module Encode and in
   utf8::is_utf8() during startup;

- decode a supposedly (or guessably) character set ISO-8859-1 as
   Windows-1252, which is a proper superset of ISO-8859-1 and often
   mistaken for ISO-8859-1; (this follows advice of HTML5)

- with MySQL: changed character set 'utf8' to 'utf8mb4' for fields
   msgs.subject and msgs.from_addr;

- in case the Net::Server receives a connection over a Unix socket
   (e.g. from amavisd-release) but is unable to determine a socket name,
   supply a dummy socket name 'UNKNOWN' so that a policy bank 'SOCK'
   can still be loaded;

- the setting $mail_digest_algorithm is now a dynamic setting, i.e.
   can be configured per policy bank. The change makes it consistent
   with a new setting $mail_part_digest_algorithm, which is also dynamic.

- updated the @av_scanners Avast entry ( http://www.avast.com/ )
   in the sample config file amavisd.conf to a new version
   of their scanner:
     ['avast! Antivirus', '/bin/scan', '{}', [0], [1], qr/\t(.+)/m]
   Thanks to Martin Tůma from Avast for the new entry;

- updated a default $map_full_type_to_short_type_re to distinguish
   encrypted PGP/GnuPG files from other PGP/GnuPG containers like
   a detached signature, exported public key files, etc., if a
   newer version of a file(1) utility is in use (5.20?);

- updated a default $map_full_type_to_short_type_re to recognize
   a Microsoft Word document as type doc; thanks to Jörg Backschues;

- added PhishTank.Phishing to a default @virus_name_to_spam_score_maps;

- reworded some notification texts;


SUPERVISED PROCESS NOTES

Socket activation (running under a superserver with fd-holding) is
currently not available. Note that 'amavisd reload' does fd-holding
and socket passing to a new incarnation of amavisd server on its own,
which means that a client (an MTA) does not see a disruption on a
reload (= warm restart), unlike in case of restarting amavisd.

As a reminder: a full restart is only necessary when changing the set
of listening sockets in a configuration file. For all other needs
(like changing other settings, updating SpamAssassin rules, upgrading
the amavisd program or perl modules) a reload suffices.

Here is a sample file amavisd.service for use under systemd
(indented for clarity).  Lightly tested on Debian 8.0 (Jessie)
on a Raspberry Pi 2.

/lib/systemd/system/amavisd.service

   [Unit]
   Description=amavisd-new mail filter
   Before=shutdown.target
   After=systemd-journald-dev-log.socket network-online.target 
local-fs.target
   Wants=network-online.target
   Conflicts=shutdown.target

   [Service]
   Type=notify
   NotifyAccess=main
   KillMode=mixed
   TimeoutStartSec=1min
   TimeoutStopSec=3min
   User=amavis
   Group=amavis
   WorkingDirectory=/var/lib/amavis/tmp
   StandardOutput=syslog
   SyslogFacility=mail
   SyslogIdentifier=amavis
   ProtectSystem=full
   ProtectHome=yes
   NoNewPrivileges=yes
   ExecStart  = /usr/sbin/amavisd-new -P '' foreground
   ExecReload = /usr/sbin/amavisd-new -P '' reload
   ExecStop   = /usr/sbin/amavisd-new -P '' stop

   [Install]
   WantedBy = multi-user.target


Consider the following amavisd.conf settings when running as a
supervised process:

   $pid_file = '';  # can be overridden by a command line option -P ''
   $daemonize = 0;  # also implied by a command line argument 
'foreground'
   $do_syslog = 0;  # or set it to 1 to log to syslog instead of stderr




Mark