ANNOUNCE: amavisd-new-2.11.0 released
Mark Martinec
Mark.Martinec+amavis at ijs.si
Tue Apr 26 21:46:04 CEST 2016
A release 2.11.0 of amavisd-new now is available at:
https://www.ijs.si/software/amavisd/amavisd-new-2.11.0.tar.xz
Release notes are at:
https://www.ijs.si/software/amavisd/release-notes.txt
amavisd-new-2.11.0 release notes
Contents:
DEPRECATION NOTICE
COMPATIBILITY
BUG FIXES
NEW FEATURES
OTHER
SUPERVISED PROCESS NOTES
DEPRECATION NOTICE
- The old DomainKey signatures (a predecessor to DKIM) has been
published
as a historic document RFC 4870 and obsoleted by RFC 4871 in May 2007;
Support for DomainKey signatures is likely to be removed with a next
version of amavisd.
- It is expected that the next release of amavisd will start using some
of the features made available with perl 5.10.0 (such as a defined-or
operator, or a possessive quantifier in regular expressions), so
consider running amavisd under perl 5.8.9 or earlier as unsupported.
In practice, using such old version of perl is problematic even now,
as their support for Unicode / UTF-8 is incomplete and unreliable.
COMPATIBILITY
There are some minor incompatibilities between versions 2.10.1 and
2.11.0:
- During startup more detailed testing is performed for taint bugs of
a module Encode and the function utf8::is_utf8(), which may produce
warnings on old versions of perl with its old core module Encode,
or may exit on detecting more sinister bugs in these modules.
Note that the module Encode may be upgraded independently of perl,
if desired;
- with MySQL: changed character set 'utf8' to 'utf8mb4' for fields
msgs.subject and msgs.from_addr, as previously some of the UTF-8
characters could not be stored in a database;
- when logging to stderr a timestamp prefix to each message is only
still inserted if $DEBUG is true. When $DEBUG is false each message
is prefixed with a syslog log level in angle brackets, and a timestamp
is omitted (for compatibility with systemd);
- a perl module Digest::SHA is now a required module. It is a perl core
module since perl 5.10, so it shouldn't introduce a new dependency,
and it was a de-facto required module even previously, as it was
needed
for DKIM processing;
BUG FIXES
- delivery method was undefined when always_bcc was used;
reported by Marieke Janssen;
- avoid warnings issued by perl 5.21.7 and later:
Negative repeat count does nothing at ./amavisd line 16408
and similarly in amavisd-status;
- releasing from an SQL quarantine failed to provide the original
envelope sender address to a released message;
reported, and a fix suggested by Tom Johnson and Tobias;
- remove a stale database file __db.nanny.db on a reload or restart,
as it can prevent a successful start when a previous start failed
for some reason; a patch by Trent Lloyd;
NEW FEATURES
- Polished rough corners to facilitate running amavisd as a
non-daemonized
supervised process, e.g. under systemd:
* make it possible/easier to disable use of a pid_file;
* send status notifications to systemd when a NOTIFY_SOCKET
environment
variable is provided;
* improved logging to stderr when $do_syslog and $logfile are
undefined
(although logging through syslog might still be preferred, as
writing
to a shared pipe from multiple child processes only guarantees
atomicity
of writes shorter than PIPE_BUF, which is typically 512 bytes on
*BSD,
and 4096 bytes on Linux systems);
See below for a sample amavisd.service file.
- A log template macro 'report_json' can now take arguments, which can
include or exclude fields (key/values) from the JSON report object.
Arguments to a macro are either field names (keys) to be included
in a report, or are field names to be excluded, each prefixed with
an exclamation mark, to produce a report with all but excluded fields.
Field names are case-sensitive. The order of fields in a serialized
JSON object is unaffected by the order of field names in a filter.
Unknown or non-present field names in a filter are silently ignored.
Example:
[:report_json|mail_id|action|content_type|queued_as|mail_from|size]
or:
[:report_json|!recipients|!elapsed|!os_fp|!subject|!subject_rot13]
For better clarity, instead of listing field names as individual
arguments to a macro, it is also possible to provide a single argument
to a macro, in which field names are separated by whitespace:
[:report_json|mail_id action content_type queued_as mail_from size]
or:
[:report_json| !message !recipients !to_addr !elapsed !os_fp
!subject !subject_rot13 !user_agent !tests !tests_ham
!tests_spam]
As an example, a setting in a config file may look like:
$log_templ = '[:report_json|mail_id action queued_as mail_from]';
If at least one field name has an exclamation mark (i.e. is to be
excluded), all but excluded fields are implied, so any field names
without an exclamation mark are redundant.
Currently this is a simple filter where subfields of a structured
object cannot be selectively filtered (e.g. elapsed.SpamCheck).
For finer control on JSON content use some external JSON-processing
utility. Based on a patch by Markus Benning.
- Two new configuration settings are added: %smtpd_tls_server_options
and %smtp_tls_client_options. These two associative arrays are passed
to IO::Socket::SSL->start_SSL when establishing a server-side or a
client-side TLS session with an MTA, and provide more control over
a TLS session - like providing certificates and restricting ciphers.
See documentation of a perl module IO::Socket::SSL for a list of
all options with their descriptions and their defaults.
When TLS is in use, it is recommended to stick to fresh versions
of the module IO::Socket::SSL and the underlying ssl library,
as it can provide a safer set of defaults (e.g. excluded SSLv2).
Existing config options $smtpd_tls_cert_file and $smtpd_tls_key_file
are now deprecated in favour of a more generic
%smtpd_tls_server_options.
Preferably set fields 'SSL_key_file' and 'SSL_cert_file' directly in
%smtpd_tls_server_options instead. For compatibility with 2.10 the
values of $smtpd_tls_cert_file and $smtpd_tls_key_file are fed into
the associative array %smtpd_tls_server_options if fields
'SSL_key_file'
and 'SSL_cert_file' are not provided (do not exist) there.
Example:
%smtp_tls_client_options = (
SSL_verifycn_scheme => 'smtp',
SSL_version => '!SSLv2,!SSLv3',
SSL_cipher_list => 'HIGH:!MD5:!DSS:!aNULL',
# SSL_client_ca_file => ... ,
);
%smtpd_tls_server_options = (
SSL_verifycn_scheme => 'smtp',
SSL_session_cache => 2,
SSL_key_file => "$MYHOME/cert/amavisd-key.pem",
SSL_cert_file => "$MYHOME/cert/amavisd-cert.pem",
SSL_dh_file => "$MYHOME/cert/amavisd-dh.dat",
# SSL_ca_file => ... ,
SSL_version => '!SSLv2,!SSLv3',
SSL_cipher_list => 'HIGH:!MD5:!DSS:!aNULL',
);
Or just to change some field and leave the rest at their default:
$smtp_tls_client_options{SSL_verify_mode} = 0; # SSL_VERIFY_NONE
Suggested by Marc Grooz and Patrick Ben Koetter, based on a patch
by Markus Benning.
- Supports receiving SMTP/LMTP connections through a HAProxy,
recognizing 'PROXY protocol Version 1' data on the first line read,
after a connection from HAProxy to amavisd has been established.
Connection data (IP addresses and ports) received via this protocol
end up replacing such data in the the Amavis::In::Connection object
($conn). Set configuration variable $haproxy_target_enabled (also
a member of policy banks) to true in order to enable this protocol.
- redis: allow a scoped / link-local IP address specification
(avoiding current limitation in IO::Socket::IP [rt.cpan.org #89608]);
- the Amavis::Unpackers::Part::digest method now holds a digest (SHA1,
hex) of a decoded (base64 or quoted-printable) MIME part contents,
followed by a colon and a lowercased Content-Type of the MIME part.
Canonical line endings CRLF in decoded textual parts are normalized
to a native newline (\n) before feeding them to a digest algorithm.
These digests are passed to SpamAssassin through a 'mimepart_digests'
supplementary attribute, and are available to custom hooks. As of
version SpamAssassin 3.4.1, these are used as additional tokens in
a Bayes plugin. Even though SpamAssassin is capable of computing
the same or similar digests on its own, the advantage of computing
them in amavisd is that they reflect all and completely unmodified
and untruncated MIME parts of a mail message, including non-textual
attachments.
For debugging, search the log for "mimepart digest: ", logged at
log level 5, and ".* Content-Type: .*, size:" at log level 2.
Based on a suggestion by Andreas Schulze back in 2014.
A configuration setting $mail_part_digest_algorithm was added, which
chooses an algorithm name for generating digests of decoded MIME
parts of a message. The value is an algorithm name as accepted by
Digest::SHA->new(), e.g. 'sha1' or 'SHA-1' or 'SHA-256' or 'sha256',
or a string 'MD5' (case-insensitive) which chooses the MD5 algorithm
as implemented by a module Digest::MD5. An undefined value disables
generating digests of MIME parts. The $mail_part_digest_algorithm
setting is a dynamic setting, i.e. it is a member of policy banks.
For compatibility with SpamAssassin the chosen algorithm should be
SHA1 (which is a default), otherwise bayes tokens won't match those
generated by sa-learn (which is typically used for off-line learning).
Bayes auto-learning in SpamAssassin is unaffected by a mismatch of
the algorithm, as it believes digests received from amavisd.
- Policy bank names in a @client_ipaddr_policy setting can now accept
a comma-separated list of policy names to be loaded on a match
(for loading of policy banks based on an IP address of a SMTP client).
Whitespace around each policy name is allowed and is stripped.
Previously only a single policy bank name was allowed in each entry
of @client_ipaddr_policy.
This makes it consistent with loading of policy banks based on a
DKIM-based setting @author_to_policy_bank_maps, and on virus checker
results via the @virus_name_to_policy_bank_maps setting.
- Experimental feature: IP lookups (as implemented by lookup_ip_acl()
and used by @client_ipaddr_policy) can now also do DNS-based lookups,
in addition to array- and hash- based lookups.
Suggested by Patrick Ben Koetter and loosely based on his patch.
DNS lookups follow RFC 5782 conventions (DNS Blacklists and
Whitelists:
DNSBL, DNSWL, collectively known as DNSxL). A DNS query of a type 'A'
is performed on a reversed IP address prepended to a specified domain
name (zone name). RFC 5782 suggests that only type-A resource records
of a DNS reply in an address range 127.0.0.0/8 may be considered.
For example, given a zone name 'rbl.example.org' and a SMTP client's
IP address 198.51.100.12, a DNS type-A query for a domain name
"12.100.51.198.rbl.example.org" would be sent to a specified or to a
default DNS resolver or server. Similarly, an IP address 2001:db8::2:f
would produce a DNS type-A query for a domain name "f.0.0.0.2.0.0.0.0.
0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.rbl.example.org" .
The setting @client_ipaddr_policy contains a list of pairs, each pair
consisting of a lookup object (arrayref or hashref, or now also an
Amavis::Lookup::DNSxL object), followed by a policy bank name (which
is a string: one or more policy bank names, comma-separated).
The object constructor Amavis::Lookup::DNSxL->new accepts as its
arguments: a dns zone name, expected result(s) for a match, and a
resolver object. Only the first argument (a DNSxL zone name) is
required, the remaining two arguments are optional. A default
expected result is '127.0.0.2', and a default Net::DNS::Resolver
persistent object is provided implicitly if not provided by a caller
(it reads a DNS resolver's IP address from /etc/resolv.conf).
The "expected result(s) for a match" argument (the second argument)
is compared to the address found in a DNS reply (in a 127.0.0.0/8
range).
It can be:
a) an integer between 0 and 255 (or a string representing such
integer), which is used to match the last byte on the 127.0.0.x
quad;
b) a string in a dotted-quad form of an IPv4 address in a 127.0.0.0/8
range, where leading bytes may be omitted (e.g. '1.8' ==
'127.0.1.8');
c) a reference to an array consisting of entries in an (a) or (b)
form,
where a match with any of the array elements suffices for a match;
d) a perl regular expression object (e.g. qr{^127\.[3-8]\.0\.\d*$} ).
If an IP address in a DNS reply matches the provided "expected result"
argument, the policy banks associated with that entry are loaded,
and a search through a @client_ipaddr_policy list stops.
As a shorthand a subroutine Amavis::Conf::q_dns_a() is provided,
which is just a convenient wrapper for Amavis::Lookup::DNSxL->new().
Example:
@client_ipaddr_policy = (
[qw( 0.0.0.0/8 127.0.0.0/8 [::] [::1] )] => 'MYNETS, LOCALHOST',
[qw( 169.254.0.0/16 [fe80::]/10 )] => 'MYNETS, LINKLOCAL',
[qw( 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 )] => 'MYNETS,
PRIVATENET',
\@mynetworks => 'MYNETS',
q_dns_a('rbl.example.org') => 'MY-CUSTOMER-A', #
127.0.0.2
q_dns_a('rbl.example.org', 3) => 'MYNETS,MY-CUSTOMER-B', #
127.0.0.3
q_dns_a('rbl.example.org', '0.2.99') => 'MY-CUSTOMER-C', #
127.0.2.99
q_dns_a('rbl.example.org', '127.0.0.7') => 'MY-CUSTOMER-D', #
127.0.0.7
q_dns_a('rbl.example.org', qr/^127\.1\.\d+\.2\d*\z/) =>'X', #
127.1.*.2*
q_dns_a('rbl.example.org', '192.0.2.0.2')=>'never matches', # not in
127/8
);
Below is an example of an amavisd.conf section with an explicitly
provided Net::DNS::Resolver object, which offers finer control over
its settings:
use Net::DNS;
my $dnsxl_res = Net::DNS::Resolver->new(
config_file => '/etc/resolv.conf',
port => 5333, retry => 1, persistent_udp => 1,
tcp_timeout => 2, udp_timeout => 2, retrans => 1,
);
$dnsxl_res or die "Module Net::DNS not available for DNSxL usage";
$dnsxl_res->udppacketsize(1220);
my $myrbl = 'rbl.example.org';
@client_ipaddr_policy = (
\@mynetworks => 'MYNETS',
q_dns_a($myrbl, 2, $dnsxl_res) => 'MY-CUSTOMER-A', #
127.0.0.2
q_dns_a($myrbl, [3,4,5], $dnsxl_res) => 'MY-CUSTOMER-B', #
127.0.0.{3,4,5}
);
This DNS-lookup feature is considered experimental in a sense that
its API may change in future versions. As it is currently implemented,
each q_dns_a() entry in a @client_ipaddr_policy results in its own
DNS query, which is quite inefficient with more that one or two such
entries. It would make more sense to do a single DNS lookup and
provide
some mapping between results returned and policy bank names to be
loaded. Note also that DNS lookups are performed synchronously and
sequentially (one at a time, one after another), so a slowly
responding
DNS server combined with multisecond timeouts and retries could
severely
bog down the amavisd response time, easily to exceed the time a MTA or
a SMTP client is willing to wait for a response. YOU HAVE BEEN WARNED!
OTHER
- Relax a check on a PID number found in a pid file, considering
that amavisd may run as PID #1 under Docker; reported by Imre Rad.
- Relax a check on $pid_file being configured or provided by a command
line option -P. Amavisd can now run without checking or providing a
PID file of a running master process, which is appropriate for running
non-daemonized amavisd as a supervised process (e.g. under supervision
suites such as systemd, s6, nosh, runit, launchd or similar). Also,
specifying a command line option -P '' (i.e. giving it an empty name
of a pid_file) overrides a configuration option $pid_file and is a
quick way to disable usage of a pid_file.
A default value of $pid_file is now only provided if a global
setting $daemonize is true (which is a default, unless running
with 'foreground' or 'debug' command line options).
A non-daemonized amavisd leaves $pid_file undefined as a default,
which facilitates running amavisd as a supervised process, e.g.
$daemonize = $pid_file = $daemon_user = undef;
When a pid_file is disabled and running under systemd, amavisd obtains
a PID of a master process from systemd through environment variable
MAINPID, which allows operations like 'amavisd reload' and 'amavisd
stop'
from a .service file (ExecReload and ExecStop in systemd.exec(5)).
Btw, a command line argument 'foreground' is a quick way to override
a configuration setting $daemonize - it sets its value to 0.
To let amavisd provide and use a PID file even when not daemonized,
configure a PID file explicitly, e.g.: $pid_file =
"$MYHOME/amavisd.pid";
- provide sensible diagnostics when $daemon_user is undefined and
starting as root;
- 'sanitize_nul' function is now enabled by default (this is currently
not configurable). Null octets found in a message are replaced by a
pair of octets \xC0 \x80, which is a "Modified UTF-8" encoding of a
NUL. This is done to avoid a mailbox server (like Cyrus) or a mail
client on choking on such mail. The downside is that such sanitation
can invalidate a DKIM signature - but non-encoded NUL octets are not
allowed in mail anyway, so not much harm is done;
- overhauled a client side of the ClamAV clamd protocol;
- updated decoder for 7z archives to improve handling of encrypted
content; based on a patch by Markus Benning;
- recognize and handle completely encrypted zip archives by 7z
(in do_7zip); a patch provided by Thomas Jarosch;
- adjusted log levels of some log/debug messages;
- reject a message with an 8BITMIME body type if a back-end MTA does
not announce 8bit-MIMEtransport capability in its EHLO response
( 550 5.6.3 Conversion to 7BIT required but not supported );
- replaced calls to Encode::is_utf8() by utf8::is_utf8() - less buggy
in old versions of perl, but requires perl 5.8.1 or later;
- replaced calls to Encode::encode_utf8() by utf8::encode() - is
much faster, and is less buggy in old versions of perl;
- more detailed testing for taint bugs of a module Encode and in
utf8::is_utf8() during startup;
- decode a supposedly (or guessably) character set ISO-8859-1 as
Windows-1252, which is a proper superset of ISO-8859-1 and often
mistaken for ISO-8859-1; (this follows advice of HTML5);
- with MySQL: changed character set 'utf8' to 'utf8mb4' for fields
msgs.subject and msgs.from_addr;
- in case the Net::Server receives a connection over a Unix socket
(e.g. from amavisd-release) but is unable to determine a socket name,
supply a dummy socket name 'UNKNOWN' so that a policy bank 'SOCK'
can still be loaded;
- the setting $mail_digest_algorithm is now a dynamic setting, i.e.
can be configured per policy bank. The change makes it consistent
with a new setting $mail_part_digest_algorithm, which is also dynamic;
- updated the @av_scanners Avast entry ( http://www.avast.com/ )
in the sample config file amavisd.conf to a new version
of their scanner:
['avast! Antivirus', '/bin/scan', '{}', [0], [1], qr/\t(.+)/m]
Thanks to Martin Tůma from Avast for the new entry;
- updated a default @$map_full_type_to_short_type_re to distinguish
encrypted PGP/GnuPG files from other PGP/GnuPG containers like
a detached signature, exported public key files, etc., if a
newer version of a file(1) utility is in use (5.20?);
- relaxed the /\bscript\b.* text executable\b/ regexp entry in the
default @$map_full_type_to_short_type_re list so that a mail part
such as qualified by a file(1) utility as:
Python script, Non-ISO extended-ASCII text executable
does not qualify as an executable; reported by Tilman Schmidt;
- updated a default @$map_full_type_to_short_type_re to recognize
a Microsoft Word document as type doc; thanks to Jörg Backschues;
- added PhishTank.Phishing to a default @virus_name_to_spam_score_maps;
- reworded some notification texts;
SUPERVISED PROCESS NOTES
Socket activation (running under a superserver with fd-holding) is
currently not available. Note that 'amavisd reload' does fd-holding
and socket passing to a new incarnation of amavisd server on its own,
which means that a client (an MTA) does not see a disruption on a
reload (= warm restart), unlike in case of restarting amavisd.
As a reminder: a full restart is only necessary when changing the set
of listening sockets in a configuration file. For all other needs
(like changing other settings, updating SpamAssassin rules, upgrading
the amavisd program, or perl modules, or perl itself) a reload suffices.
Here is a sample file amavisd.service for use under systemd
(indented for clarity). Lightly tested on Debian 8.0 (Jessie)
on a Raspberry Pi.
/lib/systemd/system/amavisd.service
[Unit]
Description=amavisd-new mail filter
Before=shutdown.target
After=systemd-journald-dev-log.socket network-online.target
local-fs.target
Wants=network-online.target
Conflicts=shutdown.target
[Service]
Type=notify
NotifyAccess=main
KillMode=mixed
TimeoutStartSec=1min
TimeoutStopSec=3min
User=amavis
Group=amavis
WorkingDirectory=/var/lib/amavis/tmp
StandardOutput=syslog
SyslogFacility=mail
SyslogIdentifier=amavis
ProtectSystem=full
ProtectHome=yes
NoNewPrivileges=yes
ExecStart = /usr/sbin/amavisd-new -P '' foreground
ExecReload = /usr/sbin/amavisd-new -P '' reload
ExecStop = /usr/sbin/amavisd-new -P '' stop
[Install]
WantedBy = multi-user.target
Consider the following amavisd.conf settings when running as a
supervised process:
$pid_file = ''; # can be overridden by a command line option -P ''
$daemonize = 0; # also implied by a command line argument
'foreground'
$do_syslog = 0; # or set it to 1 to log to syslog instead of stderr
Mark
More information about the amavis-users
mailing list