ANNOUNCE: amavisd-new-2.8.1 has been released
Mark Martinec
Mark.Martinec+amavis at ijs.si
Fri Jun 28 22:06:30 CEST 2013
Version 2.8.1 of amavisd-new has been released.
It is available at:
http://www.ijs.si/software/amavisd/amavisd-new-2.8.1.tar.gz
or:
http://www.ijs.si/software/amavisd/amavisd-new-2.8.1.tar.xz
amavisd-new-2.8.1 release notes
COMPATIBILITY
- when 0MQ (a.k.a. ZeroMQ) is used between Amavis components as an
internal messaging protocol, make sure to replace all 0MQ-enabled
Amavis components on upgrading amavisd, as the internal protocol
has changed slightly, taking advantage of 0MQ multi-part messages
for better performance. Affected programs are: amavis-services,
amavisd-status, amavisd-snmp-subagent-zmq, and amavisd.
NOTE: The Crossroads I/O project (libxs) ceased development on
July 2012, to be replaced by nanomsg eventually by the same author.
The 0MQ library (libzmq) is currently (2013) the best choice,
the preferred library version is 3.2.2 or later along with
the ZMQ::LibZMQ3 Perl interface module and ZMQ::Constants.
The older version 2 of the library, along with an older perl
module ZeroMQ, should be fine too, but lacks support for IPv6.
- amavisd is compatible with perl 5.18.0 and with SpamAssassin 3.4.0
BUG FIXES
- fixed a bug in the SMTP client code, where the final SMTP status did
not reflect a failure status of a DATA command from a back-end MTA.
This caused a reception of a mail message to be confirmed but a message
was then lost, as it could not be passed to a back-end MTA. The bug
went unnoticed for years, as the commonly used MTAs normally reject
either at the MAIL FROM, at RCPT TO, or at the data-dot stage, but not
at the DATA command. Reported by Deniska-rediska;
- fixed calling an external spam scanner DSPAM or Bogofilter, which
failed with a message:
auto-learning with spam scanner ... failed: error running program
Reported by Tonio;
- if a configuration file path as given through a command line option -c
or as an argument to include_config_files() was not an absolute path,
and that file contained an error, the do() would search the @INC list
for alternative files of the same name, and reported an unrelated error
(typically: No such file or directory) instead of reporting the true
reason for a failure;
- fixed a regular expression in amavisd.conf for an 'Avast!' AV entry
to properly extract a virus name; a patch by Ralf Hildebrandt;
- added LDAP errors LOCAL_ERROR and OPERATIONS_ERROR to the set of
expected error conditions which lets amavisd retry the failed
operation; a patch by Quanah Gibson-Mount;
NEW FEATURES SUMMARY
- new Redis storage for the "pen pals" feature;
- improved IPv6 support;
- support for p0f v3;
- new macros ip_trace_all and ip_trace_public;
- amavisd-status now shows a bar graph display
of the number of active processes;
- the timing report log entry can show CPU usage
at log level 2 if a module Unix::Getrusage is available;
NEW FEATURES
- new Redis storage for the "pen pals" feature: instead of (or in addition
to) the existing SQL storage for keeping data on past mail messages
and contributing negative spam score to recent/ongoing correspondence,
this data can now be kept on a Redis server. Unlike the SQL backend,
to minimize memory usage the Redis backend keeps only data which are
required for pen pals operation.
Redis storage for pen pals can offer a small speedup compared to a
well-tuned SQL server, offers automatic expiration of data based on a
configured time-to-live setting, and a simpler setup (no need to manually
set up an SQL schema). A drawback is that a Redis server keeps all data
in memory (with optional periodic persistence on disk), which might be
of concern for busy sites with a long time-to-live setting. Potential
drawbacks of a Redis server are also its lack of sophisticated access
controls, and lack of IPv6 support in a current version.
A redis database may be shared between hosts running amavisd. It can
be accessed either locally over a Unix socket, or using an INET socket
(IPv4 only) over a loopback interface (better security) or over a local
network. Currently (version 2.6.14) a Redis server does not offer
access over IPv6, which is planned (but not promised) for version 2.8.
Required dependencies when Redis support is enabled are a perl module
"Redis" ( http://search.cpan.org/dist/Redis/ ) version 1.954 or later,
and a redis server ( http://redis.io/ ) with support for Lua scripting
(i.e. version 2.6 or later). Most pen pals application-level details on
queries and storage management is delegated to Lua scripts running on a
Redis server.
Expiration time of items stored in a redis database is controlled by
a setting $storage_redis_ttl, which is a time-to-live time in seconds
and defaults to 16 days:
$storage_redis_ttl = 16*24*60*60;
Redis support in amavisd is enabled by setting a list @storage_redis_dsn
to a nonempty value (similar to @storage_sql_dsn for an SQL support).
If @storage_redis_dsn is empty, the redis support code is not loaded
and does not occupy any storage.
The configuration setting @storage_redis_dsn is a list of hashrefs
(a hashref is a { ... } in perl syntax), each of which specifies
one Redis server that can be used: if there is more than one entry
in the list, a connection to each server is attempted until one is
found where connection succeeds. Each entry is an associative array
of key/value options which are passed on to a new() method of a perl
Redis module unmodified and unverified. Usual options are: 'server',
'sock', 'reconnect' - see documentation of a Redis module for details.
All Redis module options have their default value, so it is alright
to specify an empty hash, which means to connect to a default server.
Apart from options which are passed to a Redis module, two additional
options are interpreted by amavisd itself and are not passed on to a
Redis perl module. The 'db_id' options is an optional database index,
used in a SELECT redis command to choose a (sub)database to use.
By default a database index is 0. The 'ttl' option overrides a global
time-to-live setting as specified in $storage_redis_ttl, allowing to
chose different expiration times of stored items for each server.
Examples:
# disables Redis (is a default)
@storage_redis_dsn = ();
# enables Redis, use a single default local redis server, all
# defaults are supplied by a Redis perl module, database index 0
@storage_redis_dsn = ( {} );
# access a local redis server over a loopback interface on TCP port
# 6379, select database index 1, try reconnecting for 20 seconds
# before giving up when a redis server is down (or restarting)
@storage_redis_dsn =
( { server => 'localhost:6379', reconnect => 20, db_id => 1 } );
$storage_redis_ttl = 16*24*60*60; # expiration time for data 16 days
@storage_redis_dsn = (
{ sock => '/tmp/redis.sock', reconnect => 20, db_id => 1 },
{ server => 'localhost:6379', reconnect => 20, db_id => 1 },
{ server => 'backup.example.com:6379', db_id => 1, ttl => 5*24*60*60 },
);
Some existing settings also affect Redis pen pals operation:
$database_sessions_persistent, $penpals_bonus_score,
$penpals_halflife, $penpals_threshold_low, $penpals_threshold_high.
Starting with version 3.4.0 the module SpamAssassin can also use Redis
storage for its global Bayes database. Amavisd and SpamAssassin can use
the same Redis server for their databases, although it is sensible that
they use separate (sub)databases by choosing a different database index
(redis SELECT command) through a 'db_id' configuration option, which
defaults to 0.
NOTE: As more experience with Redis is gained, it is possible that
a redis storage schema may change in future versions, possibly in an
incompatible way. As its purpose is short-term storage, this should
not be of great concern.
- improved IPv6 support: p0f-analyzer.pl can now communicate with amavisd
processes over an INET6 socket (or over an INET or UNIX socket as before).
Extended the protocol between amavisd and p0f-analyzer.pl to allow
queries on IPv6 addresses;
- rewritten p0f-analyzer.pl to support a newer p0f v3 output format,
while still recognizing an older p0f v2 output format;
suggested by Jernej Porenta;
NOTE: the p0f v3 does not provide a compact output on stdout like p0f v2
could by using an option -l, so leave out the option -l with p0f v3;
- improved IPv6 support: program amavisd-snmp-subagent-zmq can now
attach as an AgentX to a Net-SNMP daemon snmpd over an INET6 socket;
the AgentX socket is specified by $agentx_sock_specs near the beginning
of a file amavisd-snmp-subagent-zmq, e.g.:
$agentx_sock_specs = 'tcp6:localhost:705'; # talk to snmpd over IPv6
$agentx_sock_specs = 'tcp:localhost:705'; # talk to snmpd over IPv4
$agentx_sock_specs = '/var/agentx/master'; # talk over a UNIX socket
- improved IPv6 support: program amavisd-submit can now submit a mail
message to amavisd over an INET6 socket;
- a macro W can now produce a list of all virus scanners invoked,
along with a list of virus names each scanner detected;
suggested by Patrick Ben Koetter;
- new macros ip_trace_all and ip_trace_public:
ip_trace_all provides a list of IP addresses found in the 'Received from'
trace of a mail header, one entry for each Received header field,
including possibly invalid IP addresses and private IP addresses;
Missing addresses are substituted by with a '?' (e.g. in Received
header fields for local or other non-IP mail submissions).
The list order corresponds to the order of 'Received' header fields
as found in a mail header, top-down, i.e. first entry of the list
is the topmost (the most recent) 'Received' header field, so
chronologically in reverse;
ip_trace_public provides a list of valid public IP addresses found in the
'Received from' trace of a mail header. Missing, invalid or private
IP addresses are not included in this list, so there may be more
'Received' header fields in a mail header then entries in this list.
The list order corresponds to the order of 'Received' header fields
as found in a mail header, top-down, i.e. first entry of the list
is the topmost (the most recent) 'Received' header field with a valid
public IP address, so chronologically in reverse;
suggested by Tomislav Mihaliček;
- templates for administrator notifications, recipient notifications,
and sender notifications now use macro 'ip_trace_all' instead of macros
'e' and 't' in order to report the full 'received' trace, not just the
first hop;
- macro supplementary_info recognizes new arguments: VERSION, SUBVERSION,
and RULESVERSION, providing additional information from SpamAssassin
correspond to equivalent SpamAssassin tags;
- a new command line option -X allows controlling some exotic features,
useful for example in debugging or automatic testing. The option takes
one argument which is a comma-separated list of keywords. Currently
the only recognized option is '-X no_conf_file_writable_check',
which disables security checks on configuration files, which can be
useful in automatic testing, but is dangerous to use in production.
Suggested by Alexander Wirt;
- a configuration setting $sa_debug may now specify a comma-separated
list of SpamAssassin debug facilities, complementing a similar method
of specifying these facilities through a command line option -d.
If $sa_debug looks like a simple boolean (or is undefined), the
traditional semantics still applies: a false prepends an 'info'
to the list, while a true prepends 'info' and 'all' to the list
of SpamAssassin debug facilities.
Examples:
$sa_debug = 0; # same as: $sa_debug = 'info';
$sa_debug = 1; # same as: $sa_debug = 'info,all';
$sa_debug = 'info,dns,async,bayes';
- pass the size of an original mail body as a 'supplementary attribute'
to SpamAssassin for the benefit of a 'check_body_length' eval rule
(new with SpamAssassin 3.4.0). The original mail body size may
differ from the message as seen by SpamAssassin in case of truncation
of large messages to mail_body_size_limit.
- to the output of amavisd-status add a simple bar graph display (with
an exponential-decay peak indicator) of the number of active processes;
- if a module Unix::Getrusage is available, the timing report log entry
(at log level 2) is enhanced: in addition to total elapsed time
(wall clock) spent in processing a message, it also shows a sum of
CPU user and system times spent by amavisd process and its spawned
processes:
old format example:
size: 3815, TIMING [total 1901 ms] - ...
new format example:
size: 3815, TIMING [total 1901 ms, cpu 657 ms] - ...
Additionally, a separate RUSAGE log entry is produced at log level 2,
indicating resource usage spent by the last task. A field maxrss is
a gauge (an absolute current value), all other fields are counters,
so a difference between a previous and a current value is shown in
the log. Each field value is a pair of numbers delimited by a plus:
the first value corresponds to resource usage by the reporting amavisd
child process, the second value corresponds to its spawned processes
(e.g. file(1), gzip(1), etc.).
Example (wrapped for clarity):
size: 3815, RUSAGE minflt=10114+5223, majflt=0+0,
nswap=0+0, inblock=0+0, oublock=9+0,
msgsnd=819+9, msgrcv=211+3, nsignals=0+0,
nvcsw=128+19, nivcsw=32+41, maxrss=164304+194012,
ixrss=520+14016, idrss=66300+128392, isrss=24960+7680,
utime=0.390+0.079, stime=0.079+0.108
See getrusage(2) Unix man page for details.
OTHER
- the 'amavisd genrsa' command will now warn if the requested DKIM
signing key size is below 1024 bits, as required by RFC 6376;
- on amavisd startup a check on available private DKIM signing keys
(as declared by dkim_key) will now warn if a key size is below
1024 bits as required by RFC 6376, and log an information message
if a key size is below a configured $dkim_minimum_key_bits size
(defaults to 1024, currently 768 would still be a sensible value);
- for purposes of DKIM-based whitelisting (@author_to_policy_bank_maps)
and @signer_reputation_maps spam scores, valid signatures with
public keys shorter than $dkim_minimum_key_bits bits (default 1024,
equivalent to a lower limit as presently used by Google) are now
ignored, with an informational message logged at level 1.
To disable this check, set $dkim_minimum_key_bits to undef or to 0.
- consider Unique local addresses (ULA) fc00::/7 non-public (RFC 4193),
dropped site-local addresses fec0::/10 (deprecated by RFC 3879),
adjusting the default setting of @mynetworks accordingly;
- consider the "Shared Address Space" 100.64.0.0/10 non-public (RFC 6598);
- adjust parsing the <zone_id> syntax of a scoped IPv6 address
as per RFC 6874;
- updated an AV entry for a Sophos Anti Virus: the scanning program
used to be named 'sweep', now it is 'savscan'; thanks to mefiX;
- updated a default value of @virus_name_to_spam_score_maps: updated
entry for Doppelstern and added entries for Bofhland and PORCUPINE;
- increase an arbitrary sanity limit on %smtp_reason_by_ccat strings
from 100 to 450 characters (RFC 5321 allows 512 character reply lines);
- relax testing file type of a configuration file, now a configuration
may also be passed to amavisd through a named pipe (fifo), possibly
facilitating testing or unusual deployments;
- relax a requirement that a $QUARANTINEDIR directory needs to be
writable: if $*_quarantine_method template settings include a
subdirectory (e.g.: $spam_quarantine_method='local:W%P/spam/%m.gz'),
such subdirectories must already exist and should be writable,
but the top-level $QUARANTINEDIR directory need not be writable;
- convert an IPv4-mapped IPv6 address into a plain IPv4 dot-quad form
when found in Received header fields, in socket local or peer address,
in ADDR field of an XFORWARD smtp extension command, or in an AM.PDP
attribute client_address. See draft-cmetz-v6ops-v4mapped-api-harmful
and draft-itojun-v6ops-v4mapped-harmful for potential caveats;
- drop a support for direct queries to p0f v2, as it never worked well
due to bugs in p0f v2. The p0f v3 changed the query protocol, but
a query does not include port numbers (see RFC 6302), so using the
p0f-analyzer.pl interface is still the only reliable approach;
- use sysread() instead of read() when reading from /dev/urandom
to avoid leaving entropy data in I/O buffers; also changed interface
name to sub read_random_bytes(), which now reads directly into a
scalar buffer, provided by an argument;
- fix uniform random distribution when generating a random PIN for an
attachment password (when releasing and $release_format is 'attach')
(not a security issue);
- added keepalive options to a call to Net::LDAP->new, recognized since
Net::LDAP 0.53; a patch by Quanah Gibson-Mount;
- removed option inet6 from a default LDAP setup ( $ldap_sys_default ),
as Net::LDAP changed semantics in an incompatible way;
presumably the Net::LDAP now does the right thing by default;
suggested by Quanah Gibson-Mount;
- use a low-level 0MQ interface instead of ZeroMQ / ZMQ abstractions;
(i.e. ZeroMQ raw interface or ZMQ::LibZMQ3 or ZMQ::LibZMQ2);
- taking advantage of 0MQ multi-part messages the number of IP packets
transmitted is now radically decreased in favour of sending larger
but fewer packets;
- when generating 'Abuse Reporting Format (ARF) Reports' add a field
Source-IP and use UTC timestamps in the Arrival-Date field, in accordance
with RFC 6692;
- drop (opportunistic) loading of a module Devel::SawAmpersand and testing
the Devel::SawAmpersand::sawampersand(), variables $&, $` and $' are
no longer slow since Perl 5.17.7, the PL_sawampersand became a constant,
there is no longer any need of report it;
- documentation update: remove vestiges of a field 'spam_modifies_subj'
in README.sql, README.sql-mysql, this field was obsoleted in 2.7.0;
thanks to Patrick Ben Koetter;
Mark
More information about the amavis-users
mailing list