ANNOUNCE: amavisd-new-2.8.1-rc1 release candidate is available

[Mark Martinec]

A long-overdue preview of a version 2.8.1
of amavisd-new is available at:

  http://www.ijs.si/software/amavisd/amavisd-new-2.8.1-rc1.tar.gz

Release notes are at:

  http://www.ijs.si/software/amavisd/release-notes.txt



amavisd-new-2.8.1-RC1 release notes

COMPATIBILITY

- when 0MQ is used between Amavis components as an internal messaging
  protocol, make sure to replace all 0MQ -enabled Amavis components
  on upgrading amavisd, as the internal protocol has changed slightly,
  taking advantage of 0MQ multi-part messages.  Affected programs are:
    amavisd, amavis-services, amavisd-snmp-subagent-zmq, amavisd-status.

  NOTE: The Crossroads I/O project ceased developmenet in July 2012,
    to be replaced by nanomsg eventually (by the same author).

    The 0MQ library (libzmq) is currenty (2012/2013) the best choice,
    the preferred library version is 3.2.2 or later along with the
    ZMQ::LibZMQ3 Perl interface module.  The older version 2 of the
    library, along with an older perl module ZeroMQ should be fine too.


BUG FIXES

- fixed a bug in the SMTP client code, where the final SMTP status did
  not reflect a failure status of a DATA command from a back-end MTA.
  This caused a reception of a mail message to be confirmed but a message
  was then lost, as it could not be passed to a back-end MTA. The bug
  went unnoticed for years, as the commonly used MTAs normally reject
  either at the MAIL FROM, at RCPT TO, or at the data-dot stage, but not
  at the DATA command. Reported by Deniska-rediska;

- fixed calling an external spam scanner DSPAM or Bogofilter, which
  failed with a message:
    auto-learning with spam scanner ... failed: error running program
  Reported by Tonio;

- if a configuration file path as given through a command line option -c
  or as an argument to include_config_files() was not an absolute path,
  and that file contained an error, the do() would search the @INC list
  for alternative files of the same name, and reported an unrelated error
  (typically: No such file or directory) instead of reporting the true
  reason for a failure;

- fixed a regular expression in amavisd.conf for an 'Avast!' AV entry
  to properly extract a virus name; a patch by Ralf Hildebrandt;

- added LDAP errors LOCAL_ERROR and OPERATIONS_ERROR to the set of
  expected error conditions which lets amavisd retry the failed
  operation; a patch by Quanah Gibson-Mount;


NEW FEATURES

- improved IPv6 support: p0f-analyzer.pl can now communicate with amavisd
  processes over an INET6 socket (or over an INET or UNIX socket as before).
  Extended the protocol between amavisd and p0f-analyzer.pl to allow
  queries on IPv6 addresses;

- rewritten p0f-analyzer.pl to support a newer p0f v3 output format,
  while still recognizing an older p0f v2 output format;
  suggested by Jernej Porenta;

  NOTE: the p0f v3 does not provide a compact output on stdout like p0f v2
  could by using an option -l, so leave out the option -l with p0f v3;

- improved IPv6 support: program amavisd-snmp-subagent-zmq can now
  attach as an AgentX to a Net-SNMP daemon snmpd over an INET6 socket;
  the AgentX socket is specified by $agentx_sock_specs near the beginning
  of a file amavisd-snmp-subagent-zmq, e.g.:
    $agentx_sock_specs = 'tcp6:localhost:705';  # talk to snmpd over IPv6
    $agentx_sock_specs = 'tcp:localhost:705';   # talk to snmpd over IPv4
    $agentx_sock_specs = '/var/agentx/master';  # talk over a UNIX socket

- improved IPv6 support: program amavisd-submit can now submit a mail
  message to amavisd over an INET6 socket;

- a macro W can now produce a list of all virus scanners invoked,
  along with a list of virus names each scanner detected;
  suggested by Patrick Ben Koetter;

- new macros ip_trace_all and ip_trace_public:

  ip_trace_all  provides a list of IP addresses found in the 'Received from'
     trace of a mail header, one entry for each Received header field,
     including possibly invalid IP addresses and private IP addresses;
     Missing addresses are substituted by with a '?' (e.g. in Received
     header fields for local or other non-IP mail submissions).
     The list order corresponds to the order of 'Received' header fields
     as found in a mail header, top-down, i.e. first entry of the list
     is the topmost (the most recent) 'Received' header field, so
     chronologically in reverse;

  ip_trace_public  provides a list of valid public IP addresses found in the
     'Received from' trace of a mail header.  Missing, invalid or private
     IP addresses are not included in this list, so there may be more
     'Received' header fields in a mail header then entries in this list.
     The list order corresponds to the order of 'Received' header fields
     as found in a mail header, top-down, i.e. first entry of the list
     is the topmost (the most recent) 'Received' header field with a valid
     public IP address, so chronologically in reverse;

  suggested by Tomislav Mihaliček;

- templates for administrator notifications, recipient notifications,
  and sender notifications now use macro 'ip_trace_all' instead of macros
  'e' and 't' in order to report the full 'received' trace, not just the
  first hop;

- macro supplementary_info recognizes new arguments: VERSION, SUBVERSION,
  and RULESVERSION, providing additional information from SpamAssassin 
  correspond to equivalent SpamAssassin tags;

- a new command line option -X allows controlling some exotic features,
  useful for example in debugging or automatic testing. The option takes
  one argument which is a comma-separated list of keywords. Currently
  the only recognized option is '-X no_conf_file_writable_check',
  which disables security checks on configuration files, which can be
  useful in automatic testing, but is dangerous to use in production.
  Suggested by Alexander Wirt;

- a configuration setting $sa_debug may now specify a comma-separated
  list of SpamAssassin debug facilities, complementing a similar method
  of specifying these facilities through a command line option -d.
  If $sa_debug looks like a simple boolean (or is undefined), the
  traditional semantics still applies: a false prepends an 'info'
  to the list, while a true prepends 'info' and 'all' to the list
  of SpamAssassin debug facilities.

  Examples:
   $sa_debug = 0;  # same as:  $sa_debug = 'info';
   $sa_debug = 1;  # same as:  $sa_debug = 'info,all';
   $sa_debug = 'info,dns,async,bayes';

- pass the original mail body size as a 'supplementary attribute'
  to SpamAssassin for the benefit of a 'check_body_length' eval rule
  (new with SpamAssassin 3.4.0). The original mail body size may
  differ from the message as seen by SpamAssassin in case of truncation
  of large messages to mail_body_size_limit.

- to the output of amavisd-status add a simple bar graph display (with
  an exponential-decay peak indicator) of the number of active processes;

- if a module Unix::Getrusage is available, the timing report log entry
  (at log level 2) is enhanced: in addition to total elapsed time
  (wall clock) spent in processing a message, it also shows a sum of
  CPU user and system times spent by amavisd process and its spawned
  processes:

  old format example:
    size: 3815, TIMING [total 1901 ms] - ...

  new format example:
    size: 3815, TIMING [total 1901 ms, cpu 657 ms] - ...

  Additionally, a separate RUSAGE log entry is produced at log level 2,
  indicating resource usage spent by the last task. A field maxrss is
  a gauge (an absolute current value), all other fields are counters,
  so a difference between a previous and a current value is shown in
  the log. Each field value is a pair of numbers delimited by a plus:
  the first value corresponds to resource usage by the reporting amavisd
  child process, the second value corresponds to its spawned processes
  (e.g. file(1), gzip(1), etc.).

  Example (wrapped for clarity):

    size: 3815, RUSAGE minflt=10114+5223, majflt=0+0,
      nswap=0+0, inblock=0+0, oublock=9+0,
      msgsnd=819+9, msgrcv=211+3, nsignals=0+0,
      nvcsw=128+19, nivcsw=32+41, maxrss=164304+194012,
      ixrss=520+14016, idrss=66300+128392, isrss=24960+7680,
      utime=0.390+0.079, stime=0.079+0.108

  See getrusage(2) Unix man page for details.


OTHER

- the 'amavisd genrsa' command will now warn if the requested DKIM
  signing key size is below 1024 bits, as required by RFC 6376;

- on amavisd startup a check on available private DKIM signing keys
  (as declared by dkim_key) will now warn if a key size is below
  1024 bits as required by RFC 6376, and log an information message
  if a key size is below a configured $dkim_minimum_key_bits size
  (defaults to 1024, currently 768 would still be a sensible value);

- for purposes of DKIM-based whitelisting (@author_to_policy_bank_maps)
  and @signer_reputation_maps spam scores, valid signatures with
  public keys shorter than $dkim_minimum_key_bits bits (default 1024,
  equivalent to a lower limit as presently used by Google) are now
  ignored, with an informational message logged at level 1.
  To disable this check, set $dkim_minimum_key_bits to undef or to 0.

- consider Unique local addresses (ULA) fc00::/7 non-public (RFC 4193),
  dropped site-local addresses fec0::/10 (deprecated by RFC 3879),
  adjusting the default setting of @mynetworks accordingly;

- consider the "Shared Address Space" 100.64.0.0/10 non-public (RFC 6598);

- adjust parsing the <zone_id> syntax of a scoped IPv6 address
  as per RFC 6874;

- updated an AV entry for a Sophos Anti Virus: the scanning program
  used to be named 'sweep', now it is 'savscan'; thanks to mefiX;

- updated a default value of @virus_name_to_spam_score_maps: updated
  entry for Doppelstern and added entries for Bofhland and PORCUPINE;

- increase an arbitrary sanity limit on %smtp_reason_by_ccat strings
  from 100 to 450 characters (RFC 5321 allows 512 character reply lines);

- relax testing file type of a configuration file, now a configuration
  may also be passed to amavisd through a named pipe (fifo), possibly
  facilitating testing or unusual deployments;

- relax a requirement that a $QUARANTINEDIR directory needs to be
  writable: if $*_quarantine_method template settings include a
  subdirectory (e.g.: $spam_quarantine_method='local:W%P/spam/%m.gz'),
  such subdirectories must already exist and should be writable,
  but the top-level $QUARANTINEDIR directory need not be writable;

- convert an IPv4-mapped IPv6 address into a plain IPv4 dot-quad form
  when found in Received header fields, in socket local or peer address,
  in ADDR field of an XFORWARD smtp extension command, or in an AM.PDP
  attribute client_address. See draft-cmetz-v6ops-v4mapped-api-harmful
  and draft-itojun-v6ops-v4mapped-harmful for potential caveats;

- drop a support for direct queries to p0f v2, as it never worked well
  due to bugs in p0f v2. The p0f v3 changed the query protocol, but
  a query does not include port numbers (see RFC 6302), so using the
  p0f-analyzer.pl interface is still the only reliable approach;

- use sysread() instead of read() when reading from /dev/urandom
  to avoid leaving entropy data in I/O buffers; also changed interface
  name to sub read_random_bytes(), which now reads directly into a
  scalar buffer, provided by an argument;

- fix uniform random distribution when generating a random PIN for an
  attachment password (when releasing and $release_format is 'attach')
  (not a security issue);

- added keepalive options to a call to Net::LDAP->new, recognized since
  Net::LDAP 0.53; a patch by Quanah Gibson-Mount;

- removed option inet6 from a default LDAP setup ( $ldap_sys_default ),
  as Net::LDAP changed semantics in an incompatible way;
  presumably the Net::LDAP now does the right thing by default;
  suggested by Quanah Gibson-Mount;

- use a low-level 0MQ interface instead of ZeroMQ / ZMQ abstractions;
  (i.e. ZeroMQ raw interface or ZMQ::LibZMQ3 or ZMQ::LibZMQ2);

- taking advantage of 0MQ multi-part messages the number of IP packets
  transmitted is now radically decreased in favour of sending larger
  but fewer packets;

- when generating 'Abuse Reporting Format (ARF) Reports' add a field
  Source-IP and use UTC timestamps in the Arrival-Date field, in accordance
  with RFC 6692; 

- drop (opportunistic) loading of a module Devel::SawAmpersand and testing
  the Devel::SawAmpersand::sawampersand(), variables $&, $` and $' are
  no longer slow since Perl 5.17.7, the PL_sawampersand became a constant,
  there is no longer any need of report it;

- documentation update: remove vestiges of a field 'spam_modifies_subj'
  in README.sql, README.sql-mysql, this field was obsoleted in 2.7.0;
  thanks to Patrick Ben Koetter;


Mark