Amavis 2.10.1 dies and is unusable when put under moderate load

[Quanah Gibson-Mount]

We recently updated to Amavisd 2.10.1 from 2.9.0 internally, and have found 
that amavisd constantly dies while processing messages after being put 
under a moderate load in our QA environment.

For example, here is postfix passing off the email to amavisd:

Jan 19 06:57:42 zqa-211 postfix/smtp[24983]: 84B33102BD3: 
to=<user543 at zqa-211.eng.zimbra.com>, relay=127.0.0.1[127.0.0.1]:10024, 
delay=32179, delays=32178/0.01/0.01/0.22, dsn=4.4.2, status=
deferred (lost connection with 127.0.0.1[127.0.0.1] while sending end of 
data -- message may be sent more than once)

Here we can see amavis accept it, and then die:
Jan 19 06:57:42 zqa-211 amavis[13724]: (13724-01) ESMTP [127.0.0.1]:10024 
/opt/zimbra/data/amavisd/tmp/amavis-20160119T065742-13724-gste5uOH: 
<admin at zqa-211.eng.zimbra.com> -> <user543 at zqa-
211.eng.zimbra.com> SIZE=23143 Received: from zqa-211.eng.zimbra.com 
([127.0.0.1]) by localhost (zqa-211.eng.zimbra.com [127.0.0.1]) 
(amavisd-new, port 10024) with ESMTP for <user543 at zqa-21
1.eng.zimbra.com>; Tue, 19 Jan 2016 06:57:42 -0800 (PST)
Jan 19 06:57:42 zqa-211 amavis[13724]: (13724-01) Checking: iugKoVQZWTPd 
[10.15.32.142] <admin at zqa-211.eng.zimbra.com> -> 
<user543 at zqa-211.eng.zimbra.com>
Jan 19 06:57:52 zqa-211 amavis-services[18544]: PID 13724 went away, 
13724-01


I.e., it appears amavisd 2.10.1 is completely unstable and unsuable.  Does 
anyone know of any good methods to determine why it's croaking under a 
moderate load.  This does not occur under a light load.  We've now have a 
6% failure rate of mail delivery with this version.  In our test, we are 
simply simulating having 30 users doing cross-emails, so about 9500 emails 
over the course of 15 minutes, or about 100 emails a second, which is not a 
particularly heavy load.  We did not see this occur *at all* under 2.9.0.

Thanks,
Quanah

--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration