Inbound doesn't catch SaneSecurity signature, Outbound does

[francis picabia]

On Sat, Nov 3, 2012 at 6:53 PM, Noel Jones <njones at megan.vbhcs.org> wrote:
> On 11/3/2012 1:40 PM, Jim Knuth wrote:
>> am 03.11.12 18:03 schrieb Noel Jones <njones at megan.vbhcs.org>:
>>
>>> On 11/2/2012 1:12 PM, francis picabia wrote:
>>>>
>>>> Last night one user's account was over quota and the exchange server
>>>> tried to bounce back the phishing through our smtp gateway.  It
>>>> caught 10
>>>> emails which were not caught on inbound by amavis.  I only see these
>>>> small examples.  I'm afraid our inbound is letting hundreds of
>>>> phishing in.
>>>> Is the only solution to install Debian and configure with the
>>>> config files
>>>> from our SMTP gateway which has been good at blocking phishing?
>>>>
>>>
>>>
>>> I can guarantee you that this isn't a Redhat vs. Debian issue.
>>>
>>> This is definitely a config issue.  Debugging this will require
>>> time, patience, and careful attention to detail.  There's just one
>>> little thing wrong, and you've been overlooking it for weeks now.
>>>
>>> Most of the phishing signatures are clamav type 4 signatures. For
>>> those to work, clamav must recognize the text as an email message.
>>> Maybe your mail program is adding some header that clamav doesn't
>>> recognize, preventing clamav type 4 signatures from working properly.
>>>
>>> http://www.freelists.org/post/sanesecurity/Purpose-of-sanesecurityftm-file,1
>>>
>>>
>>> http://www.clamav.net/doc/latest/signatures.pdf
>>>
>>> Maybe your sanesecurity.ftm file is missing, or your MTA is adding
>>> some other header that clamav doesn't recognize.  If you're adding
>>> some local header, you can create your own "local.ftm" file using
>>> the clamav documentation.
>>
>> I have observed the same for a long time with myself.
>> The sanesecurity.ftm and daily.ftm is fine with
>> daily updates.
>
> It's worth while making sure that this is OK and working as
> expected.  This is a prime suspect.

My outbound catches and quarantines the message.  I copy that
file over to my MX and run clamscan.  The MX which missed this
on the inbound, running clamscan, detects the phishing
or whatever Sanesecurity signature.  Does this tell us the
clam setup is good?

> Examine the files placed on disk by amavisd; make sure clamav can
> scan it as an email file.  Just because the .ftm files are there
> doesn't guarantee that *your* mail files match.
>
>>
>>>
>>> Maybe your amavisd-new is not configured to pass the full email
>>> message to clamav, just passing decoded parts instead -- many of the
>>> signatures only work on the whole email.
>>
>> What do you mean  with that?
>
> From the amavisd-new RELEASE-NOTES
>
> - uncommented the qr'^MAIL$' in @keep_decoded_original_maps
> (amavisd.conf
>   and amavisd.conf-sample); seems it is becoming increasingly more
> important
>   for virus scanners to also see the complete undecoded message;
> suggested
>   by Michael Scheidell and others;
>
>
> check your @keep_decoded_original_maps entry, and make sure that
> part is working as expected.  This is a prime suspect.

Thanks for looking this over and for your input and suggestions.

I was hoping I had found the flaw a few weeks ago when I read about
the commented out part of "keep_decoded_original_maps".  At that
time I found mine had  been commented out through
historical configuration reasons.

I've been running with this for a few weeks:

@keep_decoded_original_maps = (new_RE(
  qr'^MAIL$',   # retain full original message for virus checking (can be slow)
  qr'^MAIL-UNDECIPHERABLE$', # recheck full mail if it contains undecipherables
  qr'^(ASCII(?! cpio)|text|uuencoded|xxencoded|binhex)'i,
# qr'^Zip archive data',     # don't trust Archive::Zip
));

While running this way, I've seen about 30 emails caught by outbound which
were forwards from cyrus sieve.  I think this helped increase the catch,
but it didn't make it complete.

I made a new amavisd.conf on the Redhat MX system from concatenated amavis
config files on Debian (with the necessary Debian specific edits
altered).  The missed
inbound phishing emails continue to be caught on the SMTP.

The other items mentioned...

Postfix built from source is same on both systems.  Configuration should be
nearly identical on both MX systems.

Sanesecurity signatures are updated from scamp shell script on both systems.

The files mentioned seem to be the same on both boxes (md5sum matches).

# md5sum /var/clamav/sanesecurity.ftm
d94277f3259f4a0e9a44136b069fde99  /var/clamav/sanesecurity.ftm

Freshclam is updated once a day on both systems.

Amavis on the SMTP is from Debian, while on Redhat it comes from rpmforge repo.

While the Debian system was put into use as primary MX for a week,
there were zero forwards blocked by amavis/clamav, as you'd expect since
the same system scanned it on inbound.

I suspect while this was configured, it was catching more Sanesecurity
signatures.

I suppose it is possible that they just catch different stuff and if I had
the Redhat system (amavis 2.6.6) doing SMTP, while Debian (amavisd-new 2.6.4)
handled primary MX, the same thing might happen?