Amavisd and Bayes (again...)
Alex Masidlover
alex.masidlover at zednax.com
Thu Nov 24 10:09:08 CET 2016
Hi,
I'm currently being deluged with spam and have been trying to use BAYES
filters to try and get rid of some of it. I've made a lot of progress
but am now very stuck.
I have go to the point where I have (temporarily) given the amavis user
a shell and when I run spamassassin on an email from the command line I
get:
spamassassin -t </tmp/sample3.txt
Content analysis details: (7.4 points, 5.0 required)
pts rule name description
---- ---------------------- -----------------------------------------
---------
3.5 BAYES_99 BODY: Bayes spam probability is 99 to 100%
[score: 1.0000]
1.8 REMOVE_BEFORE_LINK BODY: Removal phrase right before a link
0.2 BAYES_999 BODY: Bayes spam probability is 99.9 to
100%
[score: 1.0000]
0.0 HTML_MESSAGE BODY: HTML included in message
1.1 DCC_CHECK Detected as bulk mail by DCC (dcc-
servers.net)
0.8 RDNS_NONE Delivered to internal network by a host
with no rDNS
The debugs show:
[Tue Nov 22 16:12:01] amavis at mta0 ~ $ spamassassin -D -t
</tmp/sample3.txt 2>&1 | grep -i bayes
Nov 22 16:12:10.355 [10336] dbg: plugin: loading
Mail::SpamAssassin::Plugin::Bayes from @INC
Nov 22 16:12:10.603 [10336] dbg: config: fixed relative path:
/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf
Nov 22 16:12:10.603 [10336] dbg: config: using
"/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf"
for included file
Nov 22 16:12:10.603 [10336] dbg: config: read file
/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf
Nov 22 16:12:11.594 [10336] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0x27d2868) implements
'learner_new', priority 0
Nov 22 16:12:11.595 [10336] dbg: bayes: learner_new
self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x27d2868),
bayes_store_module=Mail::SpamAssassin::BayesStore::DBM
Nov 22 16:12:11.609 [10336] dbg: bayes: learner_new: got
store=Mail::SpamAssassin::BayesStore::DBM=HASH(0x2fa76c8)
Nov 22 16:12:11.609 [10336] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0x27d2868) implements
'learner_is_scan_available', priority 0
Nov 22 16:12:11.613 [10336] dbg: bayes: tie-ing to DB file R/O
/var/amavis/.spamassassin/bayes_toks
Nov 22 16:12:11.614 [10336] dbg: bayes: tie-ing to DB file R/O
/var/amavis/.spamassassin/bayes_seen
Nov 22 16:12:11.614 [10336] dbg: bayes: found bayes db version 3
Nov 22 16:12:13.528 [10336] dbg: bayes: untie-ing
However, when the same email was received through amavsid-new it
received the following headers:
X-Virus-Scanned: amavisd-new at zednax.com
X-Spam-Flag: NO
X-Spam-Score: 3.962
X-Spam-Level: ***
X-Spam-Status: No, score=3.962 tagged_above=0 required=4
tests=[DCC_CHECK=1.1, HTML_MESSAGE=0.001, RDNS_NONE=1.274,
REMOVE_BEFORE_LINK=1.587] autolearn=no autolearn_force=no
I start amavisd in screen with debugs on as the same user; the debugs
from amavisd show:
Nov 22 15:55:46.359 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: loading Mail::SpamAssassin::Plugin::Bayes from @INC
Nov 22 15:55:46.569 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
config: fixed relative path:
/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf
Nov 22 15:55:46.569 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
config: using
"/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf"
for included file
Nov 22 15:55:46.569 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
config: read file
/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf
Nov 22 15:55:47.565 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'learner_new', priority 0
Nov 22 15:55:47.566 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: learner_new
self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00),
bayes_store_module=Mail::SpamAssassin::BayesStore::DBM
Nov 22 15:55:47.566 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: learner_new: got
store=Mail::SpamAssassin::BayesStore::DBM=HASH(0x5b102c8)
Nov 22 15:55:47.566 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'learner_is_scan_available', priority 0
Nov 22 15:55:47.566 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks
Nov 22 15:55:47.567 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_seen
Nov 22 15:55:47.567 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: found bayes db version 3
Nov 22 15:55:49.089 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'learner_close', priority 0
Nov 22 15:55:49.089 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: untie-ing
Nov 22 15:55:49.089 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'prefork_init', priority 0
Nov 22 15:55:49.090 mta0.zednax.com /usr/sbin/amavisd[7630]:
SpamAssassin loaded plugins: AskDNS, AutoLearnThreshold, Bayes,
BodyEval, Check, DCC, DKIM, DNSEval, FreeMail, HTMLEval, HTTPSMismatch,
HeaderEval, ImageInfo, MIMEEval, MIMEHeader, Pyzor, Razor2, RelayEval,
ReplaceTags, SpamCop, URIDetail, URIEval, VBounce, WLBLEval,
WhiteListSubject
Nov 22 15:55:49.104 mta0.zednax.com /usr/sbin/amavisd[7649]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'spamd_child_init', priority 0
at startup, then when processing the message:
Nov 22 16:08:37.091 mta0.zednax.com /usr/sbin/amavisd[9727]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'spamd_child_init', priority 0
Nov 22 16:08:37.223 mta0.zednax.com /usr/sbin/amavisd[9727]: (09727-01)
SA dbg: bayes: tie-ing to DB file R/O
/var/amavis/.spamassassin/bayes_toks
Nov 22 16:08:37.224 mta0.zednax.com /usr/sbin/amavisd[9727]: (09727-01)
SA dbg: bayes: tie-ing to DB file R/O
/var/amavis/.spamassassin/bayes_seen
Nov 22 16:08:37.224 mta0.zednax.com /usr/sbin/amavisd[9727]: (09727-01)
SA dbg: bayes: found bayes db version 3
I'm even seeing debugs that show amavisd learning messages it detects
as spam (using non-bayes rules):
Nov 22 16:07:40.228 mta0.zednax.com /usr/sbin/amavisd[9064]: (09064-11)
SA dbg: locker: safe_lock: created
/var/amavis/.spamassassin/bayes.lock.mta0.zednax.com.9064
Nov 22 16:07:40.228 mta0.zednax.com /usr/sbin/amavisd[9064]: (09064-11)
SA dbg: locker: safe_lock: trying to get lock on
/var/amavis/.spamassassin/bayes with 0 retries
Nov 22 16:07:40.228 mta0.zednax.com /usr/sbin/amavisd[9064]: (09064-11)
SA dbg: locker: safe_lock: link to
/var/amavis/.spamassassin/bayes.lock: link ok
Nov 22 16:07:40.229 mta0.zednax.com /usr/sbin/amavisd[9064]: (09064-11)
SA dbg: bayes: tie-ing to DB file R/W
/var/amavis/.spamassassin/bayes_toks
Nov 22 16:07:40.229 mta0.zednax.com /usr/sbin/amavisd[9064]: (09064-11)
SA dbg: bayes: tie-ing to DB file R/W
/var/amavis/.spamassassin/bayes_seen
Nov 22 16:07:40.230 mta0.zednax.com /usr/sbin/amavisd[9064]: (09064-11)
SA dbg: bayes: found bayes db version 3
Nov 22 16:07:40.396 mta0.zednax.com /usr/sbin/amavisd[9064]: (09064-11)
SA dbg: bayes: learned '3cbcccb5747f8488582ac93a965e6c8590b465c2 at sa_gen
erated', atime: 1479830854
Having read numerous threads of admins with similar issues I expect it
will come down to permissions, but I've tried 0666 and 0777 as the file
mode. The options currently set in the spamassassin config are:
skip_rbl_checks 0
use_bayes 1
auto_learn 0
bayes_path /var/amavis/.spamassassin/bayes
bayes_file_mode 0777
bayes_auto_expire 0
The directory looks like:
[Wed Nov 23 09:13:55] mta0 ~ # ls -la /var/amavis/.spamassassin/*
-rw-rw-rw- 1 amavis amavis 22 Nov 22 16:19
/var/amavis/.spamassassin/bayes.lock
-rw-rw-rw- 1 amavis amavis 2200 Nov 23 09:14
/var/amavis/.spamassassin/bayes.lock.mta0.zednax.com.18174
-rwxrwxrwx 1 amavis amavis 167673856 Nov 22 16:19
/var/amavis/.spamassassin/bayes_seen
-rwxrwxrwx 1 amavis amavis 5382144 Nov 22 16:19
/var/amavis/.spamassassin/bayes_toks
-rwxrwxrwx 1 amavis amavis 1869 Nov 22 11:29
/var/amavis/.spamassassin/user_prefs
And the magic dump looks like:
[Wed Nov 23 09:14:14] mta0 ~ # sa-learn --username=amavis --dump magic
0.000 0 3 0 non-token data: bayes db
version
0.000 0 387589 0 non-token data: nspam
0.000 0 922763 0 non-token data: nham
0.000 0 175867 0 non-token data: ntokens
0.000 0 1478796541 0 non-token data: oldest atime
0.000 0 1479831533 0 non-token data: newest atime
0.000 0 1479831423 0 non-token data: last journal
sync atime
0.000 0 1479802087 0 non-token data: last expiry
atime
0.000 0 0 0 non-token data: last expire
atime delta
0.000 0 0 0 non-token data: last expire
reduction count
Any help would be appreciated before I drown in spam...
--
Technical Director - Zednax Limited
W: http://www.zednax.com
T: +44 333 444 0160
F: +44 161 660 8010
Zednax Limited is registered in England and Wales, Company no.
05321754.
Registered address: Meadow House, Meadow Lane, Nottingham, NG2 3HS.
Zednax Limited is VAT registered, VAT registration no. GB 855 4468 92.
More information about the amavis-users
mailing list