sa-learn - train SpamAssassin's Bayesian classifier
sa-learn [options] [file]...
sa-learn [options] --dump [ all | data | magic ]
Options:
--ham Learn messages as ham (non-spam)
--spam Learn messages as spam
--forget Forget a message
--use-ignores Use bayes_ignore_from and bayes_ignore_to
--sync Syncronize the database and the journal if needed
--force-expire Force a database sync and expiry run
--dbpath <path> Allows commandline override (in bayes_path form)
for where to read the Bayes DB from
--dump [all|data|magic] Display the contents of the Bayes database
Takes optional argument for what to display
--regexp <re> For dump only, specifies which tokens to
dump based on a regular expression.
-f file, --folders=file Read list of files/directories from file
--dir Ignored; historical compatibility
--file Ignored; historical compatibility
--mbox Input sources are in mbox format
--mbx Input sources are in mbx format
--showdots Show progress using dots
--progress Show progress using progress bar
--no-sync Skip synchronizing the database and journal
after learning
-L, --local Operate locally, no network accesses
--import Migrate data from older version/non DB_File
based databases
--clear Wipe out existing database
--backup Backup, to STDOUT, existing database
--restore <filename> Restore a database from filename
-u username, --username=username
Override username taken from the runtime
environment, used with SQL
-C path, --configpath=path, --config-file=path
Path to standard configuration dir
-p prefs, --prefspath=file, --prefs-file=file
Set user preferences file
--siteconfigpath=path Path for site configs
(default: /etc/mail/spamassassin)
--cf='config line' Additional line of configuration
-D, --debug [area=n,...] Print debugging messages
-V, --version Print version
-h, --help Print usage message
Given a typical selection of your incoming mail classified as spam or ham (non-spam), this tool will feed each mail to SpamAssassin, allowing it to 'learn' what signs are likely to mean spam, and which are likely to mean ham.
Simply run this command once for each of your mail folders, and it will ''learn'' from the mail therein.
Note that csh-style globbing in the mail folder names is supported;
in other words, listing a folder name as * will scan every folder
that matches. See Mail::SpamAssassin::ArchiveIterator for more details.
SpamAssassin remembers which mail messages it has learnt already, and will not re-learn those messages again, unless you use the --forget option. Messages learnt as spam will have SpamAssassin markup removed, on the fly.
If you make a mistake and scan a mail as ham when it is spam, or vice versa, simply rerun this command with the correct classification, and the mistake will be corrected. SpamAssassin will automatically 'forget' the previous indications.
Users of spamd who wish to perform training remotely, over a network,
should investigate the spamc -L switch.
message(s) as ham. If you have previously learnt any of the
messages as spam, SpamAssassin will forget them first, then re-learn them as
ham. Alternatively, if you have previously learnt them as ham, it'll skip them
this time around. If the messages have already been filtered through
SpamAssassin, the learner will ignore any modifications SpamAssassin may have
made.
message(s) as spam. If you have previously learnt any of the
messages as ham, SpamAssassin will forget them first, then re-learn them as
spam. Alternatively, if you have previously learnt them as spam, it'll skip
them this time around. If the messages have already been filtered through
SpamAssassin, the learner will ignore any modifications SpamAssassin may have
made.
ham:type: or spam:type:,
sa-learn will learn that folder appropriately, otherwise the folders will be
assumed to be of the type specified by --ham or --spam.
type above is optional, but is the same as the standard for
ArchiveIterator: mbox, mbx, dir, file, or detect (the default if not
specified).
file(s) containing the emails to be learned,
and will process them in mbox format (one or more emails per file).
file(s) containing the emails to be learned,
and will process them in mbx format (one or more emails per file).
bayes_ignore_from or a to address matches bayes_ignore_to.
The option might be used when learning from a large file of messages
from which the hammy spam messages or spammy ham messages have not
been removed.
Note: --force-expire also causes the journal data to be synchronized
into the Bayes databases.
Can also use the --regexp RE option to specify which tokens to display based on a regular expression.
WARNING: This is destructive and should be used with care.
The dump will include token and seen data. It is suitable for input back into the --restore command.
WARNING: This is a destructive operation, previous Bayes data will be wiped out.
NOTE: This option will not change to the given username, it will only attempt to act on behalf of that user. Because of this you will need to have proper permissions to be able to change files owned by username. In the case of SQL this generally is not a problem.
/usr/share/spamassassin or similar).
/etc/mail/spamassassin or similar).
$HOME/.spamassassin/user_prefs).
spamassassin -D bayes,learn,dns
For more information about which areas (also known as channels) are available, please see the documentation at:
C<http://wiki.apache.org/spamassassin/DebugChannels>;
Higher priority informational messages that are suitable for logging in normal circumstances are available with an area of ``info''.
sa-learn --sync once all the folders have
been scanned.
Clarification: The state of --no-sync overrides the bayes_learn_to_journal configuration option. If not specified, sa-learn will learn to the database directly. If specified, sa-learn will learn to the journal file.
Note: --sync and --no-sync can be specified on the same commandline, which is slightly confusing. In this case, the --no-sync option is ignored since there is no learn operation.
Note that this is currently ignored, as current versions of SpamAssassin will not perform network access while learning; but future versions may.
DB_File
module installed, it will have created files in other formats, such as
GDBM_File, NDBM_File, or SDBM_File. This switch allows you to migrate
that old data into the DB_File format. It will overwrite any data currently
in the DB_File.
Can also be used with the --dbpath path option to specify the location of the Bayes files to use.
There are now multiple backend storage modules available for storing user's bayesian data. As such you might want to migrate from one backend to another. Here is a simple procedure for migrating from one backend to another.
Note that if you have individual user databases you will have to perform a similar procedure for each one of them.
If you are migrating to SQL you can make use of the -u <username> option in sa-learn to populate each user's database. Otherwise, you must run sa-learn as the user who database you are restoring.
(Thanks to Michael Bell for this section!)
For a more lengthy description of how this works, go to http://www.paulgraham.com/ and see ``A Plan for Spam''. It's reasonably readable, even if statistics make me break out in hives.
The short semi-inaccurate version: Given training, a spam heuristics engine can take the most ``spammy'' and ``hammy'' words and apply probabilistic analysis. Furthermore, once given a basis for the analysis, the engine can continue to learn iteratively by applying both the non-Bayesian and Bayesian rulesets together to create evolving ``intelligence''.
SpamAssassin 2.50 and later supports Bayesian spam analysis, in the form of the BAYES rules. This is a new feature, quite powerful, and is disabled until enough messages have been learnt.
The pros of Bayesian spam analysis:
And the cons:
With Bayesian analysis, it's all probabilities - ``because the past says it is likely as this falls into a probabilistic distribution common to past spam in your systems''. Tell that to your users! Tell that to the client when he asks ``what can I do to change this''. (By the way, the answer in this case is ``use whitelisting''.)
Still interested? Ok, here's the guidelines for getting this working.
First a high-level overview:
sa-learn --spam /path/to/spam/folder
sa-learn --ham /path/to/ham/folder
...
Let SpamAssassin proceed, learning stuff. When it finds ham and spam it will add the ``interesting tokens'' to the database.
sa-learn --ham --no-sync mailmessage
This is handy for binding to a key in your mail user agent. It's very fast, as
all the time-consuming stuff is deferred until you run with the --sync
option.
Learning filters require training to be effective. If you don't train them, they won't work. In addition, you need to train them with new messages regularly to keep them up-to-date, or their data will become stale and impact accuracy.
You need to train with both spam and ham mails. One type of mail alone will not have any effect.
Note that if your mail folders contain things like forwarded spam, discussions of spam-catching rules, etc., this will cause trouble. You should avoid scanning those messages if possible. (An easy way to do this is to move them aside, into a folder which is not scanned.)
If the messages you are learning from have already been filtered through
SpamAssassin, the learner will compensate for this. In effect, it learns what
each message would look like if you had run spamassassin -d over it in
advance.
Another thing to be aware of, is that typically you should aim to train with at least 1000 messages of spam, and 1000 ham messages, if possible. More is better, but anything over about 5000 messages does not improve accuracy significantly in our tests.
Be careful that you train from the same source -- for example, if you train on old spam, but new ham mail, then the classifier will think that a mail with an old date stamp is likely to be spam.
It's also worth noting that training with a very small quantity of ham, will produce atrocious results. You should aim to train with at least the same amount (or more if possible!) of ham data than spam.
On an on-going basis, it is best to keep training the filter to make sure it has fresh data to work from. There are various ways to do this:
(An easy way to do this, by the way, is to create a new folder for 'deleted' messages, and instead of deleting them from other folders, simply move them in there instead. Then keep all spam in a separate folder and never delete it. As long as you remember to move misclassified mails into the correct folder set, it is easy enough to keep up to date.)
SpamAssassin does not support this method, due to experimental results which strongly indicate that it does not work well, and since Bayes is only one part of the resulting score presented to the user (while Bayes may have made the wrong decision about a mail, it may have been overridden by another system).
It should be supplemented with some supervised training in addition, if possible.
This is the default, but can be turned off by setting the SpamAssassin
configuration parameter bayes_auto_learn to 0.
sa-learn and the other parts of SpamAssassin's Bayesian learner, use a set of persistent database files to store the learnt tokens, as follows.
This database also contains some 'magic' tokens, as follows: the version number of the database, the number of ham and spam messages learnt, the number of tokens in the database, and timestamps of: the last journal sync, the last expiry run, the last expiry token reduction count, the last expiry timestamp delta, the oldest token timestamp in the database, and the newest token timestamp in the database.
This is a database file, using DB_File. The database 'version
number' is 0 for databases from 2.5x, 1 for databases from certain 2.6x
development releases, 2 for 2.6x, and 3 for 3.0 and later releases.
This is a database file, using DB_File.
sa-learn --sync) be used to synchronize
the Bayes DB.
Also, through the use of bayes_learn_to_journal, or when using the
--no-sync option with sa-learn, the actual learning data will take
be placed into the journal for later synchronization. This is typically
useful for high-traffic sites to avoid the same contention as stated
above.
Since SpamAssassin can auto-learn messages, the Bayes database files could increase perpetually until they fill your disk. To control this, SpamAssassin performs journal synchronization and bayes expiration periodically when certain criteria (listed below) are met.
SpamAssassin can sync the journal and expire the DB tokens either manually or opportunistically. A journal sync is due if --sync is passed to sa-learn (manual), or if the following is true (opportunistic):
and either:
or
Expiry is due if --force-expire is passed to sa-learn (manual), or if all of the following are true (opportunistic):
If either the manual or opportunistic method causes an expire run to start, here is the logic that is used:
Go through each of the DB's tokens. Starting at 12hrs, calculate whether or not the token would be expired (based on the difference between the token's atime and the db's newest token atime) and keep the count. Work out from 12hrs exponentially by powers of 2. ie: 12hrs * 1, 12hrs * 2, 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512 (6144hrs, or 256 days).
The larger the delta, the smaller the number of tokens that will be expired. Conversely, the number of tokens goes up as the delta gets smaller. So starting at the largest atime delta, figure out which delta will expire the most tokens without going above the goal expiration count. Use this to choose the atime delta to use, unless one of the following occurs:
If the expire run gets past this point, it will continue to the end. A new DB is created since the majority of DB libraries don't shrink the DB file when tokens are removed. So we do the ``create new, migrate old to new, remove old, rename new'' shuffle.
bayes_auto_expire is used to specify whether or not SpamAssassin
ought to opportunistically attempt to expire the Bayes database.
The default is 1 (yes).bayes_expiry_max_db_size specifies both the auto-expire token
count point, as well as the resulting number of tokens after expiry
as described above. The default value is 150,000, which is roughly
equivalent to a 6Mb database file if you're using DB_File.bayes_journal_max_size specifies how large the Bayes
journal will grow before it is opportunistically synced. The
default value is 102400.
The sa-learn command is part of the Mail::SpamAssassin Perl module.
Install this as a normal Perl module, using perl -MCPAN -e shell,
or by hand.
spamassassin(1)
spamc(1)
Mail::SpamAssassin(3)
Mail::SpamAssassin::ArchiveIterator(3)
<http://www.paulgraham.com/> Paul Graham's ``A Plan For Spam'' paper
<http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.html>
Gary Robinson's f(x) and combining algorithms, as used in SpamAssassin
<http://www.bgl.nu/~glouis/bogofilter/> 'Training on error' page. A discussion of various Bayes training regimes, including 'train on error' and unsupervised training.
Mail::SpamAssassin
The SpamAssassin(tm) Project <http://spamassassin.apache.org/>