To: users, dev, announce
Subject: ANNOUNCE: Apache SpamAssassin 3.3.0-alpha3 available

[DRAFT DRAFT DRAFT - NOT YET RELEASED - DRAFT DRAFT DRAFT]

Apache SpamAssassin 3.3.0-alpha3 is now available for testing.

Downloads are available from:
  http://people.apache.org/~jm/devel/

md5sum of archive files:

  TODO


Note that the *-rules-*.tgz files are only necessary if you cannot, or do not
wish to, run "sa-update" after install to download the latest fresh rules.

The release files also have a .asc accompanying them.  The file serves
as an external GPG signature for the given release file.  The signing
key is available via the wwwkeys.pgp.net key server, as well as
http://spamassassin.apache.org/released/GPG-SIGNING-KEY

The key information is:

pub 1024D/265FA05B 2003-06-09 SpamAssassin Signing Key <release@spamassassin.org>
    Key fingerprint = 26C9 00A4 6DD4 0CD5 AD24  F6D7 DEE0 1987 265F A05B

See the INSTALL and UPGRADE files in the distribution for important
installation notes.


Summary of major changes since 3.2.5
------------------------------------

COMPATIBILITY WITH 3.2.5

- rules are no longer distributed with the package, but installed by
  sa-update - either automatically fetched from the network (preferably),
  or from a tar archive, which is available for downloading separately

- CPAN module requirements:
  - minimum required version of ExtUtils::MakeMaker is 6.17
  - modules now required: Time::HiRes, NetAddr::IP, Archive::Tar
  - minimal version of Mail::DKIM is 0.31 (preferred: 0.36_5 or later)
  - no longer used: Mail::DomainKeys, Mail::SPF::Query

- support for versions of perl 5.6.* is being gradually revoked
  (may still work, but no promises and no support)

- preferred versions of perl are 5.8.8, 5.8.9, and 5.10.0 or later
  (of these three the 5.8.9 appears to be the most buggy)


BUILDING AND PACKAGING

- rules are no longer distributed with the package, but installed by
  sa-update

- simplify Makefile.PL and fix a bug in DESTDIR support by increasing
  the minimum ExtUtils::MakeMaker version required to 6.17

- we now include check_whitelist and check_spamd tools in the distribution,
  now called 'sa-awl' and 'sa-check_spamd'


WORKAROUNDS TO PERL BUGS AND LIMITATIONS

- let the Check.pm plugin produce smaller chunks of source code from rules
  (60 kB) to avoid Perl compiler crashing on exceeding stack size

- localize $1, $2, etc at several places, avoiding taint issue from propagating

- avoid Perl I/O bug by replacing line-by-line reading with read() where
  suitable, or play down the EBADF status in other places and only report it
  as dbg instead of a die - while also providing a little speedup (10..25%)
  on reading a message

- new sub Message::split_into_array_of_short_lines to nicely split
  a text into array of paragraph chunks of sizes between 1 kB and 2 kB,
  gives less opportunity to runaway regular expressions in rules;
  fixes bugs: 5717, 5644, 5795, 5486, 5801, 5041


MEMORY FOOTPRINT

- as a side-effect of compiling rules in smaller chunks (to avoid compiler
  crashes) virtual memory footprint of SpamAssassin is reduced;

- save some memory by not importing Pod::Usage unless it is needed;

- save 350k+ of memory in sa-compile by replacing DynaLoader with XSLoader;

- remove unneeded index from MySQL bayes_token table;


IPv6 SUPPORT

- IPv6 support for trusted_networks, internal_networks, whitelist_from_rcvd,
  msa_networks, and other stuff that uses NetSet and the Received header
  parser, using NetAddr::IP;

- allow usage of a remote dccifd host through an INET or INET6 socket;

- README.awl: increase suggested awl.ip field width to 45 to be able to
  hold IPv6 addresses (optionally);

- IP_PRIVATE now includes the ipv6 variants of private address space,
  as well as the ipv6-mapped ipv4 addresses.

- NetSet now understands that ::ffff:192.168.1.2 and 192.168.1.2 are
  the same address;

- IPv6 addresses are now recognised in Received headers;

- when reading Received header fields, the "IPv6:" prefix is stripped from
  IPv6 addresses, and "::ffff:" is removed from IPv6-mapped IPv4 addresses
  (so strings can match them as simply IPv4 addresses);

- ::1 is always included in the trusted_networks/internal_networks set
  similar to 127.0.0.0/8;


SPAMC

- add -n / --connect-timeout switch to spamc, allowing separate
  connection timeout from communication timeout;

- add --filter-retries and --filter-retry-sleep

- spamc would not time out connections to a hung spamd

- spamc client library leaks the zlib compression buffer if compression is
  used

- spamc long option '--dest' was broken


SPAMD

- when spamd is started with the daemonize option do not exit the parent
  until the child signals that it has logged the pid, to allow a wrapper
  script to simply continue immediately after starting spamd;

- additional tempfile cleanup in kill_handler;

- add SPAMD_LOCALHOST option to "make test" to allow specifying
  non-127.0.0.1 IP address for use in FreeBSD jail


API

- adding one optional argument to Mail::SpamAssassin::parse allows caller
  to pass additional out-of-band information to SpamAssassin (such as DKIM
  verification results, information about a SMTP session, or dynamic rule
  hits); this information is made available to plugins and the rest of the
  code through a 'suppl_attrib' hash;

- Plugin::Check - pick up 'rule_hits' from caller via the new mechanism
  and call got_hit() on them;

- simplified adding dynamic score hits and dynamic rules by plugins
  (such as AWL, CRM114, FuzzyOcr, Check) by letting got_hit() accept
  options tflags and description, and letting it store a supplied
  dynamic score for proper reporting;

- let the timing breakdown information be accessible to a caller through
  the existing get_tag mechanism (tag TIMING);

- let the generated header fields ('add_header' configuration options)
  be accessible to a caller through the existing get_tag mechanism
  (tag ADDEDHEADER and friends);


RULES

- rules are no longer distributed with the package;

- dropped redundant rules or rules causing too many false positives;

- added or updated rules (incomplete list in no particular order:
  vbounce, money, image spam, fill_this_form, FreeMail, European Parliament,
  HTML attachments, uri_obfu*, urinsrhsbl, urinsrhssub, urifullnsrhsbl,
  URI_OBFU_X9_WS, rDNS=localhost, INVALID_DATE_TZ_ABSURD, KHOP_SC,
  RCVD_IN_PSBL, FRT_VALIUM*, BOUNCE_MESSAGE, VBOUNCE_MESSAGE,
  __BOUNCE_UNDELIVERABLE, HELO_STATIC_HOST, FILL_THIS_FORM_FRAUD_PHISH,
  CHALLENGE_RESPONSE, DKIM_VALID, DKIM_VALID_AU, DKIM_ADSP_*,
  NML_ADSP_CUSTOM_{LOW,MED,HIGH}, __VIA_ML, MIME_BASE64_TEXT,
  FORGED_MUA_THEBAT_BOUN, FORGED_MUA_THEBAT_CS, UNRESOLVED_TEMPLATE,
  __THEBAT_MUA, __ANY_OUTLOOK_MUA, RP_MATCHES_RCVD, one-word X-Mailer, ...

- rule for plain text attachments with octet-stream MIME type;

- avoid false positives on ISO-2022-JP messages in several rules;

- updated various default whitelists, uridnsbl_skip_domain, adsp_override, ...


PLUGINS

- new plugins: FreeMail, PhishTag, Reuse
  
- now enabled by default: DKIM

- now disabled by default: AWL

- retired plugin: DomainKeys


AWL PLUGIN

- plugin AWL is now disabled by default;

- README.awl: increase suggested awl.ip field width to 45 to support IPv6
  addresses or DKIM signer domains;

- AutoWhitelist.pm: allow storing an IPv6 address (previously causing
  SQL server errors: value too long)

- let AWL keep separate records for DKIM-signed and unsigned mail
  (when auto_whitelist_distinguish_signed configuration option is true,
  and SQL field awl.ip field size is increased or made dynamic);

- gracefully handle NaN from corrupted database or a broken emulator;


DCC PLUGIN

- added support for DCC reputations, added setting dcc_rep_percent,
  new test check_dcc_reputation_range(), new tag DCCREP
  (DCC servers supply reputation data only to licensed clients);

- allow usage of a remote dccifd host through an INET or INET6 socket;


DKIM PLUGIN

- plugin is now enabled by default;

- absolute minimal version of Mail::DKIM is 0.31;
  support for ADSP requires Mail::DKIM 0.34;
  a DNS test (and rule) for NXDOMAIN is operational since Mail::DKIM 0.36_5

- supports multiple signatures (useful for whitelisting);

- distinguishes author domain signatures from third party signatures
  (useful for whitelisting);

- provides a tag DKIMIDENTITY (in addition to DKIMDOMAIN);

- DKIM now supports Author Domain Signing Practices - ADSP (RFC 5617);

- use the Mail::DKIM::AuthorDomainPolicy instead of Mail::DKIM::DkimPolicy,
  when available (since Mail::DKIM 0.34);

- implements an 'adsp_override' configuration directive and adds
  an eval:check_dkim_adsp check, which is used by new DKIM_ADSP_* rules;

- rules contain an initial set of 'adsp_override' directives, listing
  some of the more popular target domains for phishing (applicable only to
  domains which sign all their direct mail with a DKIM or DK signature);

- this plugin can now re-use Mail::DKIM verification results if made
  available by a caller, which saves resources and makes it possible
  for SpamAssassin to work on a truncated large mail without breaking
  DKIM signatures;


BUG FIXES

- fixed Rule2XSBody segfaults;

- do not treat user data as perl booleans (a string "0" is a false);

- avoid data from the wild be interpreted as perl regular expressions;

- ArchiveIterator: prevent _scan_directory from passing directories
  to _scan_file (on NFS it would fail with EISDIR on read(2);

- fixed vpopmail support;

- the 'exists:' evaluator in HEADER rules now works as documented
  and tests for existence of a header field, instead of testing for
  a header field body being nonempty; internally, the pms->get can
  also now distinguish between empty and nonexistent header fields

- fixes to header fields parsing in several places: header field names
  are case-insensitive, whitespace is not required after a colon,
  obsolete rfc822 syntax allowed whitespace before a colon;
  VBounce: match "Received:" only at the beginning of a line;

- Exporter never really worked in SA, was not enclosed in BEGIN{}


ERROR HANDLING, ROBUSTNESS

- improved error detection and reporting: test status of all system calls
  and I/O operations (or explicitly document where not), and report
  unexpected failures;

- eval calls now check for eval result instead of testing the $@, which
  is not always reliable;

- localized $@ and $! in DESTROY methods to prevent potential calls to eval
  and calls to system routines in code executed from a DESTROY method
  from clobbering global variables $@ and $!;

- Util::helper_app_pipe_open_unix: contain a failing exec with an eval
  to prevent additional cases of process cloning. The exec could fail
  this way when given tainted arguments;

- Util::helper_app_pipe_open_unix: flush stdout and stderr before forking,
  otherwise an error reported by exec (such as 'insecure dependency')
  was lost in a buffer;

- eval-protect an open($fh,'-|') to capture implied fork failures
  due to lack of system resource;

- explicit untainting: combine "use re 'taint'" with untaint_var(),
  avoiding implicit perl untainting and workarounds to prevent it;

- add 'use strict' where missing;

- avoid a bunch of warnings on "Use of uninitialized value"

- clearly report reasons for helper application process failures


OTHER

- more expensive code sections are now instrumented with timing measurements;
  timing report is logged as a debug message by the end of processing,
  and made available to a caller and to 'add_header' directives through
  a TIMING tag;

- total rewrite of URI detection in plain text body;

- pseudoheader "ALL:raw" returns a pristine header section,
  and pseudoheader "ALL" returns a cleaned header section

- many updates to the list of top level domains;

- add 'util_rb_3tld', allowing 3-level TLDs to be listed in URIBLs and
  allowing new 3TLDs to be added from rule updates;

- avoid trusted_networks bog down due to O(n^2) loop with millions of entries;

- preserve order of declared 'add_header' haeder fields;

- allow debug areas to be excluded from debugging,
  e.g.: -D all,norules,noconfig,nodcc

- fixes to Plugin/VBounce.pm, updated VBounce ruleset;

- new module Mail::SpamAssassin::BayesStore::BDB, but is not yet ready for use

- numerous additional and updated self-tests;

- updated documentation, fixed numerous typos and mistakes
  in documentation text and in log messages;

- extensive improvements to development process:
  automated testing through Hudson, improvements to mass-check and rules