To: users, dev, announce Subject: ANNOUNCE: Apache SpamAssassin 3.4.1-rc2 available Release Notes -- Apache SpamAssassin -- Version 3.4.1-rc2 Introduction ------------ Apache SpamAssassin 3.4.1 represents more than a year of development and nearly 500 tweaks, changes, upgrades and bug fixes over the previous release. Highlights include: Improved automation to help combat spammers that are abusing new top level domains; Tweaks to the SPF support to block more spoofed emails; Increased character set normalization to make rules easier to develop, block more international spam & stop spammers from using alternate character sets to bypass tests; Continued refinement to the native IPv6 support; and Improved Bayesian classification with better debugging and attachment hashing. Notable features: ================= Bug 7115: Adding SHA digests of MIME parts as Bayes tokens allows bayes to see non-textual content - added configurability rewritten Node::_normalize improved tokenization of UTF-8 -encoded or normalized text in the Bayes plugin New configuration options ------------------------- Added flag 'noawl' to the 'tflags' configuration option. parse_dkim_uris ( 0 | 1 ) (default: 0) If this option is set to 1 and the message contains DKIM headers, the headers will be parsed for URIs to process alongside URIs found in the body with some rules and moduels (ex. URIDNSBL) perl_version (Introduced in 3.4.2) This will be replaced with the version -->> THIS NEEDS TO BE FIXED in Conf.pm, WE ARE AT 3.4.1 changed implementation, may produce different result in some cases: normalize_charset ( 0 | 1) (default: 0) Whether to decode non- UTF-8 and non-ASCII textual parts and recode them to UTF-8 before the text is given over to rules processing. The character set used for attempted decoding is primarily based on a declared character set in a Content-Type header, but if the decoding attempt fails a module Encode::Detect::Detector is consulted (if available) to provide a guess based on the actual text, and decoding is re-attempted. Even if the option is enabled no unnecessary decoding and re-encoding work is done when possible (like with an all-ASCII text with a US-ASCII or extended ASCII character set declaration, e.g. UTF-8 or ISO-8859-nn or Windows-nnnn). Unicode support in old versions of perl or in a core module Encode is likely to be buggy in places, so if the normalize_charset function is enabled it is advised to stick to more recent versions of perl (preferably 5.12 or later). The module Encode::Detect::Detector is optional, when necessary it will be used if it is available. option dns_server can now specify a link-local IPv6 address, e.g.: dns_server [fe80::1%lo0]:53 new option: bayes_token_sources (default: header visible invisible uri) Controls which sources in a mail message can contribute tokens (e.g. words, phrases, etc.) to a Bayes classifier. The argument is a space-separated list of keywords: header, visible, invisible, uri, mimepart), each of which may be prefixed by a no to indicate its exclusion. Additionally two reserved keywords are allowed: all and none (or: noall). The list of keywords is processed sequentially: a keyword all adds all available keywords to a set being built, a none or noall clears the set, other non-negated keywords are added to the set, and negated keywords are removed from the set. Keywords are case-insensitive. The default set is: header visible invisible uri, which is equivalent for example to: All NoMIMEpart. The reason why mimepart is not currently in a default set is that it is a newer source (introduced with SpamAssassin version 3.4.1) and not much experience has yet been gathered regarding its usefulness. See also option "bayes_ignore_header" for a fine-grained control on individual header fields under the umbrella of a more general keyword header here. Keywords imply the following data sources: header - tokens collected from a message header section visible - words from visible text (plain or HTML) in a message body invisible - hidden/invisible text in HTML parts of a message body uri - URIs collected from a message body mimepart - digests (hashes) of all MIME parts (textual or non- textual) of a message, computed after Base64 and quoted-printable decoding, suffixed by their Content-Type all - adds all the above keywords to the set being assembled none or noall - removes all keywords from the set The "bayes_token_sources" directive may appear multiple times, its keywords are interpreted sequentially, adding or removing items from the final set as they appear in their order in "bayes_token_sources" directive(s). new option: dkim_minimum_key_bits n (default: 1024) The smallest size of a signing key (in bits) for a valid signature to be considered for whitelisting. Additionally, the eval function check_dkim_valid() will return false on short keys when called with explicitly listed domains, and the eval function check_dkim_valid_author_sig() will return false on short keys (regardless of its arguments). Setting the option to 0 disables a key size check. Note that the option has no effect when the eval function check_dkim_valid() is called with no arguments (like in a rule DKIM_VALID). A mere presence of some valid signature on a message has no reputational value (without being associated with a particular domain), regardless of its key size - anyone can prepend its own signature on a copy of some third party mail and re-send it, which makes it no more trustworthy than without such signature. This is also a reason for a rule DKIM_VALID to have a near-zero score. change: check_rbl_from_domain This checks all the from addrs domain names as an alternate to check_rbl_from_host. As of v3.4.1, it has been improved to include a subtest for a specific octet. new template tags: _SENDERDOMAIN_ a domain name of the envelope sender address, lowercased _AUTHORDOMAIN_ a domain name of the author address (the From header field), lowercased; note that RFC 5322 allows a mail message to have multiple authors - currently only the domain name of the first email address is returned INTERNAL: new methods in Mail::SpamAssassin::PerMsgStatus : $pms->get_names_of_tests_hit_with_scores_hash After a mail message has been checked, this method can be called. It will return a pointer to a hash for rule & score pairs for all the symbolic test names and individual scores of the tests which were triggered by the mail. $pms->get_names_of_tests_hit_with_scores After a mail message has been checked, this method can be called. It will return a comma-separated string of rule=score pairs for all the symbolic test names and individual scores of the tests which were triggered by the mail. New plugins ----------- New plugin (optional): # loadplugin Mail::SpamAssassin::Plugin::TxRep # loadplugin Mail::SpamAssassin::Plugin::PDFInfo ??? URILocalBL.pm ??? Rule updates ------------ Many rules were added or modified, or their score adjusted. Some of these are (in no particular order): ADMITS_SPAM, AXB_HELO_HOME_UN, AXB_XRCVD_EXCH_UUCP, BANG_GUAR, BAYES_999, CANT_SEE_AD, CN_B2B, CN_B2B_SPAMMER, DX_TEXT, DX_TEXT_02, Doctor Oz, END_FUTURE_EMAILS, FILLFORM, FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_MANY_TO, FROM_MISSP_REPLYTO, FSL_FAKE_GMAIL_RCVD, GAPPY_, FSL_HELO_BARE_IP_*, FSL_NEW_HELO_USER, HEADER_FROM_DIFFERENT_DOMAINS, HELO_LH_HOME, HEXHASH, HEXHASH_WORD, HTML_OFF_PAGE, LONG_HEX_URI, FUZZY_CLICK_HERE, LOTSA_MONEY, MSGID_NOFQDN[12], NORMAL_HTTP_TO_IP, NUM_FREE, PDS_FROM_2_EMAILS, PHP malware/phish, PUMPDUMP, RAND_HEADER, RCVD_ILLEGAL_IP, STYLE_GIBBERISH, SYSADMIN, TVD_FUZZY_SECURITIES FP, TVD_GET_STOCK, TO_IN_SUBJ, TO_NO_BRTKS_MSFT, UC_GIBBERISH_OBFU, URIBL_DBL_ABUSE_REDIR, URIBL_DBL_SPAM, URI_GOOGLE_PROXY, URI_IP_UNSUB, URI_OPTOUT_3LD, URI_OPTOUT_USME, URI_TRY_USME, VANITY, __DATE_SPACEY, __BOUNCE_RPATH_NULL, __FORGED_URL_DOM_*, __FSL_LINK_AWS_S3_WEB_LOOSE, __HAS_OFFICE1214_IN_MAILER, __HEXHASHWORD_S2EU, __LONG_HEX_URI, __RAND_HEADER, __SUBJECT_UTF8_B_ENCODED, unsubscribe URI to IP addr., advance_fee, lotsa_money, exploratory tagged-URI, pumpdump, optout, moving money rules (very short 419 fraud spams), new phrase rules, PDFinfo, protect some test rules with can(perl_min_version_5010000), test rules to detect SPF queries that produce error results, various unsubscribe rules, freshen and extend phishing rules, added missing eval:check_uri_host_in_* rules, check for references to compromised WordPress sites, other wordpress rules, some Cyrillic and Hebrew obfuscations that were overlooked, avoid Japanese-language false-positives, added 20_freemail_mailcom_domains.cf Some rules were removed or disabled, either because of ineffectiveness, or duplication with other rules, or due to false positives. Some of these are (in no particular order): DNS_FROM_AHBL_RHSBL, DOS_FAKE_SQUIRREL, FSL_MISSP_REPLYTO, KHOP_SPAMDB_SUBJ, MSGID_MULTIPLE_AT, SMF_FM_FORGED_REPLYTO, SUBJECT_UNNEEDED_ENCODING, URIBL_DBL_REDIR, XPRIO_RPATH_NULL, defunct AHBL rules, obsoleted FSL rules from 50_scores.cf, obsoleted rules in 00_FVGT_File001.cf, perl-5.8-hostile rule, removed duplicate domains in 20_freemail_domains.cf Other updates ------------- Documentation was updated or enhanced. Project's testing and evaluation hosts and tools running in the ASF infrastructure were updated. A list of top-level domains in registrar boundaries was updated several times (cw, sx, club, com.us, util_rb_2tld, ...). TLD updating process was improved, tests to account for new TLDs and changes were updated, TLD update in build/README was clarified for SA releases, RFC 2606: invalid TLD used in testing was changed to '.invalid' . Improvements ------------ Bug 7150: Allow scoped IP address in the dns_server config option Util::TinyRedis: allow a scoped / link-local IP address specification (avoid current limitation in IO::Socket::IP [rt.cpan.org #89608]) SPF max DNS terms was raised to 15 to accomodate for eBay SPF records Bug 7136: added has_check_for_spf_errors and if can() encapsulation Bug 7128: DCC plugin now uses IO::Socket::INET6 instead of IO::Socket::IP Bug 7099: Adding tags SENDERDOMAIN and AUTHORDOMAIN Bug 7068: added rule and code to counts Unicode entities Bug 7052: moved module Net::DNS::Nameserver to optional since it is just used in make test clean up on httpd.conf minor debugging improvement in Plugin::TextCat Plugin/AskDNS: additional debug logging Bug 7107: added "perl_min_version_5010000" for preprocessor conditionals Cleaned up documentation and removed rule name parameter that was not needed on the rule more informative dns debugging output added new install docs to MANIFEST improvements for disabled plugins Optimizations ------------- writing speed of large temporary files was improved by using a larger buffer and avoiding PerlIO - MS::PerMsgStatus::create_fulltext_tmpfile() unnecessary copying was avoided when reading from a temporary file in SA::Message::Node (small optimization) changed fillfactor for postgres bayes/awl tables to optimize for updates a small hotspot in DnsResolver.pm was optimized use faster utf8::encode instead of Encode::encode_utf8 disabled synchronous commit for Postgres Bayes store Notable bug fixes ----------------- Adjusted for Yahoo! using subnet 238.0.0./8 in Received headers. Bug 6751: certain character sets can use alternate characters for a period and bypass DNSBL checks Bug 7153: prevent leaking of messages to stderr in URILocalBL.pm Bug 7143: use eval instead of regex to fix MakeMaker version Bug 7148: small getopt.c change added a workaround to Node::_normalize for an Encode::decode taint laundering bug [rt.cpan.org #84879] Bug 7141: Bayes truncates ('skip') long tokens on bytes, should it count characters instead? Bug 7140: fixed DKIM/SPF Insecure dependency in require Bug 7130: Bayes tokenization mangles/chops many UTF-8 words with accented, Cyrillic etc. letters - inappropriately assuming ISO-8859 encoding Bug 7130: disable TOKENIZE_LONG_8BIT_SEQS_AS_TUPLES, seems redundant and useless with TOKENIZE_LONG_8BIT_SEQS_AS_UTF8_CHARS, e.g. turns each Cyrillic letter of longer words into an individual token Bug 7133: Revisiting Bug 4046 - HTML::Parser: Parsing of undecoded UTF-8 will give garbage when decoding entities fixed missing case for permerror in From SPF Bug 7136: modified 25_spf.t and reverted reversion in SpamAssassin.pm from previous rc1 work Bug 7135: Bayes tokenizer 'arbitrarily' breaks multibyte CJK UTF-8 characters into digrams instead of breaking on UTF-8 character boundaries Bug 7126: Incorrect character set detections by normalize_charset Bug 7125: MIME parsing of nested messages must not treat parts like delivery-status or disposition-notification as message/rfc822 Bug 6953: spamd: could not create IO::Socket::INET6 socket on [::]:783: Address already in use Bug 7106: failed IPv6 socket creation blocks creating a good IPv4 socket Bug 7124: DKIM: RFC 6376 - Signers MUST use RSA keys of at least 1024 bits Bug 7120: Perl Critic exemption Bug 7119: Perl::Critic: ControlStructures::ProhibitMutatingListFunctions reverted critic recommendations to fix undef warning, Removed undef returns for perlcritic test Bug 5399: fix MS::Util::parse_content_type, dots are allowed in Content-Type (a fix to Bug 5399 was too strict) fixed SA::Util::qp_decode for compliance with RFC 2045 (trailing whitespace must be deleted before decoding) Bug 7063: removing sawampersand Bug 7111: sa-update: wrong exit code with --checkonly (does not find new versions) Bug 7030: BayesStore/Redis.pm: authentication doesn't work with Redis 2.6 and earlier Bug 7103: bad wget option causes first fetch of third-party rules channel to fail fixed uribl matching on email addresses with commas after them Bug 6919: Added 'dedicated' to list of static IP indicators for RDNS_DYNAMIC Fixed POD error caused by trailing whitespace hacked PHP URI tuning added askdns to known debug facilities expansion of replace tags for more characters avoid a perl 5.21 warning: Negative repeat count does nothing added more UTF-8 Unicode obfuscation variants removed non AV/filter headers Set headers which may provide inappropriate cues to the Bayesian classifier Plugin/HeaderEval: header field names are case-insensitive Bug 7074, sa-update: improved error reporting of a failed spawned process db_id not initialized, || -> ||= renamed __freemail_hdr_replyto to __smf_freemail_hdr_replyto avoiding name collision changed bayes_auto_learn_threshold_nonspam -1.0 MS::Plugin::AskDNS - avoid warning on undef in eq when a DNS response has no answer section Bug 7079: hide the Geo::IP warning Bug 7078: Mail::Spamassassin::Message::Node::header() error - normalize line endings in header, not just in body Bug 7060: allow excluding domains instead of individual hosts avoid a warning: Use of uninitialized value $pgm in concatenation Plugin/DCC.pm, line 915 Bug 7070: added rbl_timeout_min so that t_min for rbl_timeout applies even without a zone Bug 7065: debug mode breaks Bayes but only if DBM storage is used added code for check_for_ascii_text_illegal in MIMEEval and added test rule to sandbox added Cyrillic and Armenian glyphs in UTF-8 encoding to single-letter replace tags Bug 7034: Redis.pm leaks file descriptors when preforking - avoid creating a circular data structure through a closure allow an "=" char in a redis password added verbose to sync to sa2 zones server added URILocalBL.pm plugin to trunk for testing, updating MANIFEST and v341.pre file as well as optional dependencies with Net::CIDR::Lite and Geo::IP fix DNS resolving with Net::DNS 0.76 Changes in Spamhaus DBL DNSBL return codes as per http://www.spamhaus.org/news/article/713/ Fixing issues with extract_to_rsync_dir Having issues with this sandbox rule failing make test TEST_FILES="t/basic_lint.t t/basic_lint_without_sandbox.t t/basic_meta.t" fixed escaping where perl called from bash using bash variables for tick_zone_serial fixed the interpreter to reference /bin/bash instead of /usr/bin/bash Fixing the masses Makefile for pgapack for linux on new spamassassin-vm centos box Bug 7052: Fix for Net::DNS::Nameserver dependency on CentOS systems fix to install v341.pre file Bug 7050: fixed _DATE_ template tag by use of an anonymous sub, calling Util::time_to_rfc822_date() explicitly without any argument fixed newline collapse harming excessive whitespace rules added max_connections=100 as a safety feature fixed $self added get_names_of_tests_hit_with_scores_hash, get_names_of_tests_hit_with_scores functions to PMS along with trivial fixing of triggered being misspelled. uridnsbl_skip_domain vk.com (the russian facebook) fixed wrong plugin in IF Bug 7032: added tflag for noawl If a subrule is in an if block, ensure it appears in an else block to avoid breaking dependent rules. Fix some rules depending on subrules in if blocks in other sandboxes so they don't break if the conditional check suppresses that subrule. Bug 6994: small change for systems with ACLs in testing fixed SQLBasedAddrList re-learning frequently seen domains on ns1.msedge.net added windows-1251 to likely FP list Bug 7024: check_rbl_from_host/check_rbl_from_domain/check_rbl_envfrom did not support the subtest functionality. Fixed and removed has_check_rbl_from_domain as pointless now. Bug 7018: fixed misspelling on Razor configuration item Bug 7005: sa_compile.t test failures with MacPorts' perl - safe quoting use Config to get path when non-standard sitebin is set Bug 7015: fixed untaint var bug Bug 7013: added a small fix for bayes_auto_learn_on not working with BAYES_999 Bug 7000: dnsbl_subtests.t hangs on Windows Bug 7008: fixed CPAN Parsing added eval for testing a quoted printable ratio for spaminess fixed SA version check Bug 7004: Test suite fails when using FreeBSD's 'script' utility Downloading and availability ---------------------------- Downloads are available from: http://spamassassin.apache.org/downloads.cgi md5sum of archive files: 06ce92812b84bd51f20bc90fa931933c Mail-SpamAssassin-3.4.1-rc1.tar.bz2 5cc08804e32adeb104f0ef9b68de8d8d Mail-SpamAssassin-3.4.1-rc1.tar.gz c3cc867edbf875d157e8a871b73838a6 Mail-SpamAssassin-3.4.1-rc1.zip cafc1a8b3a870e1c5634d39df99f37f7 Mail-SpamAssassin-rules-3.4.1-rc1.r1645877.tgz sha1sum of archive files: 9a26266720114d907596a078671e10e14025ec1d Mail-SpamAssassin-3.4.1-rc1.tar.bz2 1029b7da3e279455ff2e8ea9619b0eb9222a484a Mail-SpamAssassin-3.4.1-rc1.tar.gz 0fee42eb54bec29fd817082d31cc4749a81e0b77 Mail-SpamAssassin-3.4.1-rc1.zip d63d73515445b15980a3155ff8004fc069527d93 Mail-SpamAssassin-rules-3.4.1-rc1.r1645877.tgz Note that the *-rules-*.tar.gz files are only necessary if you cannot, or do not wish to, run "sa-update" after install to download the latest fresh rules. See the INSTALL and UPGRADE files in the distribution for important installation notes. GPG Verification Procedure -------------------------- The release files also have a .asc accompanying them. The file serves as an external GPG signature for the given release file. The signing key is available via the wwwkeys.pgp.net key server, as well as http://www.apache.org/dist/spamassassin/KEYS The key information is: pub 4096R/F7D39814 2009-12-02 Key fingerprint = D809 9BC7 9E17 D7E4 9BC2 1E31 FDE5 2F40 F7D3 9814 uid SpamAssassin Project Management Committee uid SpamAssassin Signing Key (Code Signing Key, replacement for 1024D/265FA05B) sub 4096R/7B3265A5 2009-12-02 To verify a release file, download the file with the accompanying .asc file and run the following commands: gpg -v --keyserver wwwkeys.pgp.net --recv-key F7D39814 gpg --verify Mail-SpamAssassin-3.4.0.tar.bz2.asc gpg --fingerprint F7D39814 Then verify that the key matches the signature. Note that older versions of gnupg may not be able to complete the steps above. Specifically, GnuPG v1.0.6, 1.0.7 & 1.2.6 failed while v1.4.11 worked flawlessly. See http://www.apache.org/info/verification.html for more information on verifying Apache releases. About Apache SpamAssassin ------------------------- Apache SpamAssassin is a mature, widely-deployed open source project that serves as a mail filter to identify spam. SpamAssassin uses a variety of mechanisms including mail header and text analysis, Bayesian filtering, DNS blocklists, and collaborative filtering databases. In addition, Apache SpamAssassin has a modular architecture that allows other technologies to be quickly incorporated as an addition or as a replacement for existing methods. Apache SpamAssassin typically runs on a server, classifies and labels spam before it reaches your mailbox, while allowing other components of a mail system to act on its results. Most of the Apache SpamAssassin is written in Perl, with heavily traversed code paths carefully optimized. Benefits are portability, robustness and facilitated maintenance. It can run on a wide variety of POSIX platforms. The server and the Perl library feels at home on Unix and Linux platforms, and reportedly also works on MS Windows systems under ActivePerl. For more information, visit http://spamassassin.apache.org/ About The Apache Software Foundation ------------------------------------ Established in 1999, The Apache Software Foundation provides organizational, legal, and financial support for more than 100 freely-available, collaboratively-developed Open Source projects. The pragmatic Apache License enables individual and commercial users to easily deploy Apache software; the Foundation's intellectual property framework limits the legal exposure of its 2,500+ contributors. For more information, visit http://www.apache.org/