News and Updates Archive
2012-11-23 New GBUdb Tool
We have been playing with a new utility that some of you may enjoy.
GBUDB Tool allows you to create a list of IP addresses from your GBUdb snapshots (.gbx files). You can select IPs that are "blacker" or "whiter" than a provided probability figure and confidence figure. It outputs one IP per line, optionally with details about the statistics for the IP. This can be useful for feeding-forward blacklists to block at your firewall or for other research purposes.
Run GBUDBTool without any parameters and it will tell you about its command line options.
2012-06-26 Message Sniffer Rule #5,000,000 Coded!
Message Sniffer Rule #5,000,000 was coded by Andy (Worm Thunder) 20120626.1408 SortMonsters Rock!
2012-02-23 Message Sniffer System Upgrades
Here are some of the system upgrades that we have made recently:
- We have boosted our rulebase production system. New rulebase updates will arrive about 25% faster on average.
- We have optimize Rulebot productivity to respond to a wider range of spam / malware variants automatically.
- We have augmented our QC processes to seek out more potential false positive cases and stop them before they occur.
- We have added additional research channels to help identify more threats more quickly.
Note that over the next few weeks we will be making additional changes to our technical infrastructure. During service windows occurring at times of low-activity there may be short disruptions in SYNC server connections and/or rulebase delivery. We will do our best to avoid these, and those that do occur should go unnoticed.
Your Message Sniffer software installation is designed for high performance and high availability. It will continue to function normally even if we have a disruption during our upgrades, and it will automatically recover from any such disruption without any assistance.
2011-11-01 New Small Business Rates Offered!
We are now offering special pricing for small businesses. We are offering two rates: SMB Rate at $199/instance/year and SOHO Rate at $99/instance/year.
During your 30 day free trial, we will be monitoring your telemetry. We will be monitoring your HAM ratio (average number of good messages per day) that your system processes. Based on the numbers we see, we will notify you via email if you qualify for either of the special rates.
For current customers, you will be notified in your renewal notices if you qualify for these rates.
For questions about these rates, please contact sales at firstname.lastname@example.org.
2011-09-26 SNF Server/Client for *nix Updated - Important Bug Fixed
Tarball snf-server-3.0.13.tar.gz has been posted to the Downloads page.
This distribution contains some minor bug fixes and code improvements bringing the SNFMulti Engine up to 3.0.17.
IMPORTANT: This distribution also contains a "clean" SNFServer/main.cpp that fixes a random crash bug!
The previous distribution snf-server-3.0.12 contained testing code that would intentionally force a crash (seg fault) under specific load conditions. The testing code would make it appear that SNFServer was crashing at random with crashes being more likely under higher load conditions.
The testing code should not have escaped the lab and was not intended for use in production. We have reviewed adn revised our publishing procedures to ensure this does not happen again. This new distribution snf-server-3.0.13 does not contain the testing code.
This bug was not included in Win* distributions - only snf-server-3.0.11.tar.gz and snf-server-3.0.12.tar.gz included the errant testing code.
2011-04-06 4 Millionth Rule!
We have reached our 4 Millionth Rule! -- Our rule bots now have more than 4 Million heuristics available for activation at any moment. When new spam is spotted that matches an old rule, that rule is reactivated automatically.
The vast majority of our rules have been coded by hand over the years by our amazing Rule-Techs (The SortMonsters). These highly trained professionals work around the clock (24x7x365) and consistently produce the most accurate rules available anywhere. They are really a fantastic team and a great bunch of folks to boot. :-)
2011-01-18 CommuniGate Pro Plugin for MS Windows Updated
We've updated the MS Windows version of our Anti Spam / Anti Malware plugin for CommunigGate Pro.
We have rewritten the documentation and distribution files to make the installation process simpler and clearer. We've also updated the main configuration file with CSS and XSL so that you can view a clear, human friendly version of your snf_engine.xml file simply by opening it in your web browser.
For more information, visit the SNF4CGP page in the Documentation section.
To download the SNF4CGP plugin, visit the Downloads page.
2010-11-13 Rulebase Compiler Retuning Completed
Over the past few days we've finished a major re-tuning of our rulebase compiler system. The improved rulebase compiler bots are just a bit smarter and as a result many systems are receiving their updated rulebase files sooner than ever before. This means capturing more spam early on more systems and as a result more accurate data in GBUdb for new bot-nets. A win for everyone.
2010-06-22 GBUdb.com Website Launched!
We have launched the GBUdb.com website: http://www.gbudb.com.
We have also updated the generator for the truncate.gbudb.net list so that the TXT records include a link to the list descriptor at http://www.gbudb.com/truncate/ and the IP address in [square brackets].
Please tell us what you think.
2010-04-29 Opening truncate.gbudb.net
We have been testing a blacklist based on real-time GBUdb data (generated from Message Sniffer).
We have decided to experiment with opening up the blacklist for a wider audience and so as of now you can use truncate.gbudb.net as an ip4r test.
You should get a result of 127.0.0.1 if the IP is well into the truncate range -- That is: truncate.gbudb.net is designed to be ultra-conservative so that it should be safe to reject connections based on the test in most cases. This also means that it won't block everything -- only the worst of the worst. That said, the folks who have been testing it have reported that it did drop a significant amount of traffic from their systems on average.
UPDATE: RFC 5782 states:
"IPv4-based DNSxLs MUST NOT contain an entry for 127.0.0.1."and also states:
"The A record contents conventionally have the value 127.0.0.2"So we will be changing the result code for truncate.gbudb.net to 127.0.0.2 effective immediately.
Please keep us all posted about how it's working for you.
2010-03-30 SNF4SA Upgrade
We have posted two new files to our Downloads page containing an upgrade to our Message Sniffer for Spam Assassin plugin:
The newest version of Message Sniffer for Spam Assassin (SNF4SA) contains minor bug fixes, but most importantly provides support for older implementations of SpamAssassin that do not support dynamic scores from plugins.
When SNF4SA detects a version of SpamAssassin prior to 3.2 it will automatically produce a static score based on reaching the configured threshold. This allows SNF4SA to work automatically in both old and new versions of SpamAssassin to dramatically increase spam filtering performance and accuracy without additional tuning or tweaking.
We implemented this feature because there are some systems out there using older versions of SpamAssassin and the administrators of those systems do not want to upgrade SpamAssassin to the latest version for some reason. Message Sniffer generally runs on these systems without a problem and now so does SNF4SA.
Previously if you were to install SNF4SA on an older version of SpamAssassin it would not work properly and no score would be added when Message Sniffer detected spam. If you have had this experience in the past you should try again with this new version and please let us know.
2010-02-05 Rulebase updates increased by 25%!
After more back-end improvements and some careful analysis we have increased our rulebase update rate by another 25%.
This will mean:
- Less time for new spam to get through between updates
- More accurate IP reputation information against new bots
- Faster removal of troublesome rules (fewer false positives)
2010-02-04 New Proactive False Positive Prevention Initiatives
Unqualified false positive candidates: Through this review process we are able to remove and modify pattern rules that cause occasional low-level false positives that would otherwise not be reported. This system is already allowing us to recode or remove dozens of rules per day to make them more accurate; and to update our rule coding practices and support systems to further improve our accuracy moving forward.
Real-time rule / IP conflict analysis: This system monitors conflicts between IP reputations and pattern rule matches across the entire fleet of Message Sniffer installations in real-time. Any time a pattern match is in disagreement with a source IP's reputation that information is analyzed and pumped through a sophisticated collection of filters and data-mining tools. The resulting analysis is displayed in real-time in our spam-weather center so that our staff can respond immediately (24x365) if there is any sign of a "bad rule".
2010-01-04 Message Sniffer DLL now used in Declude
The Declude folks have announced version 4.10.42. With this version Declude now integrates Message Sniffer via our DLL.
- Improved performance
- Not an external test, so no program must be launched
- Uses the message already in RAM thus saving disk IO
- SNFMulti engine runs inside of the Declude service (one less program / service)
- No XCI calls required to request scans (reduced communications overhead)
- Provides direct access to the GBUdb IP Reputation system for additional scoring options
Here is a link to their announcement as archived on "The Mail Archive".
2010-01-01 New Year's Message Sniffer Promotion
2009-11-21 Message Sniffer Antispam/Antimalware plugin for CommuniGate Pro Beta Released
Today we're releasing version 0.1.0 (a beta) of our spam filter plugin for CommuniGate Pro (CGP). You can find the distributions on our Downloads page.
We've been testing this for a while in the lab and in our spamtrap processing servers. It's very fast and very stable.
More documentation is on it's way -- however each distribution also contains the documentation typical of CGP plugins.
SNF4CGP (CGPSNF) does everything a typical CommuniGate filter plugin does and a bit more. In addition to providing X- headers that can be used with filter rules, CGPSNF can also be configured to take any of these actions (configurable by result code, of course):
Allow - This is the typical CommuniGate plugin response. CGPSNF will provide X- headers as configured. The X- headers can be used to trigger CGP message processing rules.
Bypass - This action bypasses SNF4CGP -- the message has been scanned and logged, but CGP is not provided with headers and no additional action is taken.
Delete - This action tells CGP to discard the message.
Hold - This action takes the message as it was provided by CGP, injects the SNF headers, and then puts that message in a folder of your choice for later processing. This is a great hook to use if you are a service provider and you want to build sophisticated quarantine and/or policy review processes.
Reject - This action tells CGP to reject the message with the provided reason.
CGPSNF can also be configured to add its log entries to the CGP log for easy review -- even if the log is not stored as a file by SNF (use mode='api'). Also, just like SNFServer, the XCI interface is provided so you can use SNFClient for GBUdb manipulation or "out of band" message scanning. The full SNFServer engine is in place whenever the CGPSNF plugin is active.
As always - there is no need to restart SNF after making changes to the configuration -- so you can change these options on the fly as needed.
If you have any questions please let us know.
2009-09-11 SNFMilter 1.0.3 released -- bug fix
Those of you using SNFMilter should upgrade to the latest.
We have fixed a bug which would cause SNFMilter to exit with a SIGSEGV under some conditions -- Specifically the error would occur when mlfi_connect() was called with a NULL host address.
2009-08-30 Postfix with Milter, Out-of-Sync Issue Fixed
This week Postfix stable release 2.6.5 as well as Postfix legacy release 2.5.9, 2.4.13, and 2.3.19 have been posted. These versions fix the Milter out-of-sync problem. If you are using SNFMilter with postfix, you should consider upgrading to one of these version so that you can enable use of the quarantine method.
2009-08-26 Updates for SNFServer and SNFMilter
We have posted the following new *nix distributions for SNFServer and SNFMilter & Windows SNFServer:
These new versions fix a rare memory leak bug that occurs when corrupt rulebase files are presented to the SNF engine. The SNF engine would read and ultimately reject the bad rulebase file but would not release the memory associated with it.
Most systems never saw this bug because their update mechanism would validate the rulebase (.snf) file before swapping it into place.
As a result most folks don't technically _need_ this update--- but it is best if you update to this latest version when you can schedule it in.
Windows users can download the SNFServerV3.0.2-E3.0.11.exe file,
Stop SMTP (to prevent queuing)
Rename SNFServer.exe to SNFServer.exe.bak
Copy SNFServerV3.0.2-E3.0.11.exe over SNFServer.exe
2009-07-29 SNFMilter Released
2009-07-29 Updated Client/Server Distribution for Linux, BSD, and & *nix Systems
We've posted a new version of our Client/Server distribution for Linux, BSD, & other *nix systems. You can find snf-server-3.0.9.tar.gz on our Downloads page.
This update contains a fix for a minor bug in the CodeDweller/Networking code: Under some (rare) circumstances SNFServer would exit with SIGPIPE. The new code includes an appropriate use of MSG_NOSIGNAL or SO_NOSIGPIPE depending on the platform used to build the software.
The SIGPIPE bug does not affect Windows systems. However, a new update to the Windows installer is due relatively soon just to keep all of the versions up to date and to update some documentation for some of the integrated platforms.
This update includes improved control scripts that provide for a special debug mode. The debug mode runs SNFServer with a number of debugging options enabled to capture detailed information about how SNFServer is running. Most folks will never need this ;-)
Other improvements to the source code have also been included.
2009-05-12 SNF4SA - Message Sniffer Anti-Spam Plugin for SpamAssassin Released
We have just released a MUCH improved plugin for SpamAssassin. Our new plugin makes full use of the SpamAssassin Plugin API to provide features like:
- Add weights for specific scan result codes.
- Add (or subtract) additional weight based on IP reputation statistics.
- Optionally skip other tests.
- Inject SNF headers.
The SNF4SA plugin is included in the latest *nix distribution of SNF on our Downloads page.
Also we have packaged the SNF4SA plugin separately for those of you running SpamAssassin on Windows machines -- or if you already have SNF up and running and just want to switch to the latest SpamAssassin plugin.
For more information visit our SNF4SA page.
We look forward to your feedback!
2009-03-18 Server Side Solutions is now offering an eWall + Message Sniffer package
Server Side Solutions, creator of eWall, is now offering an eWall + Message Sniffer package, as well as Message Sniffer subscriptions and renewals. Please visit the SNF section of their website for more information.
2009-03-17 Updated Windows Installer
Our Windows installer (available on the Downloads page) now directly supports the following platforms:
- Imail + Declude
- Imail + MXGuard
- Imail stand alone (using MINIMI)
- MDaemon (as a plugin)
- SmarterMail + Declude
- SmarterMail + MXGuard
It also provides the option to install SNFServer and SNFClient without any particular integration in case you want to do something custom or integrate with a platform that is not yet supported by the installer.
A number of items have been removed from the products page now that they are included in the new installer. Also note that the installer is up to date and now uses the CURL based getRulebase.cmd script.
Note: There are many more Windows based platforms that are compatible with SNF. Some of those install the SNF engine as part of their own installation process. Others require some manual tweaking. Please use the custom / other option from the installer to help with those.
2009-03-17 WIN* SDK (DLL) Prelimary Release
A number of folks have recently asked us about our SDK. Although we are not yet finished with the SDK package (documentation, examples, etc...) the 32 bit DLL itself has been in production on a number of large systems for several years now.
From now on a snapshot of the SDK package will be available from our Downloads page so that folks can download the package, try it out, and give us feedback on how we can improve it.
Yes, for those who are about to ask, we have a 64 bit DLL also (available upon request). Once we've had some more fun testing it we will add it to the SDK package.
2009-03-17 SNF4ASSP Posted
Last year on the prompting of several SNF users we built an ASSP plugin for SNF. We held back posting it to our site because we wanted to see more testing and feedback before making it public. Now we've posted it.
2009-02-02 Announcing ClamAID - Clam AV installer for Windows
We've noticed that folks often have trouble getting Clam AV (the free open source anti-virus scanner) working correctly on their mail servers, so we've created a free product to help solve that: ClamAID (Clam AV Assisted Install Device).
What ClamAID does is collect all of the bits and pieces that make ClamAV work, configure them, install them, and get them running with your email / filtering platform.
So far ClamAID supports IceWarp, Declude/IMail, and Declude/SmarterMail. We will add support for additional platforms as requested (time permitting).
Please take a look, keep us posted on your progress, and tell your friends about ClamAID if it helps you. If you have any questions or run into problems then please let us know.
2008-10-09 SNF Now directly supported in IMGate!
Message Sniffer is now directly supported in Len Conrad's IMGate. IMGate + SNF allows you to move your spam filtering out in front of your mail server improving scalability, stability, and performance.
Here are some links:
2008-07-31 Installers Posted (available in the Products section)
We have re-posted our Windows Installer for the new Client/Server version of SNF 3.0. This installer will help you upgrade from previous versions of SNF if you are using any of:
- IMail + Declude
- IMail + mxGuard
- SmarterMail + Declude
Other combinations are also supported and often detected automatically including raw client/server installations for use on systems we don't know about yet ;-)
This installer is relatively new (but well tested in our lab). Please keep us posted on how it works for you.
If you prefer to get the new SNF and install it yourself:
MDaemon users-- don't forget that we nave an installer for the new MDaemon plugin also!
If you prefer to install the new SNF MDaemon plugin manually:
Linux/BSD/OSX users-- If you haven't heard, the new *nix distribution of SNF has been updated with improved V3 specific instructions and example control & update scripts. Also - if you had trouble compiling SNF before on your PowerPC or 64bit box the latest version includes big/little endian detection and bug fixes. At this time there are no known problems on all of these platforms. As always: Keep us posted please :-)
Everyone should upgrade to the newest version as soon as practical. While we will continue to support version 2 for a time, version 2 of SNF is deprecated. Of course, that's not the only reason to upgrade. SNF Version 3.0 has many improvements that reduce leakage, reduce the chance for false positives, reduce administration costs, and reduce system loads.
- More efficient, fully multi-threaded scanning engine.
- Realtime collaborative IP reputation system.
- Realtime rulebase checking and telemetry (no need to upload logs)
- Realtime system status information in XML format.
- Integrated, customizable rulebase update mechanism.
Some of our site is still under construction (it is large). If you find something under construction that would help you please let us know and we will reschedule that work to get it done more quickly. In the mean time we'll be happy to answer your questions directly.
2008-07-31 2 Millionth Rule!
We have reached our 2 Millionth Rule! -- Our rule bots now have more than 2 Million heuristics available for activation at any moment. When new spam is spotted that matches an old rule, that rule is reactivated automatically.
The vast majority of our rules have been coded by hand over the years by our amazing Rule-Techs (The SortMonsters). These highly trained professionals work around the clock (24x7x365) and consistently produce the most accurate rules available anywhere. They are really a fantastic team and a great bunch of folks to boot. :-)
At present about 122315 rules are typically active at one time.
Our most active rule at the moment was coded some 2062 days ago (has it been that long? Wow!).
Here's to the next 2 million !
The newest version of eWall from Server Side Solutions includes direct support for SNF:
- Simplified installation - just provide your license ID and Authentication string.
- eWall communicates directly with the SNFServer via XCI for speed and efficiency.
- New SNF specific actions and conditions.
- Automated filter generation tools in the "New Agent Wizard".
Here is a link to the announcement: http://forum.sssolutions.net/showthread.php?p=14524
Our rulebase delivery subsystem has been upgraded. The new system supports 10x the previous bandwidth and a minimum of 5x the the number of transactions per second.
The *nix source distribution has been updated to include Version 3 specific install instructions and to correct a minor bug.
2008-06-26 It's official. SNF Version 3.0 is Ready!
Back in Q1 we were sure we'd be ready with the new SNF after nearly a year of testing on both large and small systems. What a surprise!
After publishing the first release candidate we went from version 1-5 to version 2-27 at a breathtaking pace!
Thank you to everyone who has tested, poked, prodded, and twisted the new SNF -- not to mention keeping up with all of those updates during the final phase of testing. I can't imagine getting to this point without your patience, trust, attention to detail, and persistence! Bravo!
Without further fanfare: Today the latest release candidate becomes the official production release of Message Sniffer (SNF) Version 3.0.
- Minor updates to readme files.
- Changed the build / version information and recompiled.
- Removed redundant comments from the configuration file.
We have been bug free for more than 2 months with several hundred systems using the new engine.
You can download the latest distributions from the Downloads page.
You may also notice that we've published our new web site! There are a few bits of documentation still under construction here and there, but we're well on our way to filling those in along with a stream of continues improvements and additions based on our work with you!
Once again, Thanks to everyone for a fantastic job!Thanks for all of your support, comments, and efforts! As always we're hear to help. Now, onward to the next upgrade... always work to do ;-)
2008-06-20 ARM Research Labs Launches New Website!
2008-06-10 Final RC before Version 3 (fingers crossed)
The latest SNF distributions have just been posted:
- SNFMulti engine 2-9rc 25
- SNFClient 2-9rc 7
This release is a performance update, no new bugs in many weeks now.
Here is a snip from the change log:
20080524 - Version V2-9rc2.25.7
- Optimized networking library for additional speed & stability by moving receive buffer allocation from heap to stack (automatic).
- Optimized timing parameters in SNFClient for improved speed. Polling dealys are now reduced to 10ms from 30ms.
- Removed speed-bug in SNFClient, 100ms guard time between retries was always executed after an attempt (even a successful attempt). The guard time is now condition and only fires on unsuccessful attempts.
- Updated XCI server logic to ensure non-blocking sockets for clients in all socket implementations.
PS: ****** We expect to begin wide testing of two new pieces of software soon: Windows Installers for the MDaemon plugin and Command Line versions of the new SNF. Stay tuned!
2008-04-25 New version: Engine 24, MDPlugin 6
This release is an upgrade more than a bug fix. Replace your SNFServer.exe or snfmdplugin.dll as appropriate.
No changes have been made to the configuration file.
This version improves memory management in the SNF Engine for improved performance, improves the header injection mechanism for improved reliability, and improves logging for IP scans done with the MDaemon plugin.
As usual you can get the latest distributions here:
Here is an excerpt from the change log (this time from the MDaemon plugin change log since it contains all changes from the last version):
20080424 - Version V2-9rc6.24.6
- Refactored snfScanData.clear() to reduce heap work and fragments.
- Added mutex to scanMessageFile() entry point just in case some app attempts to put multiple threads through a single engine handler. scanMessage() is already protected and fully wraped by the new scanMessageFile() mutex.
- Added non-specific runtime exception handling to XHDR injection code.
- Added 2 retries w/ 300ms delay to remove original message in XHDR inject code. If remove fails after 3 attempts the injector throws.
- Added 2 retries w/ 300ms delay to rename temp file to msg in XHDR inject code. If rename fails after 3 attempts the injector throws.
- Added IPTest logging.
2008-04-16 New Version: Engine 23 - fix for network bug on some win* systems.
This update fixes a bug that effects some Win* systems.
Please replace your SNFServer or snfmdplugin.dll and your SNFClient.
You can always get the latest distribution here:
Here is a snippet from the change log:
20080416 - Version V2-9rc2.23.6
Fixed bug where SNCY open() would fail on some Win* platforms with
WSAEINVAL instead of the standard EINPROGRESS or EALREADY which were expected.
Also added WSAEWOULDBLOCK to cover other "ambiguities" in windows sockets
implementations. InProgress() on Win* now test for any of:
WSAEINPROGRESS, WSAEALREADY, WSAEWOULDBLOCK, WSAEINVAL
2008-04-13 New Version SYNC bug fix! SNFEngine 22
It seems that in our last update we introduced a bug that effects SYNC operations for some customers - particularly those with longer network transit times to our servers.
The bug could cause SYNC sessions to fail either consistently or intermittently depending on the transit time. If SYNC sessions consistently fail then the new UpdateReady feature will not fire. GBUdb collaboration is also diminished with failed SYNC sessions.
A new version has been posted that solves this problem. Please upgrade your SNFServer or snfmdplugin.dll files from the new distributions as soon as possible to avoid missing telemetry, UpdateReady information, and GBUdb collaboration traffic.
We have also included a new build of the SNFClient program since it uses the same networking library. Although it is unlikely this bug would cause a problem with the SNFClient program you should update to the newest build to be sure.
No configuration changes are necessary with this update.
Here is a description of the changes in this newest distribution:
20080413 - Version V2-9rc2.22.6
- Fixed bug in TCPHost.open() where [WSA]EALREADY was not counted as a version of [WSA]EINPROGRESS. This would cause open() to throw an unnecessary exception when a socket open() required extra time.
20080413 - Version V2-9rc2.21.6
- Extended timeout for SYNC session open() to the full session length. This way if a session takes a long time to open it still has a shot at success.
2008-04-11 Latest RC release SNFMulti 20, SNFServer 2, SNFClient 6, MDaemon 5
The newest RC release has been posted in the usual location:
There are NO changes to the configuration file. You need only replace SNFServer.exe and SNFClient.exe and/or snfmdplugin.dll (as appropriate to your system). This release resolves all known bugs / tweaks.
Snippets from the change log:
20080411 - Version V2-9rc2.20.6
- Adjusted snfNETmgr to use non-blocking open in SYNC sessions. Open timeout is 1/3 of the session timeout. Session timeout is 2 * Session pacing. Open polling uses golden spiral delay from 10ms to 340ms.
20080410 - Version V2-9rc2.19.6
- Adjusted XCI manager to use new snfCFGPacket paradigm in checkCFG().
- Adjusted snf_RulebaseHandler::addRulePanic() to use MyMutex and eliminated the AutoPanicMutex and waiting scheme.
- Refactored scanMessage() to use a ScopeMutex() rather than lock()/unlock().
- Refactored scanMessage() to use MyCFGPacket.isRulePanic() test.
- Redesigned snfCFGPacket handling to automate grab() / drop() functions.
- Fixed lock-up bug: Redesigned AutoPanic posting and checking mechanisms to eliminate potential dead-lock condition. Under some conditions a precisely timed auto-panic posting could cause the RulebaseHandler mutex and the AutoPanicMutex to become intertwined leading to a cascading deadlock. When this occurred all XCI processing threads and eventually the XCI listener thread would become blocked waiting to get the current configuration.
20080409 - Version V2-9rc2.18.6
- Enhanced XCI exception handling and logging to provide additional detail.
- Added code to explicitely check for zero length files in scanMessagFile().Previously a zero length file would cause the CBFR module of the filter chain to throw an invalid buffer exception. Now if the message file is empty scanMessageFile() will throw a FileError stating FileEmpty!.
20080407 - Version V2-9rc2.17.6
- Enhanced exception reporting in snfXCImrg
2008-04-05 New Version Engine: 16, Client 6
The newest distributions for the Command Line (Std Test Package), MDaemon plugin, and Source have been posted. You can find them
here as always:
This update is important because it includes a bug fix to the networking library. This update also includes some tweaks intended to improve network performance under heavy traffic conditions.
Please upgrade to the new DLL, SNFClient, and SNFServer. There is no need to change your configuration file ;-)
A snippet from the change log:
20080405 - SNFServer V2-9rc2.16.6
- Reduced safety limits on status reports to 100K for status reports and 100K for samples. Previous values were 10M. Most full sessions from the busiest systems are < 50K total.
- Recoded sendDataTimeout() to break uploads into 512 byte chunks and insert delays only when a chunk is fragmented. This methodology improves reliability on Win* systems without any significant penalty on systems that don't need socket sends() to be in smaller chunks.
- Fixed TCPClient::transmit() and TCPHost::transmit() bug where returned byte count might be -1. Now returned byte counts can only be 0 or more.
2008-03-27 More progress SNF2-9 SNFMulti engine goes to version 15
Here is a new beta/rc release. The changes are internal and should solve a bug that happens on a handfull of systems. You should upgrade so that you're on the latest version. If you're not having trouble you can put off upgrading until some later time (but not too long please).
Please find the newest release here:
The long version:
This release goes further to eliminate the "hanging" bug on those few systems that see it. One case should be solved completely by this revision and possibly all cases (we shall see).
This release also eliminates a minor bug (not worth a revision) that was in the previous release. It seems I failed to remove a line of code that forces the Debug mode in SNFServer before pushing out the last release -- so SNFServer in the previous release would be in debug mode no matter what -- thus creating extra monitor data on the screen (if not run as a service or piped to /dev/null).
The big change in this release is in the snfNETmgr module that handles SYNC operations (GBUdb & Telemetry). The previous version used blocking IO and a separate thread (TCPWatchdog) to kill off connections that lasted too long. The new version uses non-blocking IO and has been refactored to consolidate some of it's communications routines.
In one case the "hanging" bug presented as a loss of telemetry without errors or exceptions. It appeared from the debug data that the snfNETmgr thread had gotten stuck in an IO call and that even though the TCPWatchdog thread had killed the connection the function call never returned.
The theory supporting this change is that after some number of these TCPWatchdog events the TCP stack might become unstable on some systems and cause this kind of behavior. The new non-blocking methodology eliminates this possibility.
It is possible, if the above theory is true in any way, that this change will solve the other "hanging" cases also -- In those cases the snfXCImgr thread appears to get stuck while attempting to accept() another client. If the TCPWatchdog methodology used before did cause instability in some way to cause this, then this presentation of the "hanging" bug should also disappear.
If the new revision doesn't solve the XCI related "hanging" bug then the addition of very detailed status tracking in the snfXCImgr module should help us see more clearly where to look.
Excerpts from the change log:
20080326 - SNFServer V2-9rc2.15.4
- Refactored snfNETmgr::sync() to consolidate non-blocking io routines.
- Added detailed thread status data to XCI listener thread.
- Fixed minor bug in main (not changing revision), Debug flag for internal use was left on in the last build cycle. It is commented out now.
20080325 - SNFServer V2-9rc2.14.4
- Updated snfNETmgr with comprehensive thread status data.
- Refactored snfNETmgr::sync() to check a Timeout, removed TCPWatchdog.
20080325 - SNFServer V2-9rc2.13.4
- Upgraded TCPWatcher code to use new threading features (type, status).
2008-03-25 New 2-9rc Versions Posted Client - v2, Server - v4, MDaemon DLL v4, Engine v12
Three new releases today:
You can find them here as usual:
Some items of note:
There is an obscure, intermittent bug in SNFServer that has been reported on a handfull of systems. The vast majority of systems run normally (including our lab systems) for hundreds of days -- only stopping when we tell them to.
The bug manifests as either:
SNFServer stops listening for requests.
SNFServer stops sending telemetry.
In both of the above cases there are no errors in the logs, no core dumps, no unhandled exceptions, no corruption of any kind--
Only two cases show any kind of pattern so far:
- In one case SNFServer will stop sending telemetry after approximately 1 day give or take a few hours. Scanning and all other functions continue normally.
- In another case SNFServer appears to stop accepting requests via XCI after approximately one week give or take a few days.
- Other cases are completely random (est fewer than 5 cases total).
If you come across this scenario please let us know all of the data you can about the situation and then please run your SNFServer in debug mode (see next item) to help us track down this critter.
SNFServer now has a debug mode. If "debug" or "Debug" are found in the path to the SNFServer.exe then debug mode is turned. Most commonly to run SNFServer in debug mode rename it to SNFDebugServer.exe.
When in debug mode SNFServer will make a thread status report to the console once per second along with the usual activity information. The idea is to pipe all of this information to a log file so that when the above bug occurs we can record the status of all of the active threads at that time, before, and after.
/SNF/SNFServer.exe /SNF/snf_engine.xml > debuglog
---- now some good news ---
There is a new feature in SNFServer. When there is a new rulebase file available SNFServer can call a user-defined script to retrieve the new rulebase file. We've also provided that script and set up the default settings to call it ;-)
The script name is getRulebase.cmd on Win* systems and simply getRulebase on *nix systems. Please read the readme files and check your configuration files to make sure that the script is setup properly for your system. Wget and Gzip utilities are included in all of the above distributions for your convenience.
If the script fails to replace the rulebase file then it will be retried after 3 minutes (default). Retries will continue until the script is successful.
-- The feature can be turned off.
-- SNFServer still produces an UpdateReady.txt file so if you want to
continue using that methodolgy nothing will break -- though you should
turn off the
-- If you write your own script or want to launch the script some other way (such as calling cmd or start with special options) then you can do that -- but be careful! The update-script engine runs in it's own thread and makes a system() call when triggered. If your script fails to return then the update-script thread will be stuck waiting for it to return. Remember: what you put into the call= attribute will be passed to system() when the feature is triggered. The best way to do anything special there is to write a script that does what you want and have the update-script mechanism call that script. It's probably not a good idea to put a lot of special switches and "other craziness" in the call= attribute --- If you need them, put them in your script and keep the call= simple :-)
2008-03-20 MDaemon Plugin SNFv2-9rc4.11.4 Posted
I have just posted the latest beta (release candidate) MDaemon plugin.
You can find the latest betas here:
This distribution includes an automated update utility that is triggered from the SNFServer engine running in the plugin. When a newer rulebase file is available an UpdateAvailable.txt file is created in the SNF directory. The getRulebase.cmd script can be scheduled to run once per minute. When the UpdateAvailable.txt file is present the script will download, validate, and install the latest rulebase file. Before using the getRulebase.cmd script be sure to edit the top of it to establish the correct working directory, license ID, and authentication string.
Engine improvements and updates to the SNFClient utility are also included...
A few excerpts from the change log:
20080319 - Version SNF2-9rc4.11
- Added IPScan on-off to snfmdplugin.xml. This allows users to turn off the IPScan feature without editing the Plugins.dat file as was previously required. The feature can now be enabled or disabled at will by editing the configuration file.
- Added Configuration editor options to snfmdplugin.xml. Previously the built- in configuration function was hard coded to start notepad with the config file. Now the system() call made by the ConfigFunc() can be edited in the configuration file. The configuration file name can be appended to the command optionally. The default is still to start notepad and append the configuration file path so that it is loaded automatically. It is hoped that GUI based configuration editors for the SNF plugin will be built by third parties and in the mean time folks can now configure their favorite XML file editor to modify their SNF plugin configuration.
- Modified API use fixed shutdown bug - The plugin used to initialize the SNF scanning engine when the DLL was loaded and would shut it down when the DLL was unloaded. Now the Startup and Shutdown functions in the MDaemon plugin API. This ensures that the engine components are started and shutdown in the proper sequence.
- Included new SNFEngine core (excerpts from that change log included).
20080318 - SNF2-9rc1.11.exe Consolidated several mods/fixes
- Corrected scan error logging bug. Was posting <s/> now posts <e/>.
- Updated scan error logging to be more uniform with non-scan errors.
- Enhanced error and exception reporting in SNFMulti.cpp scanMessageFile().
- Enhanced exception handling in networking module. All exceptions now throw descriptive runtime_error exceptions.
2008-03-07 Version 2-9rc1.8.2 Release Candidate (Std Test Package) Released
This is the first release candidate for what will become version 3 this quarter!
You can find the latest updates here as they arrive:
Over the next few days we will be updating the MDaemon DLL with the new engine and a new feature or two. Then we will update the source distribution for *nix & OEM systems. Then we will be launching two SDKs -- one is a .SO for *nix systems and the other is a DLL for Win* systems. Along the way we will be launching a new web site with documentation for the new version. Then later this year (Q2 - Q3 perhaps) we'll be launching DNS based IP reputation services.
For now -- back to this moment in time and the new SNFServer and SNFClient release. There are extensive updates to both the client and server programs. Be sure to go through the readme files if you are upgrading.
Also - if you are upgrading you will want to update your snf_engine.xml file to cover the new features. (GHASP! What if I forget to do that?!!) -- If you don't get to it right away then your existing snf_engine.xml file will work fine... but do get the update process on your to-do list so you can take advantage of the new features and improved default settings.
Here is a chunk of the change log to show you what is new since version 2-9b1.5.1:
20080306 - SNF2-9rc1.8.exe (FIRST RELEASE CANDIDATE for VERSION 3!)
- Added Drilldown Header Directive Functions - When the candidate source IP comes from a header matching a drilldown directive the IP is marked "Ignore" in GBUdb and the candidate is no longer eligible to be the source for that message. This allows SNF to follow the trusted chain of devices (by IP) down to the actual source of the message. It is handy for ignoring net blocks because it can match partial IPs but it is designed to allow SNF to learn it's way through the servers at large ISPs so that the original source for each message can be evaluated directly.
- Added Source Header Directive Functions - This feature allows SNF to acquire the source IP for a message from a specific header rather than searching through the Received headers in the message. This is useful when the original source for a message is not represented in Received headers. For example: Hotmail places the originating source IP in a special header and does not provide a Received header for that IP. This feature is protected from abuse by a "Context" feature which only activates the source header directive when specific content is found in a specific received header. Using the above example, this feature can be configured so that a Hotmail source header would only be read if the top Received header contained "hotmail.com [" indicating that the ptr lookup for the header matched the hotmail domain. Note: When a source is pulled from a header directive that source is put into a synthetic Received header and injected into the scanning stream (not the message) as the first Received header.
- Added forced source IP to XCI - It is now possible to "inject" or "force" the source IP for any message by providing that IP in the XCI request or directly in a scan...() function call. This allows the calling application to provide the source IP for a message ahead of any Received headers that might be in the message. This is useful when the calling application knows the original source IP for the message but that IP is not represented in the Received headers and it is not desireable to use the Source Header Directive mechanism.
- Added forced source IP mode to SNFClient - It is now possible to call the
SNFClient utility with an IP4Address using the syntax:
The -source mode of SNFClient exercises the forced source IP feature in the XCI (see above)
- Added Status Report features to SNFClient and XCI - It is now possible to
request the latest status.second, status.minute, or status.hour data via
the XCI and SNFClient. The syntax for requesting a status report using the
In addition to providing status reports the SNFClient in this mode will return a nonzero value (usually 99) if it is unable to get a status report from SNFServer. This feature can be used to verify that SNFServer is up and responding. If SNFServer is OK then the result code returned is 0.
- Added result codes to SNFClient - test and XCI IP test functions - The XCI engine has been upgraded to provide the range value for the IP under test as well as the symbolic result code associated with that range. This allows the -test function to provide results that are consistent with the GBUdb configuration without additional processing: For example, if the IP falls in the Caution range then the Caution result code will be returned just as if a message had been scanned with the same IP and no pattern match occurred. The same is true for Truncate and Black range hits.
- Added Timestamp and Command Line Parameter data to SNFClient.exe.err - When an error occurs with SNFClient that may not appear in the SNFServer logs an entry is appended to the SNFClient.exe.err file. That in itself is not new. The new feature is that the entries added to the SNFClient.exe.err file now include timestamp and command line data to aid in debugging.
- Added BIG-ENDIAN Conversion - When the SNFServer program is compiled on a
system that uses a BIG-ENDIAN processor (such as a power-mac) the rulebase
load process now includes a routine to convert the token matrix from it's
native LITTLE-ENDIAN format to a BIG-ENDIAN format. This solves a bug where
Power-Mac (and presumably other BIG-ENDIAN systems) could compile and run
the SNF* software but were unable to capture spam because the token matrix
in the rulebase file was misinterpreted.
Note: The BIG-ENDIAN Conversion feature is still considered experimental because it has not yet been thoroughly tested.
- Updated the Configuration Log to include all of the current configuration features and to improve it's readability.
20080207 - SNF2-9b1.7.exe
- SYNC Timeout now 2x SYNC Schedule
- SNFServer now produces an UpdateReady.txt file when the UTC timestamp on the SYNC server is newer than the UTC timestamp of the active rulebase. It is presumed that a suitable update script or program will run periodically and download a fresh rulebase file if the UpdateReady.txt file is present. The update script should remove the UpdateReady.txt file when it completes a successful download of the new rulebase file.
- Added available rulebase UTC in status reports <udate utc.../>
- Added Automatic path fixup for ending / or \
- Added option to use local time in log rotation <rotation localtime='no'/> The default is still utc.
2008-03-05 MX Uptime adds Fully Integrated SNF!
The newest version of Message Sniffer is now an integral component of MX Uptime's plugin for MailEnable. Anyone wishing to use SNF only needs to enter their license ID and authentication string, then check the box :-) Screenshot for integration.
2007-10-17 Message Sniffer Version 2-9b1.5 Wide Beta
This version is considered stable for production environments. The next release will include some minor feature additions and improved default settings (thus our long wait while we monitor installed systems and refine our data). If there are no problems with the next release then we will freeze all features and create the official production release in Q1.
2007-10-05 Message Sniffer Version 2-9b1.1 Wide Beta
At your earliest convenience, please follow the following link to read about the newest version of Message Sniffer which has just been released for wide beta testing.
The command line client/server version is available now. It is a drop-in replacement for folks who have been running the current command line version (2-3.5) with a persistent instance on Winx platforms. The version in the posted distribution file requires a P3 or better.
MDaemon and *nix (source) distributions will be coming shortly.
This new engine has been in testing on a number of production systems from the very big to the very small for quite some time. There are no known bugs at this time. None the less, please be careful :-) and read carefully!
A GREAT BIG THANK-YOU goes out to the folks who have helped us alpha test and refine this version over the previous months and weeks through scores of alpha iterations! We really appreciate the help.
Over the next few days/weeks we will be adding documentation and answering questions to help folks explore and make the most use of the new features. We will also be looking for any last minute tweaks that might be needed; and we will be building a list of any additional features and/or refinements that come to light so we can get them into the production release, or at the very least the .1 that will follow.
As always, your comments, questions, and feedback will help guide our efforts. The value of the discussions we share both privately and on this list cannot be overstated.
Thanks for your patience, trust, and participation!
2007-06-26 Rulebase Compiler Upgrade
We have just completed an upgrade to the rulebase compiler software. The new version is 20-50% more efficient - as a result, updates will be produced a bit more quickly and consistently.
There is no need to make any changes on your systems.
2007-02-05 SurgeMail adds a feature to call Message Sniffer.
2007-01-05 Rulebase Update Rate Increased by 16.6%.
Now that the new delivery server is in place and functioning properly, we have re-tuned the rulebase compilers to deliver updated rulebase files 16.6% more quickly on average.
This means that you will receive updated rules more frequently throughout the day and as a result you should also see less leakage and quicker responses to new mutations of spam.
2007-01-05 FTP Access to Rulebases Being Deprecated
Note that FTP downloads of SNF rulebases is deprecated. If you are using FTP to download your rulebase files you should switch to using http w/ gzip as soon as practical.
FTP access to SNF rulebase files will continue for a time but support may be removed without notice in the future. It's a safe bet that FTP access for SNF rulebase files will remain functional through the end of this month however.
2007-01-03 Upgrading SNF rulebase delivery servers
Over the next few days we will be upgrading the SNF rulebase delivery servers. If all goes well - nobody will notice except that downloads will become faster and (likely) more frequent.
On the off- chance that this might effect you or that something unpredicted might happen we am making this announcement :-)
Expect to see the IP change for http://www.sortmonster.net. If you have closed your firewall to outgoing traffic then this may effect you - you will need to make a new "hole". Please also note that the authentication realm has changed on our delivery servers. The old realm was "SortMonster". The new realm is "SNF".
It is possible that you may miss one or more updates during the transition. We will do what we can to minimize this possibility.
2006-10-23 Version 2-3.5 Release -- Faster Engine
The plan was to hold off until the next major release, however in light of recent increases in spam traffic we are pushing out a new version with our faster engine included. All other upgrades are will wait for the major release ;-)
The scanning engine upgrade results in a 2x speed increase that hopefully will help with the higher volumes we are seeing now. Version 2-3.5 also rolls up 2-3.2i1 which included the timing and file locking upgrades. Version 2-3.5 can be found in our wiki, in the Distributions area.
2006-06-19 Rulebase Pacing Updated
We have just reduced our rulebase update pacing from 150 minutes to 120 minutes. This means rulebase updates will now arrive 20% faster.
If you are using a scheduled task to retrieve your updates, please adjust your timing appropriately (about every 60 minutes should be reasonable provided your script checks for an updated file before performing the download).
If you are triggering your updates based on the arrival of our update notification messages then you need not take any additional action - the change will be automatic.
2006-06-07 WeightGate Available
This program is distributed AS-IS, with no warranty of any kind. You are welcome to use this program on your own systems or those that you directly support. Please do not redistribute this program except as noted above, however feel free to recommend this program to others if you wish and direct them to our wiki where they can download it for themselves. Thanks!
This program is most commonly used to control the activation of external test programs from within Declude based on the weight that has been calculated thus far for a given message.
2006-05-12 Compressed Log Files Now Accepted!
We are now able to accept compressed log files. Compressed log files can either be in zip or gzip form. For complete guidelines on submitting compressed log files please visit the Log Files Technical Details page in our wiki.
2006-04-26 Update Notification FROM Address Changing
We are changing the rulebase update notification's FROM address to:
You shouldn't have to take any action on this, but just in case you have any filtering or whitelists set up you should change them.
2006-04-05 SNFRV2R3i1 - ready for testing...
The first in a long line of coming updates has been posted for those brave souls who wish to test or may have use for the changes. You are looking for the file: snfrv2r3i1-EngineOnly.zip You can find the current interim release, Version 2-3.2i1 (Engine Only) on the following page:
Be aware - this distribution only contains the SNF executable for Winx systems and source code for BSD, Linux, & other GNU (g++) capable *nix boxen.
BTW: The source now contains a handy make file for a change ;-) Also, we are now using all gnu compilers for testing and development. We previously used Code Warrior for Winx and g++ for *nix. We now use minGW (Code::Blocks) on Winx and g++ on Linux (RHES3) for testing and development.
This release addresses two key areas that are related:
* The timing functions have been replaced using a new cross-platform Timing Module. If you are curious or interested in cross-platform development in C++ you can find more info on that module here:
The Timing Module simplifies a number of critical timing features in SNF and made it simple to correct some unusual timing and control conditions that would occur on some systems under very specific circumstances -- these were odd, difficult to reproduce bugs which by all indications have been solved now. That is to say, those that I have been able to reproduce have been repaired and tested -- those that I had strong theories about have also been addressed and are very likely solved -- I will know more after your reports ;-)
* During the refit I also did some additional testing and tuning to improve SNF's command-line scanner performance under heavy loads, in transition (dynamic loads) and during live configuration changes (switching from persistent mode to peer-server mode and back), and on systems with multiple processors and higher speed processors (it still works great on slower boxen too). Comparative testing in the lab shows some noticeable improvements in throughput and resilience - YMMV, I look forward to your reports.
There is NO NEED to upgrade to this version at this time unless you are looking for a tiny bit more speed or solving one of the previous timing and/or control bugs (reload, rotate & stop commands for example, or the "Adjusted Persistence Race Condition" on some bsd or linux boxes -- these are now fixed and tested as far as we can test them).
The other reason you might try the new version is if you would like to help us (and others who are cautious of early adoption) by testing the latest and greatest.
Folks using the MDaemon plugin are not effected by these updates since they apply almost exclusively to command line coordination code -- the plugin has no such code ;-) Folks using other plugins, DLLs, SNFMulti or other custom configurations are also not effected by these updates.
Please keep us posted on your results.
2006-03-10 New RuleBot F002 Online
This rulebot captures and creates geocities web links from the "chatty" campaigns. This is largely a time saver for us humans... we will focus our attention more on abstracts for these campaigns now that F002 will be capturing the raw links. Rules from F002 will produce a 60 result code (Ungrouped).
2006-03-06 New Rulebase Compliers Online
Work has been completed to upgrade the rulebase compiler bots.They are now significantly more efficient. As a result, you will be seeing updates more frequently. Previous lag was between 40-120 minutes. Current lag (sustained) is < 5 minutes. More timely updates should equate to lower spam leakage for new spam.
2006-03-06 New Rulebot F001 is Online
Rulebot F001 creates IP rules for sources that consistently failmany tests while also reaching the cleanest of our spamtraps. The rules will appear in group 63. Expect an increase in your rulebase size while F001 catches up with current spamtrap data.
2006-02-15 Updated Expired Rulebase Cleanup Code
New code has been added to the server that delivers rulebase files. The code removes any rulebase file where the license is disabled. This was a task tha was done manually, but is now automated.
2005-12-21 Sniffer Engine Updates
Increased Updates per Day: Standard rulebase delivery pacing has been changed from 200 to 150. This means that, on average, rulebase files will be recompiled every 2.5 hours or so. This timing will be variable based on system loads etc, but it is a significant improvement. We have sped up our rulebase delivery process by 267%!! (from 3.6 updates/day to 9.6 updates/day).
Improved IP Rule Coding: A new piece of optimization code was added to drop any Received IP rule that has 0 rule strength and is more than 30 days old. This will help to reduce false positives caused by IP rules that "hang on" after the infection/problem with the source is fixed. It also reduces the compiler workload a bit by reducing the core rulebase size.
2005-11-02 Rule Strength Analysis Upgrades
The Rule Strength Analysis upgrade makes the rule strength calculation more sensitive to the recent activity of any given rule. This will also cause rule fitness decisions to be more competitive so that the most effective rules will be more strongly selected over time.
This will improve SNFs performance in two ways:
1. Rulebase files will be smaller and will require less bandwidth to download and to load during operation. There will also be a measurable increase in scanning speed (though this is already measured in small numbers of milliseconds on most systems).
2. The smaller, more efficient files can be compiled and delivered more quickly which will allow us to increase the rate at which we deliver updates.
2005-08-11 Message Sniffer and Assert! Used to Halt New Bagle Variant
Assert! and Message Sniffer rules were quickly updated upon news that a Bagle variant outbreak had reached very high numbers according to AppRiver, a leading anti-spam service provider. Within hours customers were protected from the rapidly spreading variant, contained in compressed .RAR and .ZIP files. Though Message Sniffer primarily focuses on anti-spam content filtering, the engine can also help prevent email-borne virus outbreaks.
2005-08-01 ARM Research Releases "Assert! Message Sniffer for SMTP and Exchange"
Assert! version 1.1 encapsulates the raw power of the Message Sniffer engine with an easy, intuitive interface for Exchange or the IIS SMTP Service. Assert! is a powerful anti-spam tool that does not require a bloated feature set or period of tuning to be effective. Assert! includes a one-year subscription to the Message Sniffer spam database, which is automatically updated multiple times daily for pinpoint accuracy.
2005-07-01 AppRiver LLC and MicroNeil Research Corporation form ARM Research Labs (ARM).
With the goal of exploring ideas and raw data as a means for producing internet-based technology products, a leading anti-spam service provider AppRiver LLC and software research innovator Microneil Research Corporation have joined efforts as ARM Research Labs LLC. ARM is dedicated to strengthening the world of computing online innovations in areas such as application development, security services and other web-based operations.