Message Sniffer

Full List of Features

We have designed Message Sniffer (SNF) to be a complete anti-spam solution for your email servers. Let us help you reduce the time and effort you spend dealing with spam. Click on the feature headings below to learn more about what SNF can do for you.

	Accuracy		Reliability
	Efficiency		Speed
	Flexibility		Support
	Intelligence

Accuracy

Capture Rates

SNF consistently captures more than 99% of spam on average (calculated from spamtrap processing data, customer reports, telemetry, and lab tests on live messages).

False Positives

SNF has the lowest false positive rates in the industry. SNF scans more than half a billion messages per day while we receive fewer than 300 false positive reports per month!

Rulebase Quality

Our rulebase is managed 24x7x365 by a highly trained, professional staff using the most sophisticated tools in the business. SNF's collaborative learning technologies ensure that as more SNF nodes are activated, accuracy and response time continually improves.

SNF rules are based on a discrete-logic system - they are either in or out. They are not combined in a weighting system. This simple distinction mathematically simplifies the quality control process by eliminating the inherent interactions between rules in a weighted system. The result is that our quality control decisions can be made more accurately and more quickly.
All of our rules undergo a continuous review process where each member of our rule coding staff (the sortmonsters) reviews all active rules for quality several times per day.
All rules are also continuously monitored by robots for performance and any signs of false-positive activity. The SNF engine is able to automatically disable (auto-panic) any new rule that fires against known good senders thus eliminating many false positives and reporting the situation to us immediately for correction.
The SNF network acts as a broad-spectrum virtual spamtrap that automatically sends us samples of new spam sent by known spam sources. (Sampling can be disabled if required for security reasons). Unlike conventional spamtraps which use addresses harvested by blackhats, our virtual spamtrap network captures new spam sent to any address - there is nowhere to hide.

Efficiency

No Extra Parts

There are no extra parts to Message Sniffer. SNF does not depend on external DNS servers or hash databases. This can be very important in large scale filtering environments where DNS lookups for conventional anti-spam tests can quickly overload internal network and DNS infrastructures. This also makes SNF particularly well suited for appliances because it eliminates key points of complexity and performance problems.

No Maintenance

Once Message Sniffer is set up properly, no additional administration is required!

Almost all software updates are delivered through the rulebase updates.
Rulebase updates can be downloaded and installed automatically.
Collaborative filtering systems are maintained, monitored, and upgraded automatically.

Heavy Metal Not Required

All of SNF's components are highly optimized and purpose built. The programs are compiled from efficient C++ code that goes "straight to the metal" to get the most out of the hardware you have. The scanning engine is so efficient that it can be run in real-time against incoming messages during the SMTP session (on compatible platforms). Other anti-spam engines with similar accuracy (such as SpamAssassin) typically require more than 10 times as much CPU power* and simply can't respond fast enough to be used in a real-time environment.

*During real-world tests running SpamAssassin and SNF in parallel on the same postfix box SNF consistently required less than 10% of the CPU required by SpamAssassin while scanning the same message stream! During the same tests SNF's capture rates were slightly higher than SA's while SNF's false positive rates were notably lower!

Working Smart

Using GBUdb (the Good, Bad, Ugly data base), Message Sniffer is able to "truncate" its content scan on between 10% and 50% of real-world messages**. If the source IP of the message has a sufficiently "black" reputation then SNF stops the scanning process immediately saving time and resources without reducing accuracy.

**Based on telemetry received from production systems in the wild. 10% truncation rates are typical of systems that pre-screen messages using filters, connection blocking, graylisting and/or black-lists driven by GBUdb data. It is typical to see 25% truncation rates on systems with partial black-list blocking, and between 30% and 50% truncation rates on systems with no front-end filtering prior to SNF.

Flexibility

Highly Configurable SNF has a comprehensive list of configurable options that allow it to work efficiently in a wide range of applications. SNF remains robust, flexible, and well-behaved as a stripped-down scanning engine running from a flash drive in an email appliance, as a filtering component on a company email server, or as a critical component in a large-scale hosted email services infrastructure. For a full list of configuration options please refer to the SNFServer Configuration documentation.		Command Line Interface Often the simplest way to integrate a new component, like a message scanner, is through scripting and simple efficient utilities. In these environments, SNFClient provides full access to message scanning, status reporting, and IP reputation database features. This has proven to be an effective and efficient way to integrate SNF into many email platforms and filtering tools. See the SNFClient Documentation for details on what SNFClient can do.
Multi-Platform Compatibility From the very beginning we wanted Message Sniffer to deliver top performance on a wide range of platforms, so we made solid cross-platform compatibility a design constraint and stuck to it! The source code for the SNF software strictly avoids complex per-platform switches unless they are absolutely necessary and the underlying libraries for critical functions like networking, threading, and timing are all tightly coded translations to native platform features. As a result, the SNF code base compiles without modification on Windows (using MinGW), Linux, BSD, and OS/X.		TCP (XCI) Interface One of the most powerful features of Message Sniffer is the TCP based XCI (XML Command Interface). The XCI allows programs to use simple text commands in an XML format to access the SNFEngine. In fact, the command line interface (described above) translates command line parameters into XCI requests and inteprets XCI responses. When used in debug mode the SNFClient is a great development tool for exploring the syntax of XCI requests and responses. The XCI interface provides an efficient interface to the SNFServer without the extra overhead of launching the SNFClient utility. It is as close as you can get to implementing SNFServer directly in your own software without actually doing it.
SpamAssassin Plugin (SNF4SA) For other systems that have a SpamAssassin infrastructure in place Message Sniffer can be added to SpamAssassin as a plugin. This provides two additional SA tests each with their own scores. One configurable test provides a score based on the source IP's reputation using GBUdb data. Another configurable test provides a score based on the content scan analysis. In addition this plugin can be configured to make use of the "short-circuit" feature so that in some cases other SA tests can be skipped when Message Sniffer detects spam or malware.		Sendmail / Postfix Milter (SNFMilter) In *nix environments Message Sniffer can be integrated as a milter with either Sendmail or Postfix. This allows Message Sniffer to either reject spam and malware during the SMTP session, quarantine the messages, or pass them on for further processing after injecting headers containing information about the scan. SNFMilter also provides the full SNFMulti engine via XCI so that SNFClient or direct XCI requests can be used to perform additional scans.
Direct Integration / Embedding Of course, there are those who want the highest possible performance and the tightest possible integration. For those folks there are three options: Windows DLL - For the Windows platform, we have a 32bit DLL (source available if needed for 64bit - some assembly required). The DLL provides the calling program with full SNFServer functionality (including XCI) as well as direct access to the scanning functions. This allows your program to pass messages as buffers rather than files for extremely efficient processing. It is fully multi-threaded. This package is in use in large scale proprietary filtering systems and will be released shortly as part of a windows SDK. If you need it sooner please let us know and we will work with you! nix Lib / .SO - For nix platforms, we have source distribution for a shared library (.so) or lib (.a) that provides a C style interface to the SNF engine while also providing the full SNFServer functionality (including XCI). Using this package, the calling program "opens" a rulebase manager and then as many scanning engines as needed (presumably one per thread) attached to that rulebase manager. The library is thread safe and accepts scan requests as buffers. This package is in use in email filtering appliances and will be released soon as part of a source based SDK. If you need it sooner please let us know and we will work with you! SNFMulti C++ Object - For those developing filtering systems in C++, the SNFMulti Engine is available as provided in the source distribution under closed and open source licensing (ask us for details). Using the SNF code as a C++ source library, you can compile the SNF engine components directly into your code and use the existing SNFSource and SNFClient code as guides and examples. We will be releasing an SNF Source SDK when the documentation is ready. The source code itself is available now and we stand ready to assist you.

Intelligence

Plays Well With Others

The active components of the SNF scanning engine adapt to load conditions in order to maintain optimal throughput without starving other processes on the system. In addition, SNF's fully multi-threaded engine is highly optimized to make the most efficient use of the available computing power whether that is a single processor core or a large multi-processor system. Load sensing, lowest queue first job allocation ensures that all available processors are used when needed and only one procssor is used when that will suffice. Natural spiral timing dynamics ensure that polling operations are kept efficient while preventing processing delays.

Real-Time Collaborative Learning

The GBUdb IP reputation system captures real-time data on IP behavior and shares that information with all other SNF nodes in the world. All GBUdb nodes work together and compare notes on message sources by sharing their experiences. Each node maintains its own purpose built database in memory while storing periodic snap-shots for reporting and restart recovery. This provides each SNF node with direct real-time access to IP reputation statistics without performing expensive per-message queries over the network. SNF is also smart about sharing GBUdb data: Each node only talks about the most relevant information, and only asks about IPs it is dealing with locally. As a result the exchanges between GBUdb nodes are short, efficient and powerful.

Unlike most conventional IP reputation systems, GBUdb is not a one-size-fits-all solution! In addition to sharing GBUdb data with the other SNF nodes in the network, each SNF node maintains its own personality and perspective. This allows each SNF node to develop its own localized behaviors. Sources that send nothing but spam to one system are blocked there even while being allowed through on other systems that receive regular legitimate mail from the same source.

Automated Training Tools

The sad truth these days is that most ISPs send out more than 90% spam! Whatever the reasons for this unfortunate fact, IP reputation systems have a difficult time distinguishing between good and bad message sources accurately. All of the messages travelling through these ISPs, no matter what their original source, appear to be coming from the ISP's servers. When you use the connecting IP to evaulate a message from one of these sources then you are actually averaging together all of the legitimate and illegitimate traffic from that ISP -- That is a very noisy (inacurate) signal. SNF includes two automated training tools that help see past this problem and dramatically improve the accuracy of the GBUdb IP reputation system:

Drilldown

The drilldown feature allows SNF to use pattern matches on trusted Recieved: headers to automatically add GBUdb Ignore flags for ISP servers. You simply enter known, trusted reverse-dns data that is added by your system for ISP sources and SNF will follow the trusted chain of Received: headers through the message to locate the original source.

The effect is that the ISP's equipment becomes known as infrastructure and becomes transparent. Then the original message source is finally selected as the source for the message. As a result, GBUdb can accurately score the original IP source for the messages going through that ISP instead of working with averaged statistics for the ISP's servers. Messages coming from bots that send through the ISP are then attributed to the bots while legitimate messages being sent through the ISP by legitimate sources are attributed to those legitimate sources.

Source-Header

Unfortunately, some ISPs don't preserve the Received: header chain. Luckily those ISPs do sometimes add headers that identify the original source for the message. In these cases, SNF has a feature to use these special headers when determining the message source.

In order to prevent forgery, the Source-Header feature is only activated when the correct pattern is identified in the top (trusted) Received: header. The result is that when a message arrives from one of these ISPs the source IP for the message is extracted from the special header and so GBUdb can then train on and score the message using that orignial source.

The Bigger Brain

Message Sniffer is built on a foundation of self-organizing, collaborative learning systems. Each SNF node contributes information about its experiences (statistics, observations, performance data, and partial analysis); and each SNF node benefits from the experiences of the others. The whole is larger than the sum of its parts.

Virtual Spamtrap Network

Each SNF node acts as part of our Virtual Spamtrap Network. Unlike conventional honey-pots and spamtrap networks our VSN does not depend on specific email addresses that must be cultivated and might be avoided or poisoned by blackhats.

Instead, our network carefully identifies and samples new spam content delivered from known spam sources no matter where those messages might be going! Without any specific spamtrap addresses to wash from their delivery lists there is nowhere for the bot-nets to hide. (This feature can be disabled if there are security concerns).

Realtime Behavior Analysis

Each SNF node provides real-time telemetry on the performance of each pattern matching rule and the observed behavior of individual message sources (IPs). This allows us to optimize the rulebase and IP reputation systems in real-time while building high-resolution data sets that can be used to generate new attack-response models and tools to take advantage of that new knowledge.

Realtime Spam-Storm Detection

GBUdb and pattern-rule matching rates can be used to generate extremely accurate "storm-sign" signals for each SNF node. This storm detection system can be used to change local response parameters when a system is under stress and allow us to focus attention on systems that are showing unusual activity in real-time.

However, the SNF "brain" is still even bigger than that -- There are also specialized systems that interact with this shared consciousness by extracting specialized information and injecting new knowledge.

One example of this is our wavefront detection system which can detect larger features in the IP behavior data, cross-reference that with pattern matching information, and detect new spam delivery patterns as they develop. This ability to see "the bigger picture" and use that data to inform the GBUdb and rule production systems improves the larger systems' ability to respond quickly and accurately to new and evolving threats.

In addition to the SNF nodes that our customers are directly familiar with, the SNF system consists of a growing number of collaborating components ranging from new bots that seek out interesting features in delivery behaviors and new patterns in unwanted messages, to improved training systems, instrumentation, and automated tools that help our people to see more clearly, respond more effectively, and plan ahead with greater precision and purpose.

Reliability

Continuous Testing and Refinement

Automated systems monitor SNF installations continously to report unusual behavior, performance metrics, and status information so that we can spot trouble quickly. We have a zero tolerance policy for bugs of any kins, and a culture of continuous improvement that drives us to locate even the most subtle flaws and eliminate them.

We Own It

Every piece of code in Message Sniffer was built in our own labs, heavily tested, and thoroughly reviewed. More than 80% of the critical code in SNF has been in reliable service for years. There are no third party libraries for critical functions such as pattern matching, networking, threading, database operations, or critical timing. This means that we know exactly what all of the code is doing, how it interacts, and if any of it misbehaves - we can fix it.

Built For High Availability

SNFServer has been designed to run on systems that remain active for years without a break. There is no need to restart SNFServer for configuration changes and no need to notify it when a new rulebase file is available. Any kind of configuration change is detected automatically and put into place without distrupting normal operations or slowing down scanning operations.

When a new rulebase file is detected, that file is loaded and validated in a separate thread. Only when the new rulebase is ready is it made available to the scanning engine. Any scans that are in progress continue to use the old rulebase file (and configuration) until they are complete. Any new scans that begin will use the new rulebase file (and configuration). When all of the scanners using the old information have finished then the old information is discarded.

Fail-Safe By Design

When we set out to build Message Sniffer, we realized that we were building a critical piece of infrastructure so we designed safety into the heart of it. Wherever possible errors and exceptions are handled so that there are no negative effects; and if a failure must occur, we do our best to fail safely so that there is as little disruption as possible. Here are just a few examples:

If SNFClient can't connect to its server, then it reports the error and returns a zero result so that the message it was scanning will go through. The result of a failure in this case is more spam and NOT more false positives!
Whenever SNF is rewriting a message in order to inject headers, the new message is first written to a temporary file and then only swapped into place after the operation is complete. This ensures that if a problem occurs the original message file remains intact or at the very least the new version is recoverable for delivery.
If the GBUdb component of SNF cannot contact our systems for an extended period of time, then it will simply continue to operate on its own by responding to and learning from its local environment.

Speed

Message Sniffer (SNF) is the fastest email scanning engine, period. In today's environment where you have to dig through ten or more bad messages for every good message you want, speed counts.

SNF's scanning engine is based on advanced AI research. By using self-organizing cellular automata, SNF's scanning engine is able to scan an entire message at once in a single pass while applying more than 120,000 heuristics (typ). Virtual creatures called "evaluators" swarm over the data and work together to focus the CPU's attention only on the parts of the message that matter.

The result is astonishing speed. Using only generic, single processor hardware SNF is able to reliably process thousands of messages per minute!

A case from our lab:

One of our inbound spamtrap processors does double duty as a performance and reliability test bed for SNF. That system handles sustained rates in excess of 3500 messages per minute and peaks in excess of 4500, all day, every day.

Specs:

OS: Windows 2003 Server Standard Edition
CPU: single 2.3Ghz Intel P6, 4GB Ram.
Storage: Two 34G IBM SCSI drives (one for system & spool, one for mailboxes)
Network: Intel Pro/100 S Server Adapter
Mail Server: IMail 8.15, SNF connected through MINIMI* (lightweight SNF shim)

*MINIMI (Minimal IMail Insert for SNF) available for IMail installations with limited message processing requirements (delete, quarantine and/or inject headers).

Support

We are Committed to You

All subscriptions come with full support for your subscription year. This includes installation guidance and personalized support for tuning and customizing rulebase(s) to fit the needs of your system.

We also offer online documentation to the SNF system and online SNF community of fellow users.

See our support page for access to all of our resources and how to reach us.