Saving Our Bacon: Snort Security Holes and Strategies for Safe Network Monitoringby Robert G. Byrnes, coauthor of Linux Security Cookbook
In April, a CERT advisory announced the discovery of two separate buffer-overflow vulnerabilities in Snort, a popular security-monitoring tool used for detecting suspicious network activities. This development was disturbing and ironic: system administrators install and run programs like Snort to improve security, and don't often consider the possibility that the tools themselves might be attacked and exploited to create entirely new security holes. It's therefore important to understand precisely what happened here, especially since the same mechanisms used against Snort could threaten other security tools.
In this article, I will review the attacks that have been launched against Snort in the past, as well as the recent (and more serious) buffer overflows. In each case, I'll discuss the ways Snort developers have responded to the attacks, and the strategies system administrators can take to minimize the risks. Furthermore, I'll show that Snort's vulnerabilities extend to other security-monitoring tools, implying that we need to be careful when we use them, as well. Finally, I'll summarize techniques to do just that: secure monitoring.
Snort is a network intrusion detection system, or NIDS. It sniffs packets from the network, and analyzes them using a database of pattern-matching rules. These rules identify a wide range of suspicious or hostile activities, such as misuse of network protocols, attempted buffer overflows, denial-of-service attacks, port scans, and so on. When such activities are detected, Snort can be configured to log the network trace data for more detailed examination, and to send alerts so that system administrators will notice.
Since the main purpose of Snort is detection, a primary goal of attackers is evasion. If attacks can be structured so that they are overlooked by Snort, then system administrators will be left with a false sense of security -- arguably a worse situation than if Snort had not been used at all.
The evolution of Snort has been driven largely by the endless arms race between the attackers and the defenders. The open source distribution of Snort allows the bad guys to study the detection mechanisms and devise ways to bypass them. The Snort developers, aided by reports from the user community and examination of often widely available attack tools, then respond by enhancing the algorithms, and the cycle continues. The history of innovation on both sides has been fascinating and instructive.
When new malicious network activities are discovered,
pattern-matching rules are added to Snort's database.
These rules are stateless,
meaning that only one packet is examined at a time.
Snort also supports a variety of preprocessors
that track multiple packets,
and hence are stateful.
The modular design of Snort allows preprocessors
to be easily added or modified, and
more complicated evasion attempts
have been countered by improvements in the preprocessors.
The increasing focus on preprocessors for Snort
is reminiscent of the expansion of Linux firewall technology
Attackers frequently exploit IP fragmentation for evasion.
For example, the
port-scanning tool provides an option
to use tiny, fragmented packets
that break headers for higher-level protocols
into pieces during transmission.
frag2 preprocessor reassembles the packets,
so the headers (and the payload) can be scanned all at once.
Similarly, many aspects of TCP can be abused by attackers, for example, by retransmitting packets with slight variations,
in the hope that they will be incorrectly handled by Snort.
stream4 preprocessor catches these protocol anomalies
and ensures that Snort will interpret the packets
as a destination host would.
stream4 reorders packets and reassembles entire TCP sessions,
which allows other preprocessors to easily access data as it
would have been reliably delivered by TCP.
These operations make
one of the most important preprocessors
in Snort's arsenal.
A favorite technique for attackers
to defeat simple pattern-matching rules
is restructuring network data in many different,
but equivalent forms.
A variety of Snort preprocessors respond
by converting specific protocols
to standardized, canonical formats
for more reliable matching
by other rules and preprocessors.
rewrites the URLs used by HTTP.
reassembles RPC data
that has been split into record fragments in various ways.
Attackers have also developed tools like
to generate polymorphic shell code for use in buffer overflows,
and the experimental
attempts to detect these;
this is an area of active research.
Some stealth strategies are based on timing.
nmap can do very slow port scans,
in the hope that probes separated by long intervals
will not be noticed.
portscan preprocessor detects these
by performing a statistical analysis of the traffic, and
the newer, experimental
offer potentially improved detection.
There is really only one good defense
against these evasion techniques
for system administrators:
keep your Snort installation up to date.
Install new rules frequently: fortunately, this is easy,
since the rules database can be downloaded separately
from the Snort web site.
Upgrade Snort frequently too,
in order to obtain the latest preprocessors,
and be sure that the desired preprocessors
are enabled in the
snort.conf configuration file.
Denial-of-service attacks can launched directly against Snort. These attacks are designed to disrupt the NIDS operation, and are in some sense complementary to evasion tactics, since they are certainly easy to detect. A barrage of traffic will at least contribute to the load on whatever machine is running Snort and will consume system resources, which makes this form of attack potentially more serious. Snort is lightweight and efficient, and can therefore tolerate high traffic rates, but other NIDS programs can be overwhelmed, and some have reportedly even crashed when subjected to these assaults.
Attack tools like
have been developed to construct random noise packets based on Snort rules.
These packets are designed
to match the patterns in the rules,
and hence produce a flood of bogus alerts.
The goal is to conceal some other genuine attack
within an avalanche of false alarms.
A cunning plan.
Snort retaliates by using the
the random noise packets can be readily identified (and ignored)
because they do not belong to a valid TCP session.
This is a consequence of the stateless nature of the Snort rules
used by the noise generators.
System administrators can best defend against denial-of-service attacks by running Snort on a separate, dedicated monitoring machine. Any extra system resources that are consumed by Snort during a denial-of-service attack will therefore not affect network services running on other machines.
If you are being bothered by stick-style attacks,
-z est option
to limit alerts to only known, established TCP connections.
must be enabled for this mode to work.
The worst kind of attack to fear is one that allows evildoers to hijack the operation of Snort itself. Unfortunately, two specific buffer-overflow vulnerabilities have recently been discovered that show this is possible.
The security holes occurred in separate preprocessors:
An attacker could exploit a flaw
in Snort's computation of the size of packets,
and consequently overflow a buffer,
causing stack corruption.
This problem was found in March 2003,
and fixed by Snort 1.9.1.
By using carefully crafted TCP sequence numbers,
an attacker could cause the overflow of an integer
used to check for buffer overruns,
leading to heap corruption.
This was exposed in April 2003,
and repaired in the latest Snort 2.0 release.
Both of these preprocessors are enabled by default,
and as we have seen,
stream4 is critical
for the stateful operation of other preprocessors,
as well as for protection against denial-of-service attacks.
You really wouldn't want to run Snort
The impact of these security holes is severe: either can be used to cause Snort to crash, or (even worse) to insert malicious code where it will be executed by Snort.
The good news is that our knowledge of these vulnerabilities and subsequent fixes are the result of security audits, and Snort is an even more valuable tool if it is ironclad. The modular design of preprocessors encourages incorporation of new code, which is an important advantage for Snort, but the latest flaws demonstrate the need for ongoing security audits, especially for any preprocessors that are enabled by default for production use.
The most obvious defensive strategy for system administrators is to keep your Snort installation up to date, and to monitor security advisories so you can apply patches for any newly discovered holes. Most system administrators are already doing this for external network services -- the key here is recognize that tools like Snort are also vulnerable, and hence need attention too.
Next, don't run Snort as root! The CERT advisory says ...
"Both vulnerabilities allow remote attackers to execute arbitrary code with the privileges of the user running Snort, typically root."
... but there is no reason to accept this risk.
Although Snort (like all network-sniffing programs) requires
superuser access to put interfaces into promiscuous mode (so they will act as wiretaps), Snort can and should run as an alternate user (and group, if appropriate)
after initialization is complete.
to specify an unprivileged user and group,
chosen to minimize risk by ensuring
that only Snort's own log files are writable.
An even better strategy is to run Snort in a
Create a directory with all of the files that Snort needs,
remembering shared libraries and so on.
(Some experimentation will probably be necessary,
but it is worth the effort.)
Then specify the directory with Snort's
chroot jail works best
when Snort runs as an unprivileged user -- this isn't an excuse to continue running Snort as root!
Finally, run Snort on a dedicated, untrusted machine. This will help to contain the damage if Snort is ever compromised, especially if the machine contains little or no state, and can be easily rebuilt.
Buffer-overflow vulnerabilities are, unfortunately, still common, as shown by even a casual examination of recent CERT advisories, and the techniques used against Snort aren't especially original or innovative. We often speak loosely of network sniffing programs (like Snort) as "listening" to traffic, so it's easy to conclude that Snort's recently discovered security holes are just another case of an attack on a network service. We've seen it all before, right?
In contrast to network servers like
sendmail, BIND, and so forth,
Snort does not accept connections or receive datagrams
from client programs.
Instead, Snort passively
observes and analyzes network traffic,
and from a security standpoint
it is actually similar to other monitoring tools like
or web log analyzers.
The network traffic plays the same role as a log file.
The easiest way to see this is to separate the packet-capture and traffic-analysis functions that are built into Snort. Although Snort is often run in a configuration that does both, this is not the only way to use the program, nor is it necessarily the most secure mode of operation.
Packet capture can be performed
by a variety of network-sniffing programs other than Snort,
This is relatively safe,
as long as the network data is not really analyzed,
but is simply dumped in "raw" form to trace files,
for example, using the command:
tcpdump -w tracefile ...
These tools all use the
and hence the trace files share a common format,
which is very convenient.
Traffic analysis can be done later,
possibly using a different program,
or on a different machine.
tracefile saved previously
could be analyzed by Snort
using the command:
snort -r tracefile ...
Traffic analysis is considerably more dangerous than simple packet capture, because (as demonstrated by the Snort buffer-overflow vulnerabilities), malformed network data can cause program operations to go awry.
Network-sniffing programs like Snort are often run on machines with unconfigured interfaces, and some system administrators even use modified network cables with severed transmit lines to ensure that only passive observation can occur. While these stealth techniques undoubtedly offer some degree of protection to the monitoring system as a whole, the CERT advisory for Snort warns that:
"... it is not necessary for the attacker[s] to know the IP address of the Snort device they wish to attack; merely sending malicious traffic where it can be observed by an affected Snort sensor is sufficient to exploit these vulnerabilities."
This is somewhat misleading, however -- the danger arises not from the observation but rather from the subsequent analysis of the traffic, and that could happen elsewhere.
The portability of network trace files amplifies the risks. Traffic analysis is often performed on trusted machines, deep within the perimeter protected by external firewalls. Trace files can be sent by email, and public collections are available on web sites such as the Honeynet Project, which offers periodic challenges for study of attacks observed in the wild, or the Internet Traffic Archive, which is more generic, and unfortunately, hasn't been updated recently.
Although Snort performs more complex traffic analysis
than most other network-monitoring tools,
and hence offers more opportunities for attack,
it's important to recognize that
the other tools are also vulnerable.
Ethereal, for example, can perform TCP stream reassembly,
tcpdump's simpler packet printing code
is aware of many protocols.
Are buffer overflow bugs lurking in this code?
other programs that analyze derived network data indirectly
could also be vulnerable.
The earlier analogy between network traffic and log files
offers a clue about how this could happen.
Could an attacker induce a network server
sendmail or BIND
to emit a syslog message
that would be crafted
when it later analyzed the system log?
could a web server be tricked
into recording an entry in its log file
that would subsequently confuse a web-log analyzer?
These kinds of attacks would be difficult
because they involve multiple interacting programs,
but they are certainly not impossible,
and they are conceptually similar
to the recently discovered Snort security holes.
When we recognize the threat of attacks on network monitoring tools,
and indeed any other tool that analyzes network data (even indirectly),
it is clear that we need to pay more attention
to writing and using these tools securely,
as we do for network servers.
Snort's buffer overflow security holes
were discovered during security audits -- similar audits for other network-sniffing programs
tcpdump and Ethereal would be welcome news,
even (or especially)
if they lead to the discovery of other vulnerabilities.
Data from untrusted sources (that is, the network)
must be viewed with suspicion by security tools,
and subjected to sanity checks before use.
Programs need to verify the assumptions
(known as invariants)
that are critical for correct operation,
that is, packet headers don't lie about sizes,
integers won't overflow,
URLs aren't malformed,
printf-style format strings
contain no stray "%" characters, etc.
These invariants are different for each program,
so derived data that is considered sanitized by one program
must still be checked by other programs to be safe.
Perl provides a wonderful facility for tracking untrusted data:
which is written almost entirely in Perl,
would benefit from use of Perl's
to track tainted data read from log files.
logwatch filter scripts
are analogous to Snort's preprocessor modules,
and they deserve a similar defensive posture.
Users of security tools and system administrators can protect themselves first by recognizing vulnerable applications. As we have seen, the most important principle is: follow the data. If it can be traced back to an untrusted source, take steps to mitigate the risk.
The best way to minimize the potential impact of decisions
based on maliciously crafted data
is to analyze the data in a restricted environment.
This can be a dedicated, untrusted machine,
At the very least, avoid running security tools as root --
choose an unprivileged user (and group, if appropriate)
that is only allowed minimal access to the data being analyzed,
and nothing else. The data often contains sensitive information
and requires protection from the general population anyway.
Separating the collection of data
from subsequent analysis steps
can make it easier to set up these safety nets.
Finally, be wary of automated processes that are triggered by decisions based on untrusted network data, especially if the actions require privileged access. For example, Snort provides experimental "flexible response" features (fortunately disabled by default) for killing connections, and a Perl script for dynamically modifying firewall rules, both based on traffic analysis. Imagine the havoc that could ensue if an attacker managed to subvert or gain control of these actions!
Monitoring can be done securely, if proper attention is devoted to the safe use of our available security tools. By recognizing the vulnerabilities and taking steps to run applications in a restricted environment, we can derive the maximum benefit and enhance security, without worrying (too much) that our tools will be hijacked and used against us.
Robert G. Byrnes is a software engineer at Curl Corporation, and has worked in the fields of networking, telecommunications, distributed computing, financial technology, and condensed matter physics.
O'Reilly & Associates will soon release (June 2003) Linux Security Cookbook.
Chapter 9, "Testing and Monitoring (Sample Recipes)", is available free online.
For more information, or to order the book, click here.
Return to the Linux DevCenter.
Copyright © 2009 O'Reilly Media, Inc.