Two years ago, almost to the day, O'Reilly Network published my first article, Introducing ModSecurity. ModSecurity was stable and useful before the article went out, but it was not widely known. The publication of the article marked a new phase in the life of ModSecurity, introducing it to a much wider audience. As I write the second article, I can't help but feel another phase is about to start. I feel we are entering the phase of maturity.
ModSecurity 1.9 was released in early November 2005, more than a year after the previous stable release, version 1.8. The delay between the two releases was much longer than I had anticipated. Looking back, I really should have released 1.9 back in April, but it so happened that I skipped that release and continued adding new features. This resulted in a release that contains double the features and more than a 40 percent increase of the source code size.
This article describes the most important new features in ModSecurity 1.9. This is somewhat difficult to do, because there are so many of them, but I have decided to group the enhancements into three major areas:
Arguably, the rule engine is the part of ModSecurity that received most of the attention in the 1.9 release. Most of these improvements went toward making the rule engine capable enough to support third-party rule databases. Although I had decided to delay the "official" ModSecurity rule database until infrastructure to share rules between multiple web application firewalls exists (for more details, have a look at the Portable Web Application Firewall Rule Format project), others (for example, Michael Shinn from Got Root) started efforts that focused on ModSecurity only.
As a first step, I decided to provide a way to associate meta-data with each rule (note that when I say "rule," I mean either a single filter statement or several statements chained together to form a single expression). I added four new actions:
Here is an example of all four new actions in use:
SecFilterSelective ARGS "<.+>" \ "id:1001,rev:2,severity:2,msg:'XSS attack attempt'"
To avoid collisions between rule identifiers, each publisher must only use the numbers from the range allocated to him. A special range is reserved for local use:
The rule meta-data will appear with each log entry, making it very easy to correlate messages with the rules that caused them:
[Sun Nov 13 18:49:39 2005] [error] [client 192.168.2.11] mod_security: Access denied with code 403. Pattern match "<.+>" at QUERY_STRING [id "1001"] [rev "2"] [msg "XSS attack attempt"] [severity "2"] [hostname "www.example.com"] [uri "/?p=%3Cscript%3E"] [unique_id "0RdXKX8AAAEAAENXACQAAAAA"]
Once you have the unique rule ID, you can use it for other purposes, too. For example, imagine you have a default configuration that does not work well with one particular aspect of your website. Using the unique rule ID, you can choose to go with the default configuration, removing just the rule that does not work:
<Location /subcontext/> SecFilterRemove 1001 </Location>
You can do it the other way around, too; not inheriting anything from the parent context but only importing a rule or two:
<Location /subcontext/> SecFilterInheritance Off SecFilterImport 1002 1003 </Location>
To help you further, it is now even possible to mark a rule as mandatory (using the action
mandatory). It is also possible to mark all rules in a given context mandatory (using the
SecFilterInheritanceMandatory directive). Mandatory rules can resist the
Serious users of ModSecurity will be happy to learn that it is now possible to specify custom actions for several rules at once. To preserve backward compatibility, I introduced a new directive:
SecFilterSignatureAction. Here is one example of its use:
# Warning-only rules SecFilterSignatureAction log,pass SecFilter KEYWORD1 SecFilter KEYWORD2 # Blocking rules SecFilterSignatureAction log,deny,status:403 SecFilter KEYWORD3 SecFilter KEYWORD4
SecFilterSignatureAction affects all rules that follow in the configuration file. Rules that do not specify custom actions will simply use the action list verbatim. Rules that do specify custom actions will have their actions merged with the actions specified in
SecFilterSignatureAction. This allows you to separate rule meta-data from security policy while still being able to override the default setting.
SecFilterSignatureAction log,deny,status:403 SecFilter KEYWORD5 id:1 # Respond with 404 instead of 403 SecFilter KEYWORD6 id:2,status:404
Of course, increasing rule action flexibility is not a desirable feature at all times, especially when you are considering whether to include a rule set produced by a third party. For example, what if the set contains a bad rule that will start rejecting legitimate requests? What if you don't want to reject requests at all but the set contains a rule that does? A new directive,
SecFilterActionsRestricted, helps with this problem. When enabled, it restricts which actions can appear in all of the rules that follow in the configuration file. Only a selected few actions, namely the meta-data actions, will be allowed. Others will be silently ignored. Now you can safely do the following:
SecFilterActionsRestricted On Include conf/modsecurity-thirdparty.conf
Still, you need to make sure the included file contains only
In the new version, ModSecurity introduces an entirely new audit log format, one that is suitable for real-time log aggregation. There are two cases when you will need real-time aggregation:
The old audit log format is a good choice if you need to occasionally look at the logs, but it suffers from two problems. First, it uses one file to store multiple entries. To make the one-file approach work properly, the web server needs to use a server-wide lock to serialize access to the file. The second problem is that the file format was making the log difficult to parse. The new log format (also called the concurrent audit log format) uses one file per audit log entry. It also uses a modular format that is much easier to parse and can be extended over time. Here is a configuration example:
SecAuditEngine RelevantOnly SecAuditLogType Concurrent SecAuditLog /var/www/audit_log/index SecAuditLogStorageDir /var/www/audit_log/
The index file will contain one line per audit log entry:
192.168.2.101 192.168.2.11 - - [15/Jul/2005:11:56:52 +0100] "POST /form.php HTTP/1.1" 403 3 "http://192.168.2.101:8080/form.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4" G3yTd38AAAEAAAM7BLwAAAAA "-" /20050715/20050715-1156/20050715-115652-G3yTd38AAAEAAAM7BLwAAAAA 0 1031 md5:dc910f6d647d47b32ae6e47326f0ca42
You will recognize most of the index entries. Each line begins with a combined log format, but it then adds the following fields:
UNIQUE_ID (generated by
- A session ID (reserved for future expansion)./li>
- The name of the file containing the audit log entry, relative to the root of the repository./li>
- The offset of the file where the audit log entry begins (to allow the repacking of multiple audit log entries into a single file)./li>
- The total length of the audit log entry./li>
- The MD5 hash of the complete audit log entry./li>
Each audit log entry will look like this:
--67458b6b-A-- [15/Jul/2005:11:56:52 +0100] G3yTd38AAAEAAAM7BLwAAAAA 192.168.2.11 4236 192.168.2.101 8080 --67458b6b-B-- POST /form.php HTTP/1.1 Host: 192.168.2.101:8080 User-Agent: Mozilla/5.0 Accept: */* Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Referer: http://192.168.2.101:8080/form.php Content-Type: application/x-www-form-urlencoded Content-Length: 5 --67458b6b-C-- f=111 --67458b6b-E-- 403 (Response body) --67458b6b-F-- HTTP/1.1 403 Forbidden Last-Modified: Fri, 08 Jul 2005 14:25:30 GMT ETag: "decb4-3-34b96a80" Accept-Ranges: bytes Content-Length: 19 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: text/html --67458b6b-H-- Message: Pattern match "111" at POST_PAYLOAD [id "1"] [rev "2"] [msg "3"] [severity "4"] Apache-Handler: application/x-httpd-php Stopwatch: 1126536042708000 11024 (7276* 7375 9842) --67458b6b-Z--
You can see how each audit log entry consists of multiple parts. Parts
H (meta-data), and
Z (trailer boundary) are mandatory. You can enable and disable everything else at will. If you look carefully, you will see that it is now even possible to log the complete response body (part
E), but this feature only works with Apache 2.x.
The key to real-time audit log processing is the index file. If, instead of a path to the file on disk, you supply a piped logging script that accepts index entries, the script will be able to handle the audit log entries as they are written to disk. We are currently busy creating a central logging console that automates the process of audit log collection. The console consists of agents that collect audit log entries from ModSecurity sensors, central database that keep the entries at one place, and a web-based GUI used to access and inspect the data.
While most users will want to use the audit log engine to store only the relevant requests (
SecAuditEngine RelevantOnly), in 1.9 there are improvements that allow you to do far more than that. The new actions
noauditlog allow you to log or not log a request to the audit log explicitly. This works well with a rule:
SecFilterSelective SOME_VARIABLE CONDITION auditlog
It is now also possible to log something to the audit log simply based on the response code. The argument of the
SecAuditLogRelevantStatus directive is a regular expression to apply to the response status code. If it matches, the request will be logged to the audit log. For example:
# Log internal server errors to the audit log SecAuditLogRelevantStatus ^5
People often ask me what ModSecurity can do about denial-of-service (DoS) attacks. Until recently, the answer was, "Not very much." It can help with the detection part or help you sort the bad requests from the good ones, but that was about it. To do more, ModSecurity would have to remember a thing or two between requests (at the moment, like Apache, it does not remember anything).
There is a bigger problem here. Even if ModSecurity were stateful, defending from a DoS attack on the web-server level is not the best possible option. The fact is that by the time you decide a request is a part of a DoS attack, it has mostly completed its purpose--it has engaged one web server instance to deal with it. A better option is to move the defense to the network level; for example, to a firewall that is nearest to the attacker. Deploying DoS defense at the network firewall is probably the best option. But if that's not an option for you, deploying in the host firewall will do almost equally well. We designed the improvements in ModSecurity 1.9 to handle both issues described here.
The multi-process nature of the Apache prefork execution model is usually an advantage (one process can go down while leaving the remaining processes unaffected; the controlling parent process simply creates another child). But there is a drawback: it is very difficult to share information between processes. Without information sharing, it is impossible to detect several classes of attack (for example, denial-of-service attacks).
One way to solve the problem of information sharing is to go the shared memory route. This approach is somewhat difficult to implement portably (although the APR library used to develop Apache 2.x helps a lot), but, ultimately, does not solve the problem entirely. Even if you manage to share the information between processes that are running on the same machine, what about the cases when you have several web servers working as part of a cluster?
I made a decision to use the piped logging mechanism, already well-supported in Apache, to export the relevant information out of ModSecurity to an external process that shared among all Apache processes. I named the new facility the Guardian log. In its first release I coupled the Guardian log mechanism together with a tool called
httpd-guardian, whose goal is to defend against denial-of-service attacks. This is how they work together:
The next time you start (or restart) Apache you will find the
httpd-guardian process running alongside it. This process will monitor the requests coming, watching for DoS attacks. The script keeps two counters for every IP address it sees: one counter for the most recent one-minute interval and another for the most recent five-minute interval. If an IP address sends too many requests in a measured period,
httpd-guardian will take an action against it. By default, the action is a mere warning, but you can configure
httpd-guardian to talk to your firewall to add the offending IP address to the blacklist (using
pf, or SnortSam).
If you have to restart Apache on regular basis, you will be happy to know that
httpd-guardian saves the counter values to a file on regular basis and reloads them the next time Apache starts.
If you need to deploy DoS protection for a cluster, you could the Spread Toolkit to centralize Guardian log output at a single location, and run only one copy of
httpd-guardian (which supports Spread directly) there. Problem solved.
There have been many smaller improvements throughout and it is difficult to list them all in a meaningful way. So I will just conclude the article with a list of the more interesting ones:
ModSecurity has come a long way, with widespread deployments in the last two years. We've done a lot, but there is still a lot to do. The 1.9 release has brought several big improvements that extend the areas where ModSecurity is useful. The wealth of smaller improvements have increased the flexibility, which is what really matters to handle specific web application security problems. ModSecurity is now a well-established tool for web intrusion detection (traffic auditing and monitoring) and for just-in-time patching (reducing the window of opportunity for the attacker). As for the future, the real problem we are facing is deciding which of the features on our list to do first!
Two features are already scheduled for immediate development. Because ModSecurity does not come with a default rule set, many of its users have complained they don't know how to start using it. To solve this problem, we will soon release a generic rule set that works as a deployment guide, but also configures ModSecurity to work as a web-intrusion-detection tool. This rule set will first appear as a standalone project. It will be a part of all subsequent releases. The focus of the 2.0 release, scheduled for Q1 2006, will be the addition of the positive security model and the surrounding tools to profile web application traffic and automatically create protection rules.
Ivan Ristic is a Web security specialist and the author of ModSecurity, an open source intrusion detection and prevention engine for web applications, and the author of O'Reilly's Apache Security.
Return to the Apache DevCenter.
Copyright © 2009 O'Reilly Media, Inc.