The first book on BSD Rootkits was recently published. Federico Biancuzzi interviewed the author, Joseph Kong, to learn more about the dark art of kernel voodoo...
Could you introduce yourself?
Joseph Kong: I am a relatively young (24 years old) self-taught computer enthusiast who enjoys working (or playing, depending on how you look at it) in the field of computer security; specifically, at the low-level. In the past I have contributed to Phrack Magazine, and I just recently finished writing my first book (Designing BSD Rootkits) published by No Starch Press.
Interested readers can find more information about me at thestackframe.org.
When did you hear about rootkits for the first time?
Joseph Kong: The first time I heard the term "rootkits" was in 2004--straight out of the mouth of Greg Hoglund, who was at the time promoting his new book Exploiting Software: How to Break Code. That's actually how I got into rootkit programming. Thanks Greg. :)
Why did you choose to write a book about BSD rootkits?
Joseph Kong: Frankly, because I wanted to fill an information gap. There is a lot of documentation regarding Windows and Linux systems programming (which is the basis for kernel-mode rootkits), but not as much documentation regarding *BSD systems programming. Also, at the time of this writing, I believe there are four different books on Windows rootkits and one on Linux rootkits.
However, if you want a really simple answer, it's because I like rootkits (for any OS) and I like FreeBSD.
Is the book focused on FreeBSD only?
Joseph Kong: The book is focused on FreeBSD, however, the methods covered will work on other OSes.
Do you have any experience or story to share about rootkits hunting?
Joseph Kong: It's hard. This is because if you are hunting a "real" rootkit (i.e., one that's not publicly available), you won't know what technique(s) it employs or which functions/objects it manipulates. Thus, you have to, more or less, validate the integrity of your entire system--and that's no small task.
This might be a bit off topic, but, this is why most rootkit detectors are signature based. It's just that much easier (in fact, it's rather trivial) to detect a rootkit when you know exactly what it's going to do.
Do you know any anti-rootkit tool/product for *BSD?
Joseph Kong: I know of two tools to detect rootkits on *nix-based (or derived) systems. They are chkrootkit, which is available in the FreeBSD ports collection, and Rootkit Hunter. Both of these tools are, more or less, signature based detectors, which makes them fairly lightweight. Also, they are both pretty easy to use.
Personally, and this isn't an attack against these tools or authors (as I respect the authors tremendously for their unpaid work), but I find these tools to be somewhat behind the curve (e.g., they don't detect some of the most common system call hooks). Of course, this isn't their fault, as there aren't a lot of security researchers dedicated to exposing *nix-centric rootkits.
As an aside, keep in mind that signature based detection, in general, will never be able to detect something (e.g., a rootkit, virus, or worm) that it doesn't have a "signature" for.
Along similar lines, I know a lot of people who refer to rootkits and rootkit-detectors as being in a big game of cat and mouse. However, it's really more like follow the leader--with rootkit authors always being the leader. Kind of grim, but that's really how it is. Until someone reveals how a specific (or certain class of) rootkit works, nobody thinks about protecting that part of the system. And when they do, the rootkit authors just find a way around it. This is what I meant earlier when I said rootkit hunting is hard--as you really have to validate the integrity of the entire system.
Are you working with FreeBSD guys to implement anti-rootkits measures?
Joseph Kong: In short, no. The interesting thing about rootkits is that, in general, every technique that a kernel-mode rootkit employs is a legitimate system building technique. For example, run-time kernel memory patching (or as Microsoft calls it, hot patching) can be used to legitimately or illegitimately patch a running system. The problem is, when a user gains root access, the system can't tell whether a request to augment the kernel is legitimate or not. Also, pretty much every anti-rootkit measure has been circumvented (although I am a fan of secure levels).
In my opinion, currently, the best anti-rootkit measure is prevention. That is, if you can prevent an unauthorized user from gaining root access, all the rootkits in the world won't help him/her. In this sense, I believe the OpenBSD methodology of fixing every bug (big or small), code auditing, secure by default, and so on, is the best course. And I believe that every system builder already knows this (they're just not all vocal about it).
Keep in mind that although I am extolling the virtues of prevention, as other computer security professionals (such as, Richard Bejtlich) have said, prevention eventually fails (e.g., Loïc Duflot showed that you can bypass secure levels in SMM), and detection is just as important. The problem is rootkit detection, as I said earlier, is difficult.
(Joanna Rutkowska presented an interesting idea for rootkit detection/OS verification back in December 2006.)
Do TrustedBSD extensions help us defending our systems?
Joseph Kong: Of course; that is their design goal after all.
I think the event auditing extension (which was included into the FreeBSD base system as of 6.2) is really useful. It allows you to log a variety of events, which can then be used for postmortem analysis or intrusion detection.
Also, the Mandatory Access Control (MAC) extension has a policy, I think, that lets you sign executables and prevents non-signed executables from running.
(I haven't played around with the MAC extension too much, but this is what John Baldwin tells me. Thanks John.)
Is there anything that a FreeBSD sysadmin should do to reduce the risk of getting rootkits in his boxes?
Joseph Kong: If you don't require KLD support, then of course you can disable it; which just requires you to raise the secure level to 1 or higher. (This is akin to disabling unnecessary services.) You can also "secure" your applications with jails.
If you haven't looked through it yet, the FreeBSD Handbook Chapters 14-17 provide some good (general purpose) information on how to harden your system.
FreeBSD keeps stable kernel ABIs in release branches. I am wondering if this means that a kernel rookit would just have to be developed for a particular major release (ie 6.0) and it will work on every following releases (6.1, 6.2,...).
Joseph Kong: Well that really depends on what your rootkit is trying to do, and whether or not there are any major changes to the OS's internals between releases. For example, if your rootkit was trying to patch some object, and that object's data structure changed between releases, your rootkit might break. From my personal experience, everything I have written for 6.0 runs perfectly under 6.1 and 6.2. Of course, this is probably because FreeBSD doesn't really change too much between STABLE releases. (However, the differences between STABLE and CURRENT sometimes are pretty drastic.)
How much effort is needed to write a rootkit that works on multiple branches (4.x, 5.x, 6.x, 7.x)?
Joseph Kong: Honestly, not much (or at least in my opinion, not much). In general, I find that it doesn't require much effort to write a rootkit that is backwards/forwards compatible. Of course, this is all subjective and dependent on what gets changed between versions. For example, in FreeBSD 4 system call functions had the following function prototype:
typedef int sy_call_t __P((struct proc *, void *));
This got changed to the following in FreeBSD 5:
typedef int sy_call_t(struct thread *, void *));
The main difference here is that the first parameter got changed from struct proc (a process structure) to struct thread (a thread structure). So, if you had a kernel-mode rootkit that hooked one or more system calls on FreeBSD 4, you would have to, at the bare minimum, change one or more parameters/arguments to get it to run on FreeBSD 5. On the other hand, if your rootkit patched some data within struct proc on FreeBSD 4, you now have to take into account that FreeBSD 5 executes processes at thread granularity, and that you may have to patch struct thread instead. This might sound like a lot of work, but it's really not that hard or time consuming. The main thing is identifying how the objects you interact with differ between versions (and having the full kernel source makes that pretty easy).
Now that multi-CPU/core systems are common, does it mean that rootkits must be carefully designed to be SMP safe? Maybe some sort of regression suite might spot syscalls that stop being SMP safe because they are handled by a non-SMP-safe rootkit...
Joseph Kong: It does, and it doesn't. On a symmetric multiprocessing (SMP) system, if a thread running on one CPU is manipulating an object and a thread on another CPU begins manipulating the same object, data corruption can result. Therefore, all kernel code, including kernel-mode rootkits, must prevent this situation from occurring. Typically, this is achieved by employing some sort of synchronization scheme (e.g., mutexes).
However, this same type of situation can occur on a uniprocessor (UP) system, because kernel code (like use-mode code) can be interrupted. For example, if a thread is manipulating an object, gets interrupted, and another thread gets scheduled to run, which manipulates the same object, data corruption can result. As you can see, this situation is, in effect, identical to the SMP situation described above, and as such, it can be prevented in the same way.
So yes, rootkits need to be designed SMP safe, but in effect, that's the same as designing a thread-safe UP rootkit, which rootkit developers have had to do for a while now. Thus, from a rootkit development standpoint, the prevalence of SMP systems isn't that big of an issue. (On the other hand, from an OS development standpoint, SMP is a very big issue--but that's a completely different discussion altogether).
On a SMP system each processor has its own interrupt descriptor table (IDT). Thus, a UP designed rootkit that manipulates the IDT, when retooled for a SMP system, will have to take this into account. This is the only point that I can think of (i.e., know about) in which a SMP system might mess up a UP designed rootkit that is thread-safe.
Some people need to use binary-only kernel modules to support their hardware. For example some NVIDIA or ATI graphic cards. This means loading something like 2 MB of code in your kernel space. Is there anything that could be done to reduce the risk of running that code?
Joseph Kong: I guess you could use super verbose audit logs (and that would help), but as an overall, I don't think there is much you can do to reduce the risk of running that code. For example, assuming the code is malicious and you load it into kernel space, it can now disable/mitigate any user/kernel space protection schemes you have installed (that it's aware of).
Obviously with this specific case, prevention is (completely) useless, and you have to depend on detection.
There is a port of DTrace for FreeBSD. Would it help to analyze binary only code and maybe spot anomalies by profiling code performances?
Joseph Kong: DTrace is a very powerful tracing framework, so it can be used to analyze binary blobs, "pinpoint" system anomalies (e.g., performance bottlenecks), profile code performance (e.g., the elapsed time of each function call), and so on.
However, the problem is "properly" designed malware wouldn't (or shouldn't) introduce any sort of system anomaly (e.g., performance bottlenecks)--it wouldn't be too stealthy if it did. This makes detecting "malicious" binaries with DTrace somewhat difficult.
So, while DTrace is great for analyzing your system, for detecting malware, it really depends on the malware itself, and how well you know your system.
You can get an accurate measure of how much time a function or syscall takes to complete with DTrace. You just set it to take a timestamp whenever the desired function or syscall gets called/entered and when it completes/returns (and subtract the first time from the second), but, the added overhead of a call hook is so minute (assuming the hook is written properly) that you probably wouldn't notice. Also, there are lots of factors that affect how long a function takes to complete (e.g., branching code paths, sleeps, and so on), so you would need to know your system's (average) performance times in order to realize that an additional millisecond was the result of a hook and not something else.
At the end of the day, it's just much easier to scan for hooks/patches over profiling code performance, in order to find a rootkit--especially if you are looking for (basic) syscall hooks.
FreeBSD handles Linux emulation pretty well. I am wondering if it handles it so well that some rootkits for Linux could work on FreeBSD too...
Joseph Kong: Well a kernel-mode rootkit for Linux would not work on FreeBSD (because of different data structures/kernel internals), but, a Linux user-mode rootkit could work. For example, if the rootkit simply trojaned the output of ps(1) or ls(1) that could work, however, if the user-mode rootkit modified some specific Linux kernel data structure (e.g., the system call table, i.e., SucKIT) that would fail. So, I guess the answer is, it depends on the Linux rootkit.
I had this question when I heard that Luigi Rizzo is working on a framework to use Linux drivers on FreeBSD. I thought that this framework, if/when integrated, might open the way to malicious code designed for Linux.
Joseph Kong: First off, thanks for pointing this framework out to me. I've been playing around with it for the last day or so. It's actually very cool from a device driver developer's standpoint, as it lets you take Linux LKM source code and compile it directly under FreeBSD, without any modifications, to produce a native FreeBSD binary--that's awesome!
However, from a rootkit developer's standpoint, it's not that useful (at least in my opinion). To understand why, you need to understand how the framework is designed. Essentially, the framework is for building Linux device drivers on FreeBSD and it works in two ways. First, it remaps Linux functions, header files, structures, and so on to equivalents in FreeBSD. If that's inadequate or infeasible, the various Linux functions are reimplemented in FreeBSD. (Of course, this description is an oversimplification, but you get the idea.) The problem is most LKM rootkits hook/patch various low-level functions/data structures that, more often than not, don't get explicitly called/interacted with from within a device driver. Thus, the framework doesn't have a remapping for these items, and as such, it's not going to pave the way for Linux LKM rootkits to be run under FreeBSD. Of course, that doesn't mean I didn't at least give it a try. :)
I wrote a simple Linux LKM which hooked the Linux system call table (sys_call_table), compiled it under the framework (with no errors), but when I loaded it, I got the following error message:
link_elf: symbol sys_call_table undefined
Basically, the framework didn't remap the Linux system call table to the FreeBSD system call table. Of course, it wouldn't--when are you ever going to need to directly manipulate the system call table when writing a device driver?
Does loading our packet filter (pf, ipfw, ...) as a KLD let a rootkit mess with it more than if it was statically built in the kernel?
Joseph Kong: No, it does not; while an attacker could Trojan your KLD packet filter, I just don't see anyone going through all that trouble. Allow me to explain.
Let's assume that an attacker does have a Trojan KLD and that you don't use a KLD for your packet filtering. Now, this means that their Trojan is useless, and that they are going to need (or have to write) another program to attack your packet filter. But, what's the point? When you load a KLD into your system, for all intents and purposes, it's as if it was statically built into the kernel. In other words, after a KLD is loaded, you can list/dump/find the address of its' functions/symbols by examining the currently running kernel image (i.e., the image in main memory). Thus, the code to hook/patch a loaded KLD is the same as the code to hook/patch the currently running kernel. In other words, writing a Trojan KLD is unnecessary.
How can an attacker control a rootkit on a remote machine?
Joseph Kong: In general, there are only two reasons to communicate with a rootkit on a remote machine:
Both of these tasks are typically achieved through a covert channel. A covert channel is defined (by the US DOD) as follows:
"Any communication channel that can be exploited by a process to transfer information in a manner that violates the systems' security policy."
Typically, covert channels work in one of two ways:
The following is an example covert channel that can be used for data exfiltration:
Whenever a TCP connection is attempted (i.e., a SYN packet is sent out) on the owned machine (i.e., the machine with the rootkit installed), the Initial Sequence Number (ISN) is modified, such that it stores the data the rootkit wishes to push out.
In order to achieve this, the rootkit must sit in between the TCP/IP stack of the system and the network interface. Additionally, the covert channel engine (within the rootkit) must constantly change the sequence (SEQ) and acknowledgment (ACK) numbers generated by the kernel and by the packets received from the wire. This is because if the covert channel engine only changed the first SEQ number (within the first SYN packet), the kernel would not understand the corresponding ACK number (within the second SYN | ACK packet); which would break the connection. Remember, the kernel believes that the ISN hasn't been modified.
Additionally, the covert channel engine should encrypt the ISN so that it looks like a random number and not data.
Notice that with this scheme, the rootkit doesn't create/establish a connection, it simply rides on one going out, which makes it stealthier and allows it to workaround stateful packet filters. But, in order to gather the data, the attacker must also own a machine in between the compromised computer and the one it's communicating with.
For more on this particular covert channel see "The Implementation of Passive Covert Channels in the Linux Kernel" by Joanna Rutkowska.
Creating a covert channel for remote command/control is similar to creating one for data exfiltration, except that it must be able to send and receive data, as opposed to just sending data. The following is an example covert channel that can be used for remote command/control. It is assumed that the "owned" machine is behind a stateful packet filter.
Periodically (or at some set time), the rootkit on the owned machine will send out a DNS request that's arranged as follows:
This will cause the owned machine's local DNS server to connect to a nameserver for
domainname.tld. The nameserver(s) at
domainname.tld are, of course, under the rootkit owner's control. Thus, the response is controlled. Essentially, this is a connect back, covert, remote shell. The rootkit can send data up in the form of queries, and get control information in the form of responses. Naturally, the queries and responses are designed to look legitimate.
Of course, DNS isn't the only option, any request/response protocol can be used as a covert channel for remote command/control. For more information on covert channels see the Gray-World Team website.
The only real criteria when choosing a protocol to use, is to make sure that its packets are fairly prolific on the network, so that you can hide within plain sight.
Federico Biancuzzi is a freelance interviewer. His interviews appeared on publications such as ONLamp.com, LinuxDevCenter.com, SecurityFocus.com, NewsForge.com, Linux.com, TheRegister.co.uk, ArsTechnica.com, the Polish print magazine BSD Magazine, and the Italian print magazine Linux&C.
Return to ONLamp.com.
Copyright © 2009 O'Reilly Media, Inc.