BSD DevCenter
oreilly.comSafari Books Online.Conferences.


Emmanuel Dreyfus Interview

by chromatic

A recent KernelTrap story brought the NetBSD team's work on Mach and Darwin binary compatibility to the spotlight. NetBSD Developer (and O'Reilly Network columnist) Emmanuel Dreyfus is leading the charge.

NetBSD has always had portability as a major goal. It's not surprising that it's able to run binaries compiled for Linux, FreeBSD, Solaris, and other operating systems. The reasons for this are partly pragmatic and partly impractical.

Manu graciously agreed to an interview to clarify the project's goals and the scope of last week's update.

ORN: Let's talk about the basics of binary compatibility for a moment. Here are the facts as I understand them:

  • it is intended for programs for which you do not have source code
  • it almost always requires running on the same architecture
  • it means emulating a binary interface
  • most of the work is to write a translator for system and kernel calls between the two operating systems

For example, you're not going to see Mac OS X apps running on non-PPC hardware anytime soon, if ever, because the architecture doesn't match.

Is there anything I'm missing?

ED: No, this sounds fine, you are right.

If we want to run a binary on a processor it was not build for, then we have to emulate the CPU in software. This kind of thing has been done by Apple for their m68k to PowerPC transition. Emulating the CPU in software costs a lot, so it will give you reasonable performances only if the native CPU is much faster than the emulated CPU (which was the case for Apple's m68k emulator for the PowerPC, but it was an unusual situation).

IRIX Series of Articles

IRIX Binary Compatibility, Part 6
With IRIX threads emulated, it's time to emulate share groups, a building block of parallel processing. Emmanuel Dreyfus digs deep into his bag of reverse engineering tricks to demonstrate how headers, documentation, a debugger, and a lot of luck are helping NetBSD build a binary compatibility layer for IRIX.

IRIX Binary Compatibility, Part 5
How do you emulate a thread model on an operating system that doesn't support native threads (in user space, anyway)? Emmanuel Dreyfus returns with the fifth article of his series on reverse engineering and kernel programming. This time, he explains thread models and demonstrates how NetBSD emulates IRIX threads.

IRIX Binary Compatibility, Part 4
Emmanuel Dreyfus tackles the chore of emulating IRIX signal handling on NetBSD.

IRIX Binary Compatibility, Part 3
Emmanuel Dreyfus shows us some of the IRIX oddities, the system calls that you will not see anywhere else.

IRIX Binary Compatibility, Part 2
Emmanual Dreyfus shows us how he implemented the things necessary to start an IRIX binary. These things include the program's arguments, environment, and for dynamic binaries, the ELF auxiliary table, which is used by the dynamic linker to learn how to link the program.

IRIX Binary Compatibility, Part 1
This article details the IRIX binary compatibility implementation for the NetBSD operating system. It covers creating a new emulation subsystem inside the NetBSD kernel as well as some reverse engineering to understand and reproduce how IRIX internals work.

ORN: I know you've been involved with other binary compatibility layers (most recently your IRIX series, for example). How does one get started? Are there any applications you really wanted to run on NetBSD but couldn't?

ED:There are a lot of motivations to write some binary compatibility software.

The first obvious reason is indeed to get some interesting applications running on NetBSD. For instance, there is no recent native Java Virtual Machine (JVM) available for NetBSD PowerPC ports, and we now have a lot of people developing Java applications because they believe Java to be more portable than C. I had the need to run Java programs, and the only machine fast enough to do this at home was my NetBSD/macppc machine. This is what pushed me to wrote the PowerPC parts of Linux binary compatibility on NetBSD. We are now able to use the Blackdown Java Developement Kit (JDK) for LinuxPPC on NetBSD PowerPC ports through the binary compatibility layer.

There is a different motivation to get a binary compatibility layer for people using UNIX workstations such as Alphas, Suns or SGIs. For various reason these people might want to migrate from their original Operating Systems (OS): the box is too old to support recent versions of the OS, or they do not have the money for an OS upgrade, or the OS support will be dropped in the future, or they cannot bear the native OS any longer.

NetBSD runs on these workstations. Recent NetBSD releases work well on old hardware, NetBSD is free, NetBSD is well supported, and NetBSD is administrator friendly (okay, this latter point is just a matter of taste). Therefore switching the box to NetBSD can be a good option. Unfortunately, there are sometimes some old locally developed application that cannot be migrated easily because they are not written in a portable way. The binary compatibility layer can help a lot here.

There are also some academic motivations. Implementing Mach binary compatibility in NetBSD will help us finding out how well a Mach-like microkernel IPC mechanism can be implemented within a traditional monolithic kernel such as NetBSD. Christos Zoulas (who set the first stone on Mach binary compatibility on NetBSD) and I are planning to write a paper covering the following topics:

  • performance analysis on mach-ipc implemented on top of a monolithic kernel;
  • challenges/complexity of implementing mach ipc messages on a monolithic kernel.

As you see, I had a lot of motivations. And of course also the pleasure to work on a challenging project.

ORN:It seems that supporting a "real-world application" on a fledgling operating system would be a good way to discover what features you don't yet support. Of course, NetBSD is a mature OS, but have you found any improvements that needed to be made?

ED:Well, yes and no. When setting up the IRIX binary compatibility on NetBSD/sgimips, I discovered a bunch of IRIX features that NetBSD does not support natively. We have been able to support a lot of them, but there are very few that we really integrated into NetBSD beyond the IRIX binary compatibility layer.

For instance, IRIX does use a libc-provided signal trampoline (see IRIX binary compatibility part 4 for more informations about this), and NetBSD did not. The libc-provided signal trampoline has many advantages, so we switched to that feature for native NetBSD binaries, thus improving NetBSD.

On the other hand, there are a lot of foreign OSes feature emulated in the binary compatibility layer that we do not want to support natively, mostly because they consume too much resources: the IRIX ability to share all the Virtual Memory (VM) space mapping but some pages within a share group is an example of such a feature.

ORN: From my (few) experiences with the Mac-on-Linux project, it looks like the project is similar. Can you speak of the similarities or differences?

ED:Well, MoL is much like the PowerPC equivalent of i386 software such as VMWare or Plex86: these are virtual machines inside which you run the whole foreign OS, kernel included. It's like another machine inside your machine.

Binary compatibility makes it possible to run the foreign OS binaries on top of the NetBSD kernel. It is possible to mix binaries from NetBSD and the emulated OSes: they are able to communicate through pipes or sockets, you see them in the process list with ps, a foreign OS binary is able to launch a native process, and so on. Binary compatibility enable me to run an anti-virus software build for Linux as a filter in my NetBSD mail server, for instance.

With virtualization software such as MoL or VMWare, the guest Operating System in the virtual machine is completely separated from the host OS, there is no way to bring the processes from the two OSes into interaction. This has some advantages, if you want to study some hostile software, but on the other hand you will never have the integration you have with binary compatibility.

Of course this integration is theorical when we come to OSes such as MacOS 9, which have nothing common with UNIX: it makes no sense to connect a MacOS application to a UNIX application through pipes for example. So virtualization is probably the best way to run MacOS 9 binaries on a UNIX system. For MacOS X, which is UNIX-based, I believe binary compatibility would be much more interesting than virtualization.

There is another point to comment on: performance. Binary compatibility is extremely efficient. The performance loss compared to running the binary on the OS it was build for is about 1-2%. And for some system calls, the NetBSD compatibility implementation is even faster than the original implementation.

Virtualization performance is good, but not as good as binary compatibility performance. Virtualization software has to catch all hardware exceptions generated by the guess OS and emulate them in software. In some situations, it can be quite expensive.

ORN:Even though you don't have the source code of applications or libraries you'd like to run on your system, if they don't come directly from the OS vendor, there has to be documentation somewhere. How much can you learn just by reading public docs on kernel interfaces? Does this help?

ED:A lot! In fact, all the necessary information is found in the kernel header files and the man pages. Here is how it happens: you try running a given binary, and you hit some system call you did not implement yet. All that you have is the system call number and the arguments the binary used when issuing the system call.

In the kernel header files, you find the system call table. From the system call number, you get the system call name. If you are lucky, the system call has a man page, which refers to some kernel header file for the data structures the system call uses. You have all the documentation about the system call, and you just implement it based on its documentation. Your life is easy.

Related Reading

Mac OS X for Unix Geeks
By Brian Jepson, Ernest E. Rothman

If you are not lucky, the system call is undocumented. Then you have to study it using gdb, trying to give it different arguments, and checking for the results. Usually you can figure out a bit about it based on its name; MacOS X's pthread_exit() must have something to do with thread termination, for example. And sometimes the system call is not documented, but you can find its prototype and the data structures it uses in a kernel header. That helps a lot.

Contrary to popular belief, having the kernel sources for the OS you want to emulate does not help very much. I experienced that with Linux. Sometimes you have a behavior which is not documented. You can read messy Linux sources and try to figure out how this is supposed to behave, or you can run some test programs on a live system and inspect things with gdb. The latter tends to produce much more interesting results, since you actually see how things are, instead of trying to understand how they should be.

ORN:What have been the responses so far? Have you helped people migrate to NetBSD without losing their necessary applications?

ED:I had a lot of feedbacks about the LinuxPPC binary compatibility, mostly from people running Java applications such as Cocoon or FOP.

IRIX binary compatibility is more recent, and less mature. There are the N32 binaries that are not supported yet which makes it rather impractical for replacing IRIX with NetBSD. But I know some people who think about it.

ORN: Conversely, have you had anyone (IRIX or HP engineers) contact you to offer help?

ED: Some SGI engineers contacted me to wish me some success on the IRIX binary compatibility project, but I got no help from them on the technical front. I had no news from Apple engineers yet.

ORN:Have you heard anything about legal considerations for NetBSD's binary compatibility layers?

ED: Well, as far as I know (I am not a lawyer) reverse engineering some software for interoperability purpose is a legal practice in a lot of countries around the world, including my home country. There is indeed some concern in the U.S. because of the DMCA.

IRIX binary compatibility is a descendant of System V Release 4 (SVR4) binary compatibility, which was developed pre-DMCA, so there should be few problems. For MacOS X binary compatibility, things are different: most of the MacOS X kernel is from the Carnegie Mellon University (CMU) Mach project and is available under the CMU copyright. There is very little Apple proprietary information involved, so it should be all right too.

But, generally speaking, there has been no legal threat against emulation projects, whether commercial or noncommercial. If that happens, one could argue that it is some kind of consumer right to run on any platform some software for which one purchased a license. A bit like you expect to play a CD you purchased on any CD-Player, whether it is from Pioneer or from Sony.

ORN:How portable are your compatibility layers? On the one side, any Mac OS X library is going to target the same Darwin/Mach system calls. On the other side, you'll need specific code for, say, the NetBSD and the Linux kernels. Given that the binary interface is hardest to find, how difficult would it be to port the code to another free Unix? (I'm assuming that figuring out what you need to implement is the hardest part. If not, please correct me.)

ED:You are mostly right, that is the hardest part and also the most interesting. Porting to another UNIX system would be an extremely boring job: nothing to discover, just thousands lines of code to tweak.

There would be a lot of tweaking because the binary compatibility layer is indeed not portable. And it is not portable because it is implemented in the kernel. There are some standards that help make userland programs portable, but there is nothing which standardizes internal kernel APIs between different UNIX systems.

ORN:Anything else we should know?

ED:Yes. NetBSD binary compatibility will enable running MacOS X binaries on NetBSD. You will still need a MacOS X license. The binary compatibility only emulates the kernel's behavior, not the userland. Therefore, in order to run a dynamically-linked application, you need the libraries from the emulated OS. The libraries are part of the emulated OS, and if you use them, you need a license for it.

I also need some manpower to help on this project. Mach emulation is much different from most of the binary compatibility we already have in NetBSD. We used to emulate other UNIX kernels, and Mach is not a UNIX kernel. (Darwin, which runs on top of Mach, is a UNIX kernel.) This makes the project a real challenge.

There are a lot of ways to help, including tasks that are limited to userland programming. A test suite of Mach traps would be wonderful, for example. So do not be shy, join the project!

chromatic manages Onyx Neon Press, an independent publisher.

Return to the BSD DevCenter.

Sponsored by: