IRIX Binary Compatibility, Part 2

by Emmanuel Dreyfus
08/29/2002

Unix Program Startup

Now that our kernel is able to distinguish the difference between IRIX binaries and other programs, we need to arrange the program environment so that the IRIX binary is able to start up. (See Part 1 in this series for more on this.)

Generally speaking, Unix kernels have to communicate a few things to user programs in order to start them up. This includes the program's arguments and environment, and for dynamic binaries, the ELF auxiliary table, which is used by the dynamic linker to learn how to link the program. All this information is transmitted to the user program through the CPU registers and the stack.

If this information is corrupted, static binaries are still likely to work, but they will lose their arguments and environment. On the other hand, dynamic executables will not start at all if the ELF auxiliary table is screwed, because the dynamic linker will not be able to link them.

Therefore, it is a good idea to start with a simple static binary. We will use IRIX 5's sed(1) here. When we run it using the NetBSD native function to set up the stack and CPU register, it is able to start up. Then it gets a SIGSYS signal and it dumps core on the first system call, because our system call table for IRIX binaries is still empty. It is possible to check what the missing system call is with the ktrace(1) command on NetBSD:

$ ktrace -di /emul/irix/bin/sed /etc/passwd
Bad system call (core dumped)
$ kdump
  1209 ktrace   EMUL  "netbsd"
  1209 ktrace   CALL  execve(0x7fffea5f,0x7fffe99c,0x7fffe9a4)
  1209 ktrace   NAMI  "/emul/irix/bin/sed"
  1209 sed      EMUL  "irix o32"
  1209 sed      RET   execve 0
  1209 sed      CALL  #4 (unimplemented write)
  1209 sed      PSIG  SIGSYS

Most of the system calls first used by /bin/sed are plain SVR4, so it was easy to emulate them: just copy the system call definition from sys/svr4/syscall.master to sys/irix/syscall.master, issue a make to refresh the files generated from syscall.master, rebuild a kernel, reboot, and retry.

Within a few minutes, it was easy to get IRIX's /bin/sed nearly working. The next problem was to have it take its arguments correctly.

Setting Up the Stack for Program Startup

For static binaries, the stack is used to transmit arguments and the environment to the user program. The way it should be done is documented in the SVR4 ABI MIPS processor supplement.

In This Series IRIX Binary Compatibility, Part 6 With IRIX threads emulated, it's time to emulate share groups, a building block of parallel processing. Emmanuel Dreyfus digs deep into his bag of reverse engineering tricks to demonstrate how headers, documentation, a debugger, and a lot of luck are helping NetBSD build a binary compatibility layer for IRIX. IRIX Binary Compatibility, Part 5 How do you emulate a thread model on an operating system that doesn't support native threads (in user space, anyway)? Emmanuel Dreyfus returns with the fifth article of his series on reverse engineering and kernel programming. This time, he explains thread models and demonstrates how NetBSD emulates IRIX threads. IRIX Binary Compatibility, Part 4 Emmanuel Dreyfus tackles the chore of emulating IRIX signal handling on NetBSD. IRIX Binary Compatibility, Part 3 Emmanuel Dreyfus shows us some of the IRIX oddities, the system calls that you will not see anywhere else. IRIX Binary Compatibility, Part 1 This article details the IRIX binary compatibility implementation for the NetBSD operating system. It covers creating a new emulation subsystem inside the NetBSD kernel as well as some reverse engineering to understand and reproduce how IRIX internals work.

The NetBSD kernel uses a function pointed to by the es_copyargs field of the struct execsw to set up the program stack on startup. Because NetBSD/mips ports conform to the SVR4 ABI, we could have expected the NetBSD version of this function (elf32_copyargs() from sys/kern/kern_exec_elf32.c) to just work with IRIX binaries. Unfortunately, this is not true. Using the NetBSD elf32_copyargs function with static o32 IRIX binary such as /bin/sed showed weird behavior with the way argument read: sometimes /bin/sed was reading the arguments correctly, sometimes it was not. The behavior was dependent upon the argument length. This suggested that something in the stack had to be aligned on a particular boundary; with some argument lengths it was being aligned, and with others it was not.

I already had to face this kind of situation when working on Linux/PowerPC binary compatibility on NetBSD (read the whole story). However, the situation is different here: IRIX is a closed source proprietary OS; therefore it is not possible to grab kernel sources and look at the way the IRIX kernel sets up the stack. Worse, because I was not able to build static binaries, it was impossible to make a static test program that dumped the stack and displayed argc, argv and envp to check what was wrong in the way the stack was set up.

The solution was to use gdb. With gdb we can run /bin/sed on IRIX, set a breakpoint at the beginning of the program and then examine the stack. The first thing to know is the program startup address. This information can be obtained using objdump on IRIX's sed:

$ objdump -f /bin/sed

/bin/sed:     file format elf32-bigmips
architecture: mips:3000, flags 0x00000102:
EXEC_P, D_PAGED
start address 0x100000c0

Then we can start gdb and set our breakpoint. It seems that it is not possible to break on the program's first instruction, but we can break on the second instruction. On the MIPS, all instructions are four bytes long, hence the second instruction is four bytes away from the program's start address, at 0x100000c0 + 0x4 = 0x100000c4:

$ gdb /bin/sed
(gdb) b *0x100000c4
Breakpoint 1 at 0x100000c4
(gdb) run aa aaa
Starting program: ./sed aa aaa

Breakpoint 1, 0x100000c4 in ?? ()
(gdb) x/16wx $sp
0x7fff2fa0:     0x00000003      0x7fff3000      0x7fff3027      0x7fff302a
0x7fff2fb0:     0x00000000      0x7fff302e      0x7fff3057      0x7fff306a
0x7fff2fc0:     0x7fff30a7      0x7fff30b4      0x7fff30be      0x7fff30d4
0x7fff2fd0:     0x7fff30e1      0x7fff30f6      0x7fff3109      0x7fff3113

In this dump, we recognize a standard startup stack layout--the argc value (three arguments: '/bin/sed', 'aa', and 'aaa'), followed by the argv array (a NULL terminated array of pointers to the argument strings), and then the envp array (a NULL terminated array of pointers to the environment strings). There are a lot of environment strings here, hence we do not see the trailing NULL here, it is a bit farther in the stack dump.

It is possible to dig up the value of an argument with its address:

(gdb) x/s 0x7fff3000
0x7fffea60:      "/bin/sed"
(gdb) x/s 0x7fff3027
0x7fffea72:      "aa"

Dumping the stack with various arguments to /bin/sed, it is possible to discover that, for an IRIX binary, the argv[0] must be aligned on a 16-byte boundary. The IRIX kernel sets the stack that way, and IRIX binaries depend on this particular layout.

It was not possible to modify elf32_copyargs() to implement this particular behavior because it is also used by native NetBSD binaries. Hence, the solution was to duplicate what was done in elf32_copyargs() in an irix_copyargs() function, which can be found in sys/compat/irix/irix_exec_elf32.c. This irix_copyargs() function just does elf32_copyargs() job and it enforces the 16-byte alignment of argv[0]. Of course, the irix_copyargs() function must be used in the es_copyargs field of the struct execsw for IRIX in sys/kern/exec_conf.c.

With this adjustment, static IRIX binaries were able to read their arguments and environment correctly.

Pages: 1, 2

Next Page