Now that our kernel is able to distinguish the difference between IRIX binaries and other programs, we need to arrange the program environment so that the IRIX binary is able to start up. (See Part 1 in this series for more on this.)
Generally speaking, Unix kernels have to communicate a few things to user programs in order to start them up. This includes the program's arguments and environment, and for dynamic binaries, the ELF auxiliary table, which is used by the dynamic linker to learn how to link the program. All this information is transmitted to the user program through the CPU registers and the stack.
If this information is corrupted, static binaries are still likely to work, but they will lose their arguments and environment. On the other hand, dynamic executables will not start at all if the ELF auxiliary table is screwed, because the dynamic linker will not be able to link them.
Therefore, it is a good idea to start with a simple static binary. We will use
sed(1) here. When we run it using the NetBSD native function to set up
the stack and CPU register, it is able to start up. Then it gets a SIGSYS
signal and it dumps core on the first system call, because our system call
table for IRIX binaries is still empty. It is possible to check what the
missing system call is with the
ktrace(1) command on NetBSD:
$ ktrace -di /emul/irix/bin/sed /etc/passwd Bad system call (core dumped) $ kdump 1209 ktrace EMUL "netbsd" 1209 ktrace CALL execve(0x7fffea5f,0x7fffe99c,0x7fffe9a4) 1209 ktrace NAMI "/emul/irix/bin/sed" 1209 sed EMUL "irix o32" 1209 sed RET execve 0 1209 sed CALL #4 (unimplemented write) 1209 sed PSIG SIGSYS
Most of the system calls first used by
/bin/sed are plain SVR4, so it
was easy to emulate them: just copy the system call definition from
sys/irix/syscall.master, issue a make to refresh
the files generated from
syscall.master, rebuild a kernel, reboot, and retry.
Within a few minutes, it was easy to get IRIX's
/bin/sed nearly working. The
next problem was to have it take its arguments correctly.
For static binaries, the stack is used to transmit arguments and the environment to the user program. The way it should be done is documented in the SVR4 ABI MIPS processor supplement.
In This Series
IRIX Binary Compatibility, Part 6
IRIX Binary Compatibility, Part 5
IRIX Binary Compatibility, Part 4
IRIX Binary Compatibility, Part 3
IRIX Binary Compatibility, Part 1
The NetBSD kernel uses a function pointed to by the
es_copyargs field of
execsw to set up the program stack on startup. Because
ports conform to the SVR4 ABI, we could have expected the NetBSD
version of this function (
to just work with IRIX binaries. Unfortunately, this is not true. Using the
elf32_copyargs function with static o32 IRIX binary such as
showed weird behavior with the way argument read: sometimes
was reading the arguments correctly, sometimes it was not. The behavior
was dependent upon the argument length. This suggested that something in
the stack had to be aligned on a particular boundary; with some argument
lengths it was being aligned, and with others it was not.
I already had to face this kind of situation when working on Linux/PowerPC
binary compatibility on NetBSD (read the whole story). However, the situation is different here: IRIX is a closed source proprietary OS; therefore it is not possible to grab kernel sources and look at the way the IRIX kernel sets up the stack. Worse,
because I was not able to build static binaries, it was impossible to make
a static test program that dumped the stack and displayed
envp to check what was wrong in the way the stack was set up.
The solution was to use
gdb we can run
/bin/sed on IRIX, set a breakpoint at the beginning of the program and then examine the stack.
The first thing to know is the program startup address. This information
can be obtained using objdump on IRIX's sed:
$ objdump -f /bin/sed /bin/sed: file format elf32-bigmips architecture: mips:3000, flags 0x00000102: EXEC_P, D_PAGED start address 0x100000c0
Then we can start
gdb and set our breakpoint. It seems that it is not
possible to break on the program's first instruction, but we can break on
the second instruction. On the MIPS, all instructions are four bytes long,
hence the second instruction is four bytes away from the program's start
0x100000c0 + 0x4 = 0x100000c4:
$ gdb /bin/sed (gdb) b *0x100000c4 Breakpoint 1 at 0x100000c4 (gdb) run aa aaa Starting program: ./sed aa aaa Breakpoint 1, 0x100000c4 in ?? () (gdb) x/16wx $sp 0x7fff2fa0: 0x00000003 0x7fff3000 0x7fff3027 0x7fff302a 0x7fff2fb0: 0x00000000 0x7fff302e 0x7fff3057 0x7fff306a 0x7fff2fc0: 0x7fff30a7 0x7fff30b4 0x7fff30be 0x7fff30d4 0x7fff2fd0: 0x7fff30e1 0x7fff30f6 0x7fff3109 0x7fff3113
In this dump, we recognize a standard startup stack layout--the
argc value (three arguments: '
aa', and '
aaa'), followed by the
argv array (a NULL terminated array of pointers to the argument strings), and then the
envp array (a NULL terminated array of pointers to the environment
strings). There are a lot of environment strings here, hence we do not see
the trailing NULL here, it is a bit farther in the stack dump.
It is possible to dig up the value of an argument with its address:
(gdb) x/s 0x7fff3000 0x7fffea60: "/bin/sed" (gdb) x/s 0x7fff3027 0x7fffea72: "aa"
Dumping the stack with various arguments to
/bin/sed, it is possible to
discover that, for an IRIX binary, the
argv must be aligned on a 16-byte boundary. The IRIX kernel sets the stack that way, and IRIX binaries depend
on this particular layout.
It was not possible to modify
elf32_copyargs() to implement this particular
behavior because it is also used by native NetBSD binaries. Hence, the
solution was to duplicate what was done in
elf32_copyargs() in an
irix_copyargs() function, which can be found in
irix_copyargs() function just does
elf32_copyargs() job and it enforces the 16-byte alignment of
irix_copyargs() function must be used in the
of the struct
execsw for IRIX in
With this adjustment, static IRIX binaries were able to read their arguments and environment correctly.
The ELF auxiliary table is used by dynamic linkers to gather
information about the program they are about to link and launch. It is a
table of pairs (type, value) stored on the stack. These pairs are called
auxiliary vectors. Documentation of the available vector types can be found
#define AT_NULL 0 /* Marks end of array */ #define AT_IGNORE 1 /* No meaning, a_un is undefined */ #define AT_EXECFD 2 /* Open file descriptor of object file */ #define AT_PHDR 3 /* &phdr */ #define AT_PHENT 4 /* sizeof(phdr) */ #define AT_PHNUM 5 /* # phdr entries */ #define AT_PAGESZ 6 /* PAGESIZE */ #define AT_BASE 7 /* Interpreter base addr */
The ELF interpreter will use these to discover the address of the ELF program header, which lists the executable ELF sections in the executable, for instance. This is used to discover the list of required shared libraries and the symbol table location.
We use the same stack dumping technique as described in the previous section
to discover what information the IRIX kernel lists in the ELF auxiliary
table. Things are just a bit different: when running a dynamic executable,
the kernel launches the interpreter first, not an ELF section from the program.
Therefore, we cannot just set a breakpoint at a collected address using
objdump(1) on our program. Instead we need to set the breakpoint at the
interpreter's entry point. On IRIX, the interpreter is
hence we can discover the entry point of a dynamic binary by using
$ objdump -f /lib/libc.so.1 /lib/libc.so.1: file format elf32-bigmips architecture: mips:6000, flags 0x00000150: HAS_SYMS, DYNAMIC, D_PAGED start address 0x0fae0774
We just have to set up the breakpoint at
0x0fae0778, and we can see the stack
as it is set up by the IRIX kernel. The ELF auxiliary table appears after the
envp array. IRIX sets up the following vector types:
AT_PAGESZ. Once we know what should be in
it, then it is quite easy to add the code to copy this table to our
function. This enables dynamic binaries to start, but they quickly die,
complaining that a mysterious system call named
syssgi() was not implemented.
We will have a closer look to
syssgi() in a future article.
The CPU registers are set on startup by the function pointed by the
e_setregs field of the struct
emul. IRIX emulation uses the NetBSD native function, which is simply called
setregs(), and this works for o32.
For n32 binaries, however, using
setregs() led to an unpleasant crash before the first system call. The crash was caused by a SIGILL signal. This signal is sent by a trap raised by an illegal instruction.
gdb is a good tool to help us understand what went wrong here. There are
several reason why programs could issue an illegal instruction: Did we start the program at its entry point? Or did we corrupt the stack and return in a random place after a function call? Is it another problem?
Using gdb on a static n32 binary : $ objdump -f sh (snip) start address 0x0e00ba44 $ gdb ./sh (gdb) b *0x0e00ba48 Breakpoint 1 at 0xe00ba48 (gdb) run Starting program: ./sh Breakpoint 1, 0xe00ba48 in ?? () (gdb) info registers zero at v0 v1 a0 a1 a2 a3 R0 00000000 00000000 00000000 00000000 7fffe9d8 00000000 00000000 0e090000 t0 t1 t2 t3 t4 t5 t6 t7 R8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 s0 s1 s2 s3 s4 s5 s6 s7 R16 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 t8 t9 k0 k1 gp sp s8 ra R24 00000000 00000000 00000000 00000000 00000000 7fffe9d8 00000000 00000000 sr lo hi bad cause pc 0000ff13 00000000 00000000 0e00ba44 00000024 0e00ba48 fsr fir fp 00000000 00000000 00000000 (gdb) x/26i 0xe00ba44 0xe00ba44: lui $a3,0xe09 0xe00ba48: lw $a0,0($sp) 0xe00ba4c: addiu $a3,$a3,200 0xe00ba50: lw $a3,0($a3) 0xe00ba54: addiu $a1,$sp,4 0xe00ba58: li $at,-16 0xe00ba5c: lui $gp,0xe09 0xe00ba60: and $sp,$sp,$at 0xe00ba64: addiu $a2,$a1,4 0xe00ba68: sll $v0,$a0,0x2 0xe00ba6c: addiu $gp,$gp,27744 0xe00ba70: addiu $sp,$sp,-16 0xe00ba74: bnez $a3,0xe00ba88 0xe00ba78: addu $a2,$a2,$v0 0xe00ba7c: lui $at,0xe09 0xe00ba80: addiu $at,$at,200 0xe00ba84: sw $a2,0($at) 0xe00ba88: lui $at,0xe09 0xe00ba8c: addiu $at,$at,15424 0xe00ba90: sw $a0,0($at) 0xe00ba94: lui $at,0xe09 0xe00ba98: addiu $at,$at,15456 0xe00ba9c: sw $a1,0($at) 0xe00baa0: sd $zero,8($sp) 0xe00baa4: jal 0xe0715dc (gdb) c Continuing. Program received signal SIGILL, Illegal instruction. warning: Hit heuristic-fence-post without finding warning: enclosing function for address 0xe00baa0 0xe00baa0 in ?? ()
The problem was caused by the
sd instruction, which stands for "store double word". The credits for debugging this go to Wayne Knowles: the
is only allowed when the processor is running in 64-bit mode. Execution of
sd in 32-bit mode causes a reserved instruction exception. The kernel turns
this exception into a SIGILL signal.
The solution is to set up the processor in 64-bit mode for execution
of n32 binaries. This is done by setting a flag in the SR register. This flag
MIPS3_SR_UX in NetBSD's
sys/arch/mips/include/psl.h. The fix to
this problem is therefore to write a
setregs_n32() function to set up the
registers for IRIX n32 binaries. This function just sets the
flag and then calls the regular
In the exec switch from
sys/exec_conf.c, IRIX has two entries: one for
o32 binaries, which uses an o32 probe function called
irix_elf32_probe_o32(); and the other the
emul (defined in
sys/compat/irix/irix_exec.c). This struct
emul contains a pointer to
setregs(). The other entry is for n32 binaries; it uses
irix_elf32_probe_n32() and the
emul_irix_n32 contains a pointer to
setregs_n32() as the function to set up CPU registers.
setregs_n32() function, n32 binaries are able to start up and do
a few system calls. They crash on the first system call manipulating 64-bit data, which is the case for
stat(). Simple static n32 are hence actually able to work because they do not need
mmap() to link.
It is possible to run a static n32
/bin/sh and use it to launch shell commands,
but it quickly dies (as soon as it hits a system call that uses 64-bit data, in fact).
To reliably run n32 binaries, we need 64-bit support in the kernel. This will be discussed in more details in a later article.
Emmanuel Dreyfus is a system and network administrator in Paris, France, and is currently a developer for NetBSD.
Return to the BSD DevCenter.
Copyright © 2009 O'Reilly Media, Inc.