Linux Compatibility on BSD for the PPC Platform: Part 2
Pages: 1, 2, 3, 4

Back to argument passing

When using ld-2.1.3.so, the argument-passing problem was a bit weird: ld.so was able to link the program, and this meant that it was able to find the program's arguments (if ld.so does not get the arguments, it complains by displaying an error message). That suggested the stack layout for arguments was good. But on the other hand, the program itself wasn't able to retrieve its arguments anymore: When running the argument printer, the program displayed a null **argv. This suggested the stack layout for the arguments was bad.

Running the stack dumper, it was obvious that the program expected its arguments 16 bytes lower than the place they actually were. Modifying the stack layout or the stack pointer did not fix the problem, because if the arguments were set up where the program expected them, then ld.so did not find them, and it was not able to link the program.

In fact, the problem is that ld.so and the executable expected the arguments to be on two different places. Duplicating the arguments was therefore a possible workaround to the problem. With such a duplication, here is the stack layout the kernel produced before transferring control to ld.so:

7fffe9b0 0000 0001 7fff eab0 0000 0000 7fff eab5 ................
7fffe9c0 0000 0001 7fff eab0 0000 0000 7fff eab5 ................

Previously in this series: Linux Compatibility on BSD for the PPC Platform -- The Linux compatibility layer allows BSD to run Linux binary applications. Emmanuel Dreyfus explains how he implemented this on NetBSD for the PowerPC platform.

You can recognize on each line argc (here 0000 0001), the **argv pointer, a null pointer, and the **envp pointer. When the kernel transferred control to ld.so, the stack pointer was at 0x7fffe9c0. Ld.so was able to find its arguments at 0x7fffe9c0, and the idea was that the program would find its arguments 16 bytes lower, at 0x7fffe9b0.

Unfortunately, this does not work, because ld.so makes use of the stack. It uses the space between 0x7fffe9b0 and 0x7fffe9bf, and when it transfered control to the program, the stack layout is like this:

7fffe9b0 0000 0000 0000 0000 0000 0000 0000 0000 ................
7fffe9c0 0000 0001 7fff eab0 0000 0000 7fff eab5 ................

And again, the program was not able to find the arguments, because the place where it expected them is erased by ld.so.

A good solution here would be to understand why ld.so gives a stack pointer that is 16 bytes too low to the program. It was not possible to achieve this, so I had to hack a bad solution. The idea here is that ld.so gives the program a stack pointer which is 16 bytes too low. So if we can regain control after ld.so has done its job, and before the program is actually started, we can adjust the stack pointer so that the program can find its arguments.

The problem is how to get control between ld.so and the program. Because ld.so does not return to kernel mode before launching the program, we have to fool ld.so into thinking it is launching the program, whereas it is actually running our code.

This can be done by setting up an entry in the ELF auxiliary table that describes where the program entry point is. Ld.so then uses that entry to launch the program. We can modify this entry in the ELF auxiliary table so that ld.so will transfer control to a small piece of code we uploaded onto the process stack. This code would adjust the stack pointer and then jump to the real program entry point. This approach is a ugly hack, but at least it worked. Here is the stack pointer adjustment code (thanks to Wolfgang Solfrank for helping me writing it) :

#include <machine/asm.h>
#define LINUX_SP_WRAP_OFFSET 0x10

.globl _C_LABEL(linux_sp_wrap_start)
.globl _C_LABEL(linux_sp_wrap_end)
.globl _C_LABEL(linux_sp_wrap_entry)
_C_LABEL(linux_sp_wrap_start):
  addi  1,1,LINUX_SP_WRAP_OFFSET
  mflr  12
  bl  1f
1:
  mflr  11
  mtlr  12
  lwz  12, _C_LABEL(linux_sp_wrap_entry)-1b(11)
  mtctr  12
  bctr
_C_LABEL(linux_sp_wrap_entry):
.long  0  /* orginal prog entry point. setup by the kernel
*/
_C_LABEL(linux_sp_wrap_end):

Its use is triggered by the LINUX_SP_WRAP macro, which is defined in PowerPC-specific linux_exec.h, just like the LINUX_SHIFT macro. The kernel just copies this code from kernel space to the user stack, sets up the program entry point at the linux_sp_wrap_entry location, and sets the entry point in the ELF auxiliary table to the location on the stack where the code was just uploaded.

We can have a closer look at what the assembly instructions actually do. First, we adjust the stack pointer, which is GPR1, by adding 16 to it. This is done by the addi 1,1,LINUX_SP_WRAP_OFFSET.

Then we load in GPR12, the value at the linux_sp_wrap_entry location. To do this, we will have to tamper with the Link Register, so it is saved prior to that operation and then restored. This is done with the mflr 12 instruction, which saves the Link Register to GPR12, and by the mtlr 12, which restores the Link Register to the value contained in GPR12.

The next goal is to get the value at the linux_sp_wrap_entry address in GPR12. By the bl 1f instruction (the f stands for the next label 1), we branch to label 1, and we save the Program Counter into the Link Register. mflr 11 copies the value contained in the Link Register into GPR11. We now have the address of label 1 in GPR11.

The difficult part is the lwz 12, _C_LABEL(linux_sp_wrap_entry)-1b(11) instruction, which adds the difference between the address of linux_sp_wrap_entry and the address of label 1 (the 1b stands for the previous label 1) to GPR11 and loads the word located at the resulting address into GPR12. We end up with the linux_sp_wrap_entry address in GPR12.

We copy the value of GPR12 to the CTR register, using the mtctr 12 instruction. Then we can use the bctr instruction, which branches to the address contained in CTRM.

This may look a bit complicated, but this is caused by two problems we need to address: First, we want the code to be able to be relocated (hence the use of the Link Register), and second, we want to do a long branch to the program entry. We must use the CTR to do this long branch.

This hack was rather inelegant, but it fixed the problem. Using this method, it was possible to get arguments in programs linked with ld-2.1.3.so. What is surprising is that it did not break linking with ld-1.7.0.so.

Emmanuel Dreyfus is a system and network administrator in Paris, France, and is currently a developer for NetBSD.

---

Return to ONLamp.com.