Puffy and the Cryptonauts: What's New in OpenBSD 4.3
Pages: 1, 2
Kernel-land and Development
I read that you Improved lkm(4) subsystem on amd64. What is changed?
Mike Belopuhov: Actually I've fixed an lkm(4) subsystem on amd64 :-)
The problem was that the code loaded by the lkm(4) is placed into the kernel_map (that is a main kernel memory allocation area) and this map is located outside of the kernel's jumpable area on the amd64 platform, i.e. kernel can't execute instructions placed in the kernel_map.
The kernel on amd64 uses only relative 32-bit jumps and is loaded into the upper 2 GB of the whole virtual address space. This is caused by a certain restrictions in the gcc compiler. Therefore kernel_map is located below the kernel load address and although the kernel can read and write any memory, it can jump to only the addresses within upper 2 GB.
So I've set up a separate lkm_map that is located in the jumpable area and taught the kernel to reserve space for lkm's there. This effectively allows the kernel to call any function within a loadable module.
This solution is based on the work by NetBSD developers.
Would you like to describe the new M_ZERO flag for malloc(9)? What are the advantages?
Artur Grabowski: Kernel size and simpler code. Since zeroing memory after allocating it is such a common idiom, being able to do both operations in one call saved a significant amount of code in the kernel.
"Many dangerous unsigned comparisons with -1 when checking the results of read and write calls have been eliminated". What was the problem and how did you fix it?
Ray Lai: The read(2) and write(2) return the number of bytes read or written on success, or -1 on error. A lot of people try to read from or write to a buffer like this:
read(..., ..., sizeof(buf)). They then check if there is either an error (returns -1) or an incomplete read or write (returns a value less than sizeof(buf)) like this:
read(..., ..., sizeof(buf)) < sizeof(buf).
The problem is that read(2) and write(2) return signed integers (ssize_t) so that they can return -1, but sizeof() returns an unsigned integer of the same size (size_t). When a signed and an unsigned integer of the same size are compared, the signed integer is first converted to its unsigned equivalent. This results in the -1 becoming the largest possible
size_t (SIZE_MAX), so the check becomes:
SIZE_MAX < sizeof(buf). This check never fails. In a loop, this can cause an infinite loop.
Kenneth Westerback fixed many of these by simply converting the check from "
read(..., ..., sizeof(buf)) < sizeof(buf)" to "
read(..., ..., sizeof(buf)) != sizeof(buf)". A signed to unsigned conversion still happens, but the check works for the majority of the cases (unless sizeof(buf) equals SIZE_MAX).
atomicio() was a created as an elegant wrapper to the clunky read(2) and write(2) API. It was first introduced in OpenSSH and has been copied to a number of other programs (netcat, OpenCVS, sendbug). atomicio() is called like read(2) and write(2), but has an additional argument in the front, either "read" or "vwrite" to specify which action you would like to perform ("vwrite" is used instead of "write" because the prototype for write(2) requires a "const void *" for the second argument. This allows atomicio() to use read(2) and write(2) interchangeably). It takes a size_t as the buffer size and also returns a size_t to show how many bytes it has processed. This makes it possible to check for both error and underflow by simply doing:
atomicio(read, ..., ..., sizeof(buf)) != sizeof(buf).
Management and Ports
Would you like to talk about the work you did on serial console automagic configuration?
Kenneth Westerback: Prior to 4.3 there was limited support for the automatic configuration of serial consoles during the install process. The install script for i386 and amd64 offered the user the chance to automatically enable the first serial device found in the dmesg as a serial port. This had been added in 3.6 or 3.7 to allow easy installation on devices like Soekris that were intended to be used in production with serial consoles. But the support hadn't been extended to other architectures or really completed.
Then David Gwynne (dlg@) got a newish Sun sparc64 box whose serial console speed was 115200 and thus did not work with the default /etc/ttys 'console' entry which specified a speed of 9600. After this bit him, dlg@ expressed an interest in having the speed of the serial console automatically detected and set in the installed /etc/ttys. Theo pointed out that stty(1) was on the install media and 'stty speed' would reveal the speed of the console being used to install. As a result the install script (install.sh) was changed so that it used 'stty speed' to detect the speed of the console being used to install, and modified the 'console' entry of the installed /etc/ttys appropriately. The modification is only done if the 'console' entry is 'on' in /etc/ttys.
This modification made dlg@ happy.
After some thought it was realized that this meant serial consoles 'just worked' on all architectures that make use of the 'console' entry. i.e. all of our architectures except i386, amd64, macppc, alpha and zaurus.
Theo then pointed out that the dmesg on all architectures should have a line that specifies the serial console device, e.g. 'pccom0: console' on i386. It turned out that this wasn't quite true but some quick work by Miod Vallat (miod@) and Mark Kettenis (kettenis@) among others made it so. Then it was noticed that the /etc/ttys entries had unnecessarily diverged in their 'getty' fields. miod@ modified them to all use '... std.9600', give or take a speed or two.
With this done, it was possible to re-do the serial console logic used for i386/amd64 and extend it to alpha, macppc, and zaurus. This generalization even shrank the install scripts.
As a result, on 4.3 most architectures have serial consoles that 'just work'. The others now offer the chance to configure a device as a serial console if a serial device is discovered in the dmesg. The default choice of device and speed will the the current console or the first serial device found, and the default speed will be current console speed, or '9600'.
I saw that your work on ldattach(8) is used to attach a line discipline to a serial line to allow for in-kernel processing of the received/sent data. How does it work?
Marc Balmer: Line disciplines, in order to become active, need a program to open a filedescriptor on a tty(4) device and attach the line discipline to it using the TIOCSETD ioctl(2) call. Traditionally there was one such attachment program per line discipline which would be started either from the command line or in /etc/rc.local. With the introduction of new line disciplines I introduced ldattach(8) as a single program that can be used to attach any of the supported line disciplines. So instead of adding a new attachment program for a new line discipline, ldattach(8) can be extended. ldattach(8) has a slightly different syntax than the older slattach(8) and nmeaattach(8) programs to allow it to be used from the /etc/ttys file as well. For details, please see the manual page.
What is changed in the way sendbug(1) handle comments?
Ray Lai: sendbug(1) has two types of comments, lines that start with "SENDBUG: " or stuff between angle brackets "<...>". Not many people type lines starting with "SENDBUG: ", but angle brackets are used quite often, notably C files, which sometimes have lines such as "#include <stdio.h>". Combine the two, and you have mangled bug reports.
This issue was initially discovered by Deanna Phillips, who noticed that dmesgs included in bug reports were being mangled (notably lines such as "azalia0: <blahblahblah>"). My initial fix was to avoid parsing for angle bracket comments in the dmesg. It wasn't a very elegant fix, and added a bunch of code to determine where the dmesg began, but it worked okay for dmesgs. At the time I couldn't think of a better solution.
One day, Mickey included a diff in a bug reports and noticed that the include headers were being mangled. This time I thought about the problem some more, and realized that comments are only added by the initial bug report template, never by the bug reporter. This meant that if I could just remember the text in the comments, I could strip those out and ignore everything else.
In the latest and greatest version of sendbug(1), that is exactly what it does. It stores the comments in an array and strips out any text that matches. So unless someone includes lines such as "<PR category (one line)>", there should be no more false positives.
What are the major new features and changes included in the recent versions of OpenSSH?
Damien Miller: OpenSSH 4.8 is what was shipped on the OpenBSD 4.3 CD. 4.9 and 5.0 were security releases to fix a bad policy (executing ~/.ssh/rc for forced commands) and an X11 hijacking problem respectively. Both these fixes were backported for OpenBSD 4.3. Otherwise, all three version are pretty much identical.
The big new feature is chroot support for sshd(8). This has been requested by many users, but we have been reticient to do it because it can lead to vulnerabilities if either the sshd-side implementation or the configuration of the chroot environment is incorrect.
Most of the requests for chroot support have been by administrators who want to set up restricted file servers, so to avoid most of the common mistakes Markus Friedl implemented an in-process sftp server. This effectively links sftp-server(8) into sshd(8), rather than forking and executing it as a separate process. This particularly helps chroot setups because no special support files (e.g. /dev nodes) are required.
Users who need to chroot interactive sessions naturally need to configure a full chroot environment including binaries, /dev nodes, libraries and support files. This also applies to file serving via scp(1) too unfortunately, as it isn't as neatly designed or contained as sftp-server(8).
The other changes in this release are minor improvements and bugfixes.
Is the development on cwm(1) going on?
Okan Demirmen: cwm(1) has undergone many important changes in 4.3. In addition to a large code clean-up by oga@, license issues have been resolved. The remaining 9wm code was removed and rewritten by oga@, and with permission of the original author, cwm(1) is now ISC licensed.
With many new cwm(1) users, a few novel features have been added, such as the ability to resize windows and move the pointer with keyboard bindings. Additionally, default keybindings can now be overridden by user defined ones, while also allowing one to 'unmap' a keybinding. This can be useful to resolve a conflict with an application's key mapping.
One very convenient and valuable new feature, 'exec window manager', allows one to either restart cwm(1), or switch to another window manager, namely another version if cwm(1), without restarting the X server. Stay tuned as cwm(1) develops.
What is changed in the ports system?
Marc Espie: Not much is new in the infrastructure this release. Small bug fixes. New ports at a normal rate... 4.3 is close to 5000 packages, and we just crossed that barrier post 4.3.
We are slowly recognizing some weird update scenarios and fixing them.
As far as new ports/significant updates go, things are in ways better shape than in the past. We have a crack team of porters these days. As far as 4.3 goes, we are catching up on GTK and Gnome ports, for instance, thanks to Jasper and friends. Simon keeps Perl ports up-to-date, Deanna and Ajacoutot ensure we have a lot of nice games (highlight on Micropolis, the freed Simcity that Deanna ported), well, there are a lot of others I'm not citing by name who are doing an excellent job in keeping us up-to-date and improving a lot of things: Kurt is doing solid work on Java and foundation libraries, for instance. Our German quality insurance team makes sure nothing unseemly passes through, and the Swiss take care of our database needs.
Together with Mark Kettenis, we spent a long time tracking an obscure segfault in KDE4. Turns out that, with pthreads, every thread but the first had an incorrectly aligned stack on i386. Basically, there was a mix-up in the code between the stack and the frame pointer, with the result that the stack was always in the middle of a 16 byte block. This is usually of little consequence, except for vector operations on mmx and the like which expect a very stringent alignment. By the way, qt4 rocks: they have specific code in there to speed up most graphics operations depending on the processor...
You might think, this is so unlikely to happen, that it is just an obscure bug-fix, but in reality, this pattern (multi-threaded app, accelerated vector operation) is at the core of most video players. Turns this is a big reliability fix for vlc, xine, and friends.
In OpenBSD tradition, there's also a lot of background work that you won't see, as a user, in 4.3, but which is getting stuff prepped for the future...
As far as I'm concerned, I've spent a lot of time on make, recently. This is work that is not yet finished, but I've fixed 90% of the issues that prevented the use of parallel make to build anything non-trivial (and I really hope to fix the remaining 10% before 4.4). As a result, in 4.3, you can try to run make -j on the src tree, and there's a good chance it will work (some races still are not fixed, so depending on your machine, it may still fail). As far as I know, it works all the time on the xenocara tree, and it works on quite a few ports.
This is a huge boon for various work, including ports, since we can use a lot of machines to their full potential. You'll probably notice that SMP support across a wide variety of platforms has improved tremendously recently.
Yes, this is something that does not concern users directly. You can use packages, you don't have to recompile everything all the time. But for development, having a compile-test-tweak cycle that goes n times as fast is a tremendous help.
Federico Biancuzzi is a freelance interviewer. His interviews appeared on publications such as ONLamp.com, LinuxDevCenter.com, SecurityFocus.com, NewsForge.com, Linux.com, TheRegister.co.uk, ArsTechnica.com, the Polish print magazine BSD Magazine, and the Italian print magazine Linux&C.
Return to ONLamp.