oreilly.comSafari Books Online.Conferences.


Achieving Low-Latency Response Times Under Linux

by Dave Phillips

Latency can be defined as the elapsed time (delay) between the generation of an event and its realization. If the delay is great enough to be perceptible, you have a latency problem.

For instance, when you play an interactive game, you expect no delay between the initiation of an event (turning a steering wheel or firing a weapon) and its realization (the vehicle turns when you turn the wheel, the weapon fires when you pull the trigger). You expect the event to occur instantaneously, or nearly so. The MIDI keyboard to synthesizer (or soundcard) connection is another simple example: You press the key, you expect the sound to occur immediately. If system latency is great enough, there will be a perceptible delay between the keypress and the resulting sound.

In computer systems, multimedia software and other real-time applications are severely affected by long latency times, but system performance at all levels can be compromised. Video and animations suffer frame-loss, and sound applications suffer audio dropouts. Extreme high latency may endanger data integrity during disk read/write/copy activity. Obviously, lowering the latency response time in a system enhances I/O through the entire system. This is good.

Any real-time or interactive software hopes for zero perceptible delay, but short of quantum connections, it's impossible to completely eradicate latency in a system. However, latencies below a certain threshold may be referred to as measurable but not perceptible. So when does measurable latency become perceptible?

In the audio domain, studies have indicated that the ear is sensitive to timing differences at the millisecond level, perhaps even down to a single millisecond. However, latencies under 7 msec are not typically perceptible and are considered acceptable for desktop and semiprofessional applications. Systems achieving average latencies of less than 5 msec should be considered ideal platforms for professional latency-critical applications.

Frustrated expectations lead to dissatisfied and unhappy users. Unhappy users lead to unhappy programmers. I have some good news for users and programmers alike: A simple user-applied patch to the Linux kernel sources (and an uncomplicated disk tune-up) can reduce latency times to under 4 msec. Programmers can easily exploit this new low-latency condition by setting scheduler priority from within their applications, achieving performance with latency well within professionally acceptable limits. This article will show you what's involved and how you can do it yourself.

What causes latency in a system?

Latency is introduced into a computer system by a variety of factors. Some are hardware-related and can be improved by simply adding a faster disk, speedier CPU, more RAM, or a better video card. Other factors are found within your software. At the application level, programmers may miss opportunities for more efficient data input/output. At the system level, overall load will certainly have an impact on latency. And within the operating system itself, latency may be adversely affected by an accumulation of untuned or poorly tuned scheduling requests.

In the MIDI keyboard example noted above, several possible sources of latency exist: the keyboard's physical response time, the type and density of the MIDI message stream (coupled with the fact that MIDI is a serial protocol), and delays introduced by the synthesizer's processor scheduling factors. Engineers devote considerable efforts to minimizing latencies at every point in a time-sensitive system, and the typical acceptable latency on a professional MIDI synthesizer ranges between 2 and 5 msec.

There are patches designed to reduce latencies introduced by various activities of the Linux kernel. Although I'll be focusing on using these patches to improve audio software, these enhancements also apply to other forms of streaming and multimedia.

How can latency be minimized?

You can easily enhance hardware performance by installing faster disks, memory, processors, and video cards. But hardware upgrades can't defeat the effects of latencies introduced at the software level, so what can you do with software in order to reduce latency times?

Some obvious solutions include eliminating all non-necessary activity, particularly any disk-intensive or CPU-intensive work. Staying off any networks is probably a good idea too, especially if you intend to run an application as root.

Under certain circumstances it may be possible to run an application in a condition known as setuid root. As the name implies, this condition sets the user's ID to root status, raising the application's execution priority to real-time scheduling. Unfortunately, setuid applications can be a security hazard on networked machines, so check with your system administrator for advice on running setuid root applications on a network. Standalone and single-user machines can freely ignore the security risk (off-line, of course) of setuid root, giving multimedia applications performance priority not available otherwise. Some Linux audio applications such as XMMS and ecasound already provide a switch for setting execution priority to real-time. Other applications such as SoundTracker and the "unofficial" Linux version of Csound can be compiled with setuid enabled for better scheduling.

SCSI disks are often preferred for multimedia work, and conventional wisdom advises the purchase of SCSI disks for best performance in audio/video-intensive applications. However, in an informative on-line article at, D. Glen Cardenas states that "... there is enough evidence to say that either SCSI or IDE can offer the full level of drive performance necessary to take your system to its full potential. Although there are places for SCSI where IDE dare not go, one of them is NOT the digital audio workstation. Here, IDE can outperform SCSI just as often as SCSI can outperform IDE ..."

Whether you own SCSI or IDE/EIDE drives, Mark Lord's hdparm utility is a safe and simple way to increase your disk's performance, turning on support for such enhancements as DMA (direct memory access) and 32-bit I/O. For instance, these settings

hdparm -m 8 -d 1 -u 1 -c 1 /dev/hda

will enable the first IDE/EIDE drive (/dev/hda) with support for DMA, multicount (multiple data blocks transferred per single interrupt), 32-bit mode, and IRQ unmasking. I must stress that running hdparm is essential if you wish to achieve low latency from IDE/EIDE disks. Latency is dramatically increased if you don't run hdparm, so tune that drive! If your disks are IDE/EIDE, you owe it to yourself to use hdparm. It is included with most mainstream Linux distributions, and it even has a very nice manual page (man hdparm) that contains far more information about the utility than I can present here.

Note: Some very old IDE drives may not like hdparm. Be sure to read the hdparm manual page before using it on such a drive. Also, you might want to revisit Rob Flickenger's Speeding Up Linux Using hdparm, published earlier this year on the O'Reilly Network.

After upgrading your hardware and optimizing your system usage, you will still be left with latencies caused by factors within the Linux kernel itself. But thanks to the wonder of open source software, you can even fix your operating system kernel. Roger Larsson and Benno Senoner have designed tools to measure and represent latencies in the Linux kernel. Using these and other tools, Ingo Molnar and Andrew Morton have created simple patches that focus on points in the kernel that most critically affect latency figures. We will take a closer look at one of those patches, but first let's look at the utilities that are used to measure and identify sources of latency in the kernel (and other applications).

Identifying latency sources in the Linux kernel

Ingo Molnar identified six sources of latency in the Linux kernel:

  • Calls to the disk buffer cache
  • Memory page management
  • Calls to the /proc file system
  • VGA and console management
  • The forking and exits of large processes
  • The keyboard driver

Each of these routines delays returning control to the scheduler for several milliseconds, and with enough delay we get audio dropouts and video frame-loss. Fortunately for the Linux community, Ingo didn't stop with merely identifying the sources of latency in the kernel. He created a series of patches designed to remedy the situation, and those patches yielded impressive results. On an unpatched 2.2.10 kernel, latencies measured as high as 150 msec; with Ingo's patch, latency dropped to less than 3 msec, a truly dramatic reduction and definitely within the range needed for professional applications.

Ingo's achievement is especially remarkable with regard to the efforts required to gain low latency on other operating systems, most notably the Microsoft Windows family. In a presentation of the results from a roundtable conference at the February 2000 NAMM (National Association of Music Merchandisers) show, Ron Kuper (CTO of Cakewalk) stated that "... an obtainable target for audio latency under Win2k is 5 msec, even under heavy system loads" (from Audio I/O, Today and Tomorrow). However, in order to hit that target, it is necessary to bypass Microsoft's KMixer (kernel mode audio mixer) in their Win32 driver model, because (quoting Mr. Kuper again) "... KMixer nominally adds 30 msec of latency to audio playback streams. (At present, Microsoft does not provide a method to allow host applications to bypass KMixer.)" (also from Audio I/O, Today and Tomorrow). A plan for a set of IOCTLs (input/output controls) to bypass KMixer was presented at the Windows Professional Audio Roundtable, sponsored by Cakewalk, intended for adoption by the professional audio software industry. Clearly, latency on Windows is a problem commanding the attention of the most serious audio software manufacturers. The comparative ease with which Linux achieves latency even lower than 5 msec ought to be of compelling interest to these same manufacturers and organizations.

Pages: 1, 2, 3, 4

Next Pagearrow

Linux Online Certification

Linux/Unix System Administration Certificate Series
Linux/Unix System Administration Certificate Series — This course series targets both beginning and intermediate Linux/Unix users who want to acquire advanced system administration skills, and to back those skills up with a Certificate from the University of Illinois Office of Continuing Education.

Enroll today!

Linux Resources
  • Linux Online
  • The Linux FAQ
  • Linux Kernel Archives
  • Kernel Traffic

  • Sponsored by: