ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


TTYs and X Window: Unix Now and Then

by Hubert Feyrer
03/22/2001

In NetBSD, the wscons console driver can emulate either a VT100 or a Sun display. I was recently asked by a novice user why there was no VT220 offered, and what the difference was between the terminal type emulated by wscons and the terminal type set in /etc/tty. I realized that a discussion of the history of terminal emulation would likely be useful to other BSD developers.

First, I'll describe the type of hardware the Unix operating system was designed for. Then I'll discuss its support for applications that talk to other types of hardware, and how information on this hardware is determined and set. Then I'll jump to the modern world to explain how the X Window System works in comparison to the terminal-based I/O of traditional Unix systems. After that, we'll look at today's common terminal emulation applications to show that the ancient Unix terminal system is still in use today.

Examples in this text use the NetBSD multiplatform operating system, but the concepts are common to all Unix systems -- from older versions such as PDS/Cadmus, to modern Unix versions such as BSD, Solaris, and Linux.

Ancient Unix systems

Back in the Stone Age of Unix, computer systems were usually mainframes -- big boxes of blinking lights with memory, mass storage, and computing units that ran processes started by users or operators. As the hardware was very expensive, the systems were true multiuser systems, with many people interacting with the system at the same time. What it usually didn't have -- unlike today's Unix workstations -- was a fixed monitor and keyboard. Instead, issuing commands to the machine and retrieving output was done over serial lines, using teletypewriters first, and CRT (cathode ray tube) terminals later.

Dumb Terminal
A Unix dumb terminal connected via serial lines.

Teletypewriters -- the "ttys" in Unix -- are old electronic typewriters that send keys pressed over a serial line to a host. Replies are sent back to the teletypewriter character by character over the serial line, with a built-in printer putting the reply on paper. For a picture of a teletypewriter, see the lower left corner of the image at the Unix User Group homepage.

Early support of input devices

Some typewriters only used capital letters, while others were able to input and print both cases. Rarely found in today's Unix flavors is the traditional way to tell the system to use, or not use, mixed case: If you entered your login name in all lowercase characters, the system assumed your terminal was capable of doing both uppercase and lowercase. But if you entered your login name in all uppercase characters (or your teletypewriter sent everything you typed in uppercase), the system assumed your terminal could only do uppercase, and printed (real) uppercase characters preceded with a backslash in its reply.


AIX Version 4
(C) Copyrights by IBM and by others 1982, 1996.
login: ESP
3004-033 YOU LOGGED IN USING ALL UPPERCASE CHARACTERS.
         IF YOUR WORKSTATION ALSO SUPPORTS LOWERCASE
         CHARACTERS, LOG OFF, THEN LOG IN AGAIN USING
         LOWERCASE CHARACTERS.
\E\S\P'S \PASSWORD:

This behavior is still described in chapter 11 of 4.4BSD's Unix User's Supplementary Documents -- the "edit" tutorial (see /usr/share/doc/usd/11.edit).

How a terminal works

More advanced typewriters offered the ability to change from black to red ribbon, and boldface was available everywhere by overstriking -- printing a character, moving the printer head back (by printing a "backspace" character), and then printing the same letter again on the same spot, resulting in a bold look. Not too long after that, the "printer" part of the teletypewriter was replaced with a CRT. The CRT displayed characters by moving an electron beam on a fluorescent screen of glass with a phosphorous coating, giving them a greenish look. The combination of a CRT and keyboard is now widely known as a "terminal."

The functionality is the same as for the teletypewriter -- keystrokes get sent to a processing host over a serial line, and replies are sent back character by character, and the terminal displays them. The terminal itself has no processing capabilities, and is therefore often described as a "dumb" terminal. When the host sends an "f", an "o" and another "o", the screen will say "foo", with the cursor blinking after the second "o", waiting for further output from the host. The character set was 7-bit ASCII, which is still in widespread use in the Unix world today, and which is only slowly being superceded by various competing standards like ISO 8859-1 and Unicode.

Command sets

Staying at the blinking cursor after our "foo" a bit longer, the host has to send a command character that says "move to the next line," often even followed by another command character asking it to "move to the start of line," depending on the terminal's setting. These control characters are usually known as "carriage return" (CR) and "line feed" (LF). Command characters were originally encoded as ASCII values below 32 (see man 7 ascii for your nearest ASCII table).

As other control characters for flow control were used, it quickly became obvious that 32 characters were not enough. Command sets that consisted of several characters were used next. These often had a control sequence introducer (CSI), followed by one or more command bytes (for example, "move cursor") and data bytes (row and column). What made this worse was that manufacturers of terminals started to invent their own command sets that were not compatible with other vendors. Representatives of the more prominent command sets are the DEC (now Compaq) VT100/VT220 escape sequences (named so because they had an "escape" character, ASCII 27, as the first character), ANSI codes, and many others.

Escape sequence
Sending an escape sequence to the terminal.

Handling many different terminals in applications

As Unix machines were used as real multiuser machines, terminals from different vendors were hooked up using the standard serial protocol. However, applications that wanted to use the different terminals' special capabilities had to handle these differences in some way. Compiling hard-coded support for a certain terminal type was not an option as it meant adding a number of programs for each new terminal type -- a maintenance nightmare.

A different approach was made by putting a "translation table" that defined common operations in between, and a mapping to the vendor-specific command sets. The translation table is part of the host's operating system. When an application running on the Unix host wanted to clear a terminal's screen or move the cursor, it looked up the terminal-specific command via a system or library call, and then sent that string to the terminal. There's no need in the application to keep terminal-specific code, and if a new terminal gets added, only the translation table needs to be adjusted. A big win!

Terminal Translation Table
The Unix terminal translation table for boldface.

In today's world there are two libraries available for mapping terminal attributes. One is the terminfo library found on many System V and Linux systems, the other is the BSD-originated termcap library which stores all the translation information all in a single file, usually /usr/share/misc/termcap. Be sure to have a look!

$TERM and ttys

There's one question left: If a user logs in from one terminal, and starts an application, how does the system know what terminal type to emulate and what command sequences are needed? The answer is that each terminal is hooked up to a certain serial port (/dev/ttySomething), and the /etc/ttys file tells the terminal type. (Note: This is for the BSD world; the System V world uses /etc/inittab for a similar purpose.)

/etc/ttys
Explaining the Unix /etc/ttys file.

This is from a NetBSD 1.5/SPARC64 system, with ttya and ttyb being the machine's two built-in serial ports, and the "console" line being used for all system input and output. The first column specifies the terminal line, and third column, "type", tells the type of the terminal -- "VT220" for everything but the system console here. Have a look at your ttys(5) man page for all the details.

The terminal type from column 3 is passed through the login process to the user's login shell, and by that to any applications the user starts. The application then uses termcap to look up the specific command sequences they may need. When someone logs in from a terminal hooked up to ttya, it is assumed a VT220 (or compatible) terminal is used.

The information on the terminal type is kept in the TERM environment variable, which can be set to a different value, if necessary. In addition, the login terminal line can be determined with the tty(1) command:

Exporting Terminal Type
Exporting the terminal type.

In the above example, I'm logged in on the ttya serial port. I do not have a (hardware) terminal, but use another computer that's connected to ttya via a serial line, and that has a terminal emulation program running. The terminal program talks to the Sun over the serial interface, displays the characters it sends, and interprets any special command sequences. As I know from my terminal program's handbook, it does not emulate a VT220 properly but works fine for VT100 commands. The TERM variable is set here to reflect this in order to make applications behave properly.

Back to modern life

Modern workstations don't have separate hardware for the computer itself and one or more terminals. Instead, a CRT is usually directly attached to a graphics board, and input is done with keyboard and mouse, which relieves the necessity to do user input and output over a slow serial line.

Console drivers and virtual consoles

The model described above -- where the terminal is able to handle a certain set of commands -- still applies to modern console drivers. The console driver is a part of the kernel that handles input from the keyboard and output to the display. It passes data via a software terminal device, using the same interface inside the kernel as a serial device would, and that of course needs to be configured in /etc/ttys. Output to the console is interpreted, and just as ancient terminals, today's consoles handle a wide range of character sets, among them various VT100/VT220 variants, ANSI, or more specialized ones like the FreeBSD cons25 console.

Most popular PC operating systems offer another mechanism called a virtual console which allows a user sitting in front of a terminal to view more than one "virtual" console, usually by pressing CTRL+Alt and a function key (F1, F2, etc.). The system handles these as separate lines, and each must be configured in /etc/ttys.

The NetBSD operating system, which uses wscons as the console driver on many ports, offers a way to configure which terminal type it should emulate, with vt100 being used on most ports, and the Sun3, SPARC and SPARC64 port using sun.

Graphics

We have talked about text/console data being passed between computer and terminal, and displayed on the console. However, the question about how graphics work still stands unanswered.

Let's go back in time a bit for this. When terminal vendors added new capabilities to their hardware, one of the new features was to not only display 80x24 text information, but also bitmap data. Of course the command sequences for this were as non-standard as they could be. There was no application-level standard like terminfo or termcap that applications could use to make device-independent use of the capabilities of these graphic terminals.

Eventually, machines were built that had the main unit and graphics hardware connected with a faster connection than a serial line. At the same time, networking several main computer units to local area networks for data exchange and communication became common.

A group of researchers at MIT were exploring modern computer capabilities, and came up with a graphical user interface based on modular parts, just like the classic setup between a computer and terminals. It included an application process that issued commands like "clear screen" or "open window." The commands were sent to a "presentation server" that decoded the commands, and did whatever steps were necessary to clear a bitmap or draw a rectangular area with a border. The "presentation server" usually got its commands from a local or remote computer system, and it knew how to talk to the graphics board for output and keyboard and mouse for input. Events generated by the user operating the mouse and keyboard were sent back to the application process for interpretation and action.

Presentation and application separation
The separation of presentation and application.

Calling names

Are you familiar with the scheme described so far? No? Maybe you will be when we call the components by their names:

The window system described above is the X Window System. The application process is usually known as "X client," and the "presentation server" handling access to devices is the "X server." Communication between X client and X server is done using the X Window System Protocol, which is based on the TCP/IP stack.

Using Unix's capability to run multiple processes, one machine running X clients can issue display requests to one or more X servers running either on the local machine using the local graphics board, the keyboard and mouse, or using an X server that's running on a remote machine.

And just as vendors started to sell VT220-compatible ASCII terminals when that was mostly a standard, there were other terminals called "X terminals." They were equivalent to their text-only cousins which only knew how to handle input and output, without any capabilities to run actual application code themselves. Usually, X terminals consisted of a monitor, a keyboard and a mouse, and a small box that ran only the X server process. The X terminal's X server knew how to talk to keyboard mouse and graphics card, and passed all input and output to an X client.

An alternative to X terminals are today's Unix workstations, which have the processor, graphics card, and I/O-hardware all built into a PC or a traditional workstation machine. The workstation then runs its local instance of Unix, including the X server process to talk to the hardware and a number of X client applications, all in one case.

An X terminal emulation application -- xterm

So far, we have talked about the classic terminal concept of Unix, and about the X Window System, which follows similar principles. Now that we have the X Window System, anyone can turn a PC into a workstation -- the base for modern user interfaces like GNOME and KDE is there. We can forget about all that old cruft about terminals that pass single characters back and forth, escape sequences, and more!

But... Can we really? Let's see how communication with the Unix system works today.

Most interaction with the operating system today still happens via a command-line interface, which is used to type commands, start processes, and view their output, either via X or on the command-line environment that the command was started from. That "command-line environment" is usually a shell running either on the system's console, one of its virtual consoles as described above, or a terminal window on the X desktop.

If the system console or virtual consoles are used, it's quite obvious that the traditional terminal handling is still needed, as it is the base for input and output done by all commands. This applies to both line-oriented programs like ls(1) or cat(1) as well as so-called "screen-oriented" programs like the "vi"sual editor vi(1) and the top(1) process monitor, which still use system libraries like termcap or terminfo to address the console's capabilities.

But we just agreed that no one needs consoles anymore, and that we have the X Window System to interact with the system. There, we open up a terminal window, using programs like xterm, or one of its newer cousins like the KDE "konsole" or GNOME's "gnome-terminal." The programs' names make it quite obvious what they do, and if we take a closer look at how things work, it will be even more obvious where they got their names from.

In the following example, we will use xterm, but others do about the same. When you are using the X Window System and start xterm, a rectangular window opens, with a "shell" command-line interpreter running. You can type commands into the shell window, and output from the shell commands will be displayed in the xterm window. What's happening in more detail is, that the X server takes the keyboard events, and sends them to the xterm process.

The xterm process has a shell process connected to it. The ASCII presentation of the keys pressed are sent from the xterm process to the shell process -- just like virtual consoles and ancient, dumb serial hardware terminals. The shell process reads its standard input, processes the commands given, and output is sent back to the xterm via the standard output channel.

What happens if you run a program that needs to clear the screen or move the cursor to a certain position? Just as in the cases described above, the application knows which terminal type it's connected to, it uses the termcap or terminfo libraries to query that terminal's capabilities, and then sends the necessary escape sequences to the xterm. The xterm process recognizes the command sequences and interprets them, but does not send character-generating events to the X server for display. Instead it does what the command sequence says -- a "clear screen" sequence is sent to the X server as a command to draw a plain white rectangle, and another small black rectangle in the upper-left corner, representing the cursor.

X Window Protocol
Two apps using the X Windows protocol.

Two details are worth having another look here. First, communication between the xterm and shell process. This is done via Unix's ordinary terminal input/output routines -- no special mechanism was invented in the X Window System for user interaction between the X server and an application running in a command-line environment, with the xterm program "translating" between the two environments. This means that the terminal handling that the Unix operating system has had for more than 30 years now is still useful even in modern environments.

The other detail that's interesting here is the interaction between the xterm process and the X server. The xterm process is just like any graphical application, it sends drawing requests to the X server -- a big white-filled rectangle, a small black-filled rectangle, commands to move parts of the on-screen image around (such as scrolling), of course use the X font mechanism to use a certain font and display characters from it on the X "presentation" server. Although the xterm terminal program does not provide special command sequences, xterm allows you to change the font the X server to use, and a number of other things. To try this, hold down the control-key and press one of the three mouse buttons at the same time -- you'll see that the pop-up-menu of CTRL+left mouse allows selecting the font that the xterm process asks the X server for.

The circle closes

As you see from the previous discussion, graphical user interfaces did not supersede the traditional concepts of the Unix operating system, but interact with them so that existing applications can be used in window environments without any migration effort. The concept of terminals once used as hardware and now mostly as software continues to live on, and the circle closes, giving both "old" and "new" technology an equal right to exist. They do not compete but play together perfectly well.

The simplicity and usefulness of Unix has allowed the system to grow and gain maturity over all these years.

Recommended Reading

Richard Stevens: Advanced Programming the Unix Environment; Addison Wesley, 1992. ISBN 0-201-56317-7.

I'd like to close this overview by mentioning that I left some things like networking out deliberately -- the (pseudo) terminals that are used in communication between xterm processes and their shells are also involved in network communication for things like rsh and ssh, and to pass data between different machines as if the data source and destination are processes on the same machine. Also, the description of the X Window System left out many of the concepts of the window system. Window managers and session management are only two of many things that come to mind, but I'll leave these subjects for another time. I hope you had as much fun reading this as I had writing it. Enjoy!

Hubert Feyrer works on operating systems, databases, and artificial intelligence at the Fachhochschule Regensburg.


Return to ONLamp.com.

Copyright © 2009 O'Reilly Media, Inc.