An Introduction to the Twisted Networking Frameworkby Itamar Shtull-Trauring
Editor's note: In his upcoming session at O'Reilly's Emerging Technology Conference, Itamar will be showing attendees, step by step, how to use Twisted to develop networked applications. If you want to learn how Twisted can help you save large amounts of development time, don't miss his tutorial, slated for the opening day of Etech 2004.
Twisted is an open source networking framework, implemented in Python. It is designed to support both clients and servers and run on multiple operating systems and platforms. This article is a brief introduction to Twisted's capabilities and design goals.
Network programs are complex systems. They may need to run on multiple operating systems, support multiple protocols, operate in diverse environments, and integrate with existing systems. A mail client will typically need to support SMTP, POP3, IMAP, all running over TCP and TLS, and may need to support additional protocols as well (LDAP, for example). An integrated mail server must use SMTP (which in turn requires DNS client support), POP3, IMAP; but also HTTP for webmail support. Most existing networking frameworks are not suited to building these sorts of applications, as they tend to be either too high level and application-specific (e.g., web frameworks) or too low level and platform-specific (e.g., low-level C networking libraries). Custom development of new protocols is hard with these high-level frameworks, and takes too much effort with low-level frameworks — sending an email requires implementing a SMTP protocol client from scratch.
Twisted is an attempt to build a framework capable of supporting the needs of modern network applications, from simple custom protocol development to large-scale multi-protocol integrated systems. It provides:
- Multiple levels of abstraction, starting with a low-level platform-specific networking event loop, and moving up to generic networking code, common protocol implementations, and high-level frameworks.
- Cross platform (Windows and Unix) support while still allowing platform-specific code (as opposed to Java's design decision to provide only the lowest common denominator).
- Integration and ease of use.
- "Batteries included." The framework includes everything necessary to build a network application.
The Twisted event-driven networking framework (http://www.twistedmatrix.com) is a powerful, cross-platform toolkit for creating networked applications. In this tutorial you will learn, step by step, how to use Twisted to develop networked applications, and how this can save large amounts of development time.
O'Reilly Emerging Technology Conference
Twisted is divided into several packages, each providing different services. Typically, higher-level packages build on lower-level packages, allowing developers to depend only on those packages necessary to their application. The following sections describe some of Twisted's basic packages. There are many more, including a generic, remote-object protocol called Perspective Broker, relational database support, testing and debugging tools, and more.
At Twisted's core, the
twisted.internet package provides a
networking event loop. The event-loop networking model was chosen over threads
as it tends to be more scalable, integrates well with GUI applications, and is
supported by virtually all operating systems. In an event loop, a single thread
responds to network events (e.g. "socket readable"), deals with the
appropriate processing (reading from socket and passing data to the appropriate
handler), and then continues processing.
Much of the effort developing
twisted.internet involved abstracting these low-level,
platform-specific "events," wrapping them with generic APIs, and allowing the use
of the same code, regardless of the underlying platform. While
twisted.internet is not as fast as servers implemented on top of C
event loops, it is usually fast enough for all but the most demanding
High-level APIs for TCP, SSL, UDP, Unix domain sockets and other transports build on the event loop. Protocol implementation is separated from transport implementation, so protocols can run more or less transparently on top of transports of the same kind. A protocol developed for TCP will typically support SSL without any changes. For client connections, Twisted provides high-level functionality, such as reconnection on failure or disconnect, connection timeouts, and the ability to cancel connections, all in an implementation-generic way. The event loop also supports scheduling, thread integration, hooks for non-blocking DNS lookups, and other commonly required tasks.
The event loop is implemented atop operating-system-specific APIs. This
includes the standard
select() system call, available on both Unix
and Windows, as well as other platform-specific event loops. Although the
networking code is generic, the event loop implementation is exposed to
developers who require platform-specific functionality, be it file descriptors
on Unix or events on Windows. In addition, the event loop can integrate with
GUI toolkits (GTK, Qt, Win32, Mac OS X's Cocoa, and so on), allowing generic
networking code to be used for GUI clients as well.
Browsing through freshmeat will show at
least 15 POP3 servers available for download. Different servers will support
different authentication sources and different storage backends, but large
parts of the implementations are duplicated. The
package is an attempt to solve this duplication of effort, by promoting server
development using generic components that can be reused in different contexts.
Authorization is separated from the application code if at all possible. Both
are separated from the protocol implementation. The
package provides classes and interfaces that allow separation of the different
development roles used to build a server in a consistent, flexible way.
These roles include:
Creates the protocol implementation (e.g., POP3), which is designed to talk to an abstract implementation of the backend, defined as an interface (e.g., an IMailbox interface for a POP3 backend).
Creates a backend for a specific protocol. There can be multiple backends, for example, a Maildir IMailbox implementation as well as a MySQL backend implementation created by a different developer.
Creates generic authorization checkers for specific types of credentials, which may be reusable across protocols. For example, a username/password checker that uses
/etc/passwdwill probably work with both POP3 and HTTP.
Tie together a protocol with an appropriate credential checker and backend in order to build a running server. For example, tying a POP3 protocol to a
/etc/passwordcredential checker and a MySQL mail storage.
Several protocols in Twisted use the
including HTTP, POP3, IMAP, and FTP, allowing them to share credential checkers
and to support pluggable backend implementations.
While some applications use Twisted exclusively to develop custom protocols, many applications use common, Internet-standard protocols. Twisted provides implementations of many of the more common ones, typically supporting both client and server implementations. These implementations do not enforce policies (e.g., the FTP code is not limited to serving files from the filesystem), allowing maximum developer flexibility. The supported protocols include HTTP, FTP, DNS, SMTP, POP3, IMAP, SSH, SIP, Jabber, TOC and Oscar (AIM), MSN, IRC, NNTP, and others.
In addition to these low-level protocol implementations, Twisted also
provides higher-level frameworks, implementing the functionality commonly
implemented on top of certain protocols. Two of these frameworks are the
twisted.web package, an object-publishing web server, and
twisted.names, a DNS server framework.
Complex applications are more than just a single protocol implementation. To
aid applications development, tying together multiple subsystems, and to ease
deployment woes, Twisted provides the
Applications are designed as trees of services, providing an abstraction for
related and dependent services.
An application developed in this way, using the interfaces and
infrastructure provided by this package, is easy to deploy. An application
deployment file is constructed either as a specially structured Python script
(which might load configuration information from external application-specific
files) or uses plug-ins for the
mktap utility that comes with
mktap generates "TAP" files, which are simple, pre-configured server
setups. These deployment files can then be run using
command-line tool, which provides logging options, daemonization on UNIX,
profiling, and so on. For example, using the built-in "web" plug-in that
provides a web server configuration, makes starting a web server daemon in Twisted quite simple:
$ mktap web --path=/var/www $ twistd -f web.tap
Both Python deployment files and TAP files can also be used to generated installable Debian or RPM packages. An experimental utility — which should be available in future versions of Twisted — allows easy generation of installable, self-contained Windows NT services.
Software Using Twisted
Several applications, both open source and commercial, use Twisted. A brief overview of two of them demonstrates the power and flexibility of the framework.
Buildbot is a "system to automate the compile/test cycle required by most software projects to validate code changes." When a developer checks in code, the Buildbot build master is notified, and the build slaves (which can be run on different machines, and different operating systems such as Windows or Linux) are notified by the master (using the Perspective Broker protocol). The slaves check out the latest version of the code and run the applicable tests for the software. The status of the builds can be accessed via a web page, an IRC bot, or using a GUI tool, demonstrating Twisted's support for multiple protocols. The Twisted project itself uses Buildbot to run its tests.
Another project that uses Twisted is Quotient, an integrated messaging server. Quotient supports email — it acts as an SMTP server, POP3 server, and IMAP server, and also includes a sophisticated web-based mail-browsing and administration interface. Quotient also integrates VoIP management services using the SIP protocol. Using Twisted's multiple-protocol support and robust server architecture, Quotient can build a superior messaging server with capabilities that would be extremely hard to achieve with traditional single-purpose servers.
Of course, Twisted can be used to implement simpler systems, and has been used to build applications ranging from GUI VoIP clients to complex web applications. To learn more about Twisted, and how you can use it for your software, you can visit the Twisted web site. I will be giving a Twisted tutorial at the Emerging Technology Conference in February, and you can meet most of the Twisted development team at the PyCon conference this March.
Related Projects in Other Languages
- ACE, a C++ networking framework.
- POE, a framework for creating multitasking programs in Perl.
- Tcl's built-in event loop can be used to build network servers.
Itamar Shtull-Trauring has contributed to Python and Zope and is one of the main developers of Twisted.
Return to Python DevCenter.