Python DevCenter
oreilly.comSafari Books Online.Conferences.

advertisement


An Introduction to the Twisted Networking Framework

by Itamar Shtull-Trauring
01/15/2004

Editor's note: In his upcoming session at O'Reilly's Emerging Technology Conference, Itamar will be showing attendees, step by step, how to use Twisted to develop networked applications. If you want to learn how Twisted can help you save large amounts of development time, don't miss his tutorial, slated for the opening day of Etech 2004.

Twisted is an open source networking framework, implemented in Python. It is designed to support both clients and servers and run on multiple operating systems and platforms. This article is a brief introduction to Twisted's capabilities and design goals.

Network programs are complex systems. They may need to run on multiple operating systems, support multiple protocols, operate in diverse environments, and integrate with existing systems. A mail client will typically need to support SMTP, POP3, IMAP, all running over TCP and TLS, and may need to support additional protocols as well (LDAP, for example). An integrated mail server must use SMTP (which in turn requires DNS client support), POP3, IMAP; but also HTTP for webmail support. Most existing networking frameworks are not suited to building these sorts of applications, as they tend to be either too high level and application-specific (e.g., web frameworks) or too low level and platform-specific (e.g., low-level C networking libraries). Custom development of new protocols is hard with these high-level frameworks, and takes too much effort with low-level frameworks — sending an email requires implementing a SMTP protocol client from scratch.

Twisted is an attempt to build a framework capable of supporting the needs of modern network applications, from simple custom protocol development to large-scale multi-protocol integrated systems. It provides:

  • Multiple levels of abstraction, starting with a low-level platform-specific networking event loop, and moving up to generic networking code, common protocol implementations, and high-level frameworks.
  • Cross platform (Windows and Unix) support while still allowing platform-specific code (as opposed to Java's design decision to provide only the lowest common denominator).
  • Integration and ease of use.
  • "Batteries included." The framework includes everything necessary to build a network application.

ETech 2004 Conference

Itamar Shtull-Trauring
Network Programming with the Twisted Framework

The Twisted event-driven networking framework (http://www.twistedmatrix.com) is a powerful, cross-platform toolkit for creating networked applications. In this tutorial you will learn, step by step, how to use Twisted to develop networked applications, and how this can save large amounts of development time.

O'Reilly Emerging Technology Conference
February 9-12, 2004
San Diego, CA


Twisted is divided into several packages, each providing different services. Typically, higher-level packages build on lower-level packages, allowing developers to depend only on those packages necessary to their application. The following sections describe some of Twisted's basic packages. There are many more, including a generic, remote-object protocol called Perspective Broker, relational database support, testing and debugging tools, and more.

twisted.internet

At Twisted's core, the twisted.internet package provides a networking event loop. The event-loop networking model was chosen over threads as it tends to be more scalable, integrates well with GUI applications, and is supported by virtually all operating systems. In an event loop, a single thread responds to network events (e.g. "socket readable"), deals with the appropriate processing (reading from socket and passing data to the appropriate handler), and then continues processing.

Much of the effort developing twisted.internet involved abstracting these low-level, platform-specific "events," wrapping them with generic APIs, and allowing the use of the same code, regardless of the underlying platform. While twisted.internet is not as fast as servers implemented on top of C event loops, it is usually fast enough for all but the most demanding applications.

High-level APIs for TCP, SSL, UDP, Unix domain sockets and other transports build on the event loop. Protocol implementation is separated from transport implementation, so protocols can run more or less transparently on top of transports of the same kind. A protocol developed for TCP will typically support SSL without any changes. For client connections, Twisted provides high-level functionality, such as reconnection on failure or disconnect, connection timeouts, and the ability to cancel connections, all in an implementation-generic way. The event loop also supports scheduling, thread integration, hooks for non-blocking DNS lookups, and other commonly required tasks.

The event loop is implemented atop operating-system-specific APIs. This includes the standard select() system call, available on both Unix and Windows, as well as other platform-specific event loops. Although the networking code is generic, the event loop implementation is exposed to developers who require platform-specific functionality, be it file descriptors on Unix or events on Windows. In addition, the event loop can integrate with GUI toolkits (GTK, Qt, Win32, Mac OS X's Cocoa, and so on), allowing generic networking code to be used for GUI clients as well.

twisted.cred

Browsing through freshmeat will show at least 15 POP3 servers available for download. Different servers will support different authentication sources and different storage backends, but large parts of the implementations are duplicated. The twisted.cred package is an attempt to solve this duplication of effort, by promoting server development using generic components that can be reused in different contexts. Authorization is separated from the application code if at all possible. Both are separated from the protocol implementation. The twisted.cred package provides classes and interfaces that allow separation of the different development roles used to build a server in a consistent, flexible way.

These roles include:

  • Protocol developer
    Creates the protocol implementation (e.g., POP3), which is designed to talk to an abstract implementation of the backend, defined as an interface (e.g., an IMailbox interface for a POP3 backend).

  • Application developer
    Creates a backend for a specific protocol. There can be multiple backends, for example, a Maildir IMailbox implementation as well as a MySQL backend implementation created by a different developer.

  • Authorization developer
    Creates generic authorization checkers for specific types of credentials, which may be reusable across protocols. For example, a username/password checker that uses /etc/passwd will probably work with both POP3 and HTTP.

  • Deployment
    Tie together a protocol with an appropriate credential checker and backend in order to build a running server. For example, tying a POP3 protocol to a /etc/password credential checker and a MySQL mail storage.

Several protocols in Twisted use the twisted.cred package, including HTTP, POP3, IMAP, and FTP, allowing them to share credential checkers and to support pluggable backend implementations.

Related Reading

Python in a Nutshell
By Alex Martelli

twisted.protocols

While some applications use Twisted exclusively to develop custom protocols, many applications use common, Internet-standard protocols. Twisted provides implementations of many of the more common ones, typically supporting both client and server implementations. These implementations do not enforce policies (e.g., the FTP code is not limited to serving files from the filesystem), allowing maximum developer flexibility. The supported protocols include HTTP, FTP, DNS, SMTP, POP3, IMAP, SSH, SIP, Jabber, TOC and Oscar (AIM), MSN, IRC, NNTP, and others.

In addition to these low-level protocol implementations, Twisted also provides higher-level frameworks, implementing the functionality commonly implemented on top of certain protocols. Two of these frameworks are the twisted.web package, an object-publishing web server, and twisted.names, a DNS server framework.

twisted.application

Complex applications are more than just a single protocol implementation. To aid applications development, tying together multiple subsystems, and to ease deployment woes, Twisted provides the twisted.application package. Applications are designed as trees of services, providing an abstraction for related and dependent services.

An application developed in this way, using the interfaces and infrastructure provided by this package, is easy to deploy. An application deployment file is constructed either as a specially structured Python script (which might load configuration information from external application-specific files) or uses plug-ins for the mktap utility that comes with Twisted. mktap generates "TAP" files, which are simple, pre-configured server setups. These deployment files can then be run using twistd, a command-line tool, which provides logging options, daemonization on UNIX, profiling, and so on. For example, using the built-in "web" plug-in that provides a web server configuration, makes starting a web server daemon in Twisted quite simple:

$ mktap web --path=/var/www
$ twistd -f web.tap

Both Python deployment files and TAP files can also be used to generated installable Debian or RPM packages. An experimental utility — which should be available in future versions of Twisted — allows easy generation of installable, self-contained Windows NT services.

Software Using Twisted

Several applications, both open source and commercial, use Twisted. A brief overview of two of them demonstrates the power and flexibility of the framework.

Buildbot is a "system to automate the compile/test cycle required by most software projects to validate code changes." When a developer checks in code, the Buildbot build master is notified, and the build slaves (which can be run on different machines, and different operating systems such as Windows or Linux) are notified by the master (using the Perspective Broker protocol). The slaves check out the latest version of the code and run the applicable tests for the software. The status of the builds can be accessed via a web page, an IRC bot, or using a GUI tool, demonstrating Twisted's support for multiple protocols. The Twisted project itself uses Buildbot to run its tests.

Another project that uses Twisted is Quotient, an integrated messaging server. Quotient supports email — it acts as an SMTP server, POP3 server, and IMAP server, and also includes a sophisticated web-based mail-browsing and administration interface. Quotient also integrates VoIP management services using the SIP protocol. Using Twisted's multiple-protocol support and robust server architecture, Quotient can build a superior messaging server with capabilities that would be extremely hard to achieve with traditional single-purpose servers.

Of course, Twisted can be used to implement simpler systems, and has been used to build applications ranging from GUI VoIP clients to complex web applications. To learn more about Twisted, and how you can use it for your software, you can visit the Twisted web site. I will be giving a Twisted tutorial at the Emerging Technology Conference in February, and you can meet most of the Twisted development team at the PyCon conference this March.

Related Projects in Other Languages

  • ACE, a C++ networking framework.
  • POE, a framework for creating multitasking programs in Perl.
  • Tcl's built-in event loop can be used to build network servers.

Itamar Shtull-Trauring has contributed to Python and Zope and is one of the main developers of Twisted.


Return to Python DevCenter.



Sponsored by: