ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Enhanced Interactive Python with IPython

by Jeremy Jones
01/27/2005

Python is a multipurpose programming language: it is object oriented, is dynamic, can accomplish much in few lines of code, is syntactically clean and elegant, "fits the brain well," and is an excellent language for programmers of all ages and skill levels. These characteristics have contributed to building a loyal, knowledgeable, and vibrant community. In addition to the language syntax and the community, a benefit of using Python is its interactive interpreter. The standard interactive interpreter that ships with Python allows a user to type Python code at a command prompt and have it execute while the user watches, and allows various levels of introspection into the source code. (Python's introspection is its ability to inspect code objects at runtime; determine information about them such as their type, attributes, and defined methods; and make that information available to the user.) The Python interactive interpreter is useful for testing pieces of code, such as algorithms or getting acquainted with a particular library, or for accomplishing specific tasks, such as creating 500 directories named dir_000 through dir_499.

The standard Python interpreter is unquestionably a highly valued item in the toolbox of many Python programmers. However, the IPython developers have taken the fundamental ideas found in the standard interpreter and improved upon them so considerably that they have created a tool of amazing utility. The project's web site describes it as "an enhanced Interactive Python shell." This description is far too modest. IPython is elegant, thoughtfully crafted, and extremely extensible. Some of the thoughtfully crafted features that make IPython so useful are tab completion, object introspection, a powerful history mechanism, inline editing of source code, integration with the Python debugger, the %run mechanism, macros, the ability to create multiple profiles, and system shell access.

IPython began life with Fernando Pérez attempting to unify his own work, IPP by Janko Hauser, and LazyPython by Nathan Gray. Pérez cites Stephen Figgin's article on IPP and LazyPython as inspiration. Fernando is currently the sole developer for the IPython project.

While Fernando likes to emphasize that IPython is "meant to be a highly configurable tool for building problem-specific interactive environments," this article will focus almost exclusively on its ability to replace the Python standard interactive interpreter.

Installing

For this article, I assume you already have Python 2.2 or higher installed. I created and tested all examples using IPython 0.6.6, Python 2.4, and Fedora Core 3 Linux.

In order to run IPython, you must first download and install it. At the time of this writing, IPython 0.6.6 was the latest. Download an appropriate version and format for your system. (For example, because I installed Python 2.4 from source, IPython-0.6.6.tar.gz was best for me.) Windows users should probably download the .zip file and unzip it. In another week or so, another alternative for Windows users is to download the Python Enthought Edition, which will contain IPython as one of its default shells. If you installed an RPM, it should be as simple as rpm -Uvh <filename>. Otherwise, unpack the compressed file and change into the resultant directory, which should contain a setup.py file. Installation should be as easy as typing (as root):

python setup.py install

If on a Unix system, this will put an executable file named ipython in the same directory as your python executable. Installation on Linux systems will put documentation in /usr/share/doc/ipython/<VERSION>. Running ipython for the first time will create a configuration directory at $HOME/.ipython containing several configuration files (ipythonrc, ipythonrc-math, ipythonrc-numeric, ipythonrc-physics, ipythonrc-pysh, ipythonrc-scipy, and ipythonrc-tutorial) for different profiles:

[jjones@cerberus ~]$ /usr/local/python24/bin/ipython
**********************************************************************
Welcome to IPython. I will try to create a personal configuration directory
where you can customize many aspects of IPython's functionality in:

/home/jjones/.ipython

Successful installation!

Please read the sections 'Initial Configuration' and 'Quick Tips' in the
IPython manual (there are both HTML and PDF versions supplied with the
distribution) to make sure that your system environment is properly configured
to take advantage of IPython's features.
Please press <RETURN> to start IPython.

In summary, for installation:

As mentioned above, in order to start IPython, execute the installed ipython executable. That will yield a prompt that may appear unfamiliar to those who already work with the standard Python shell prompt:

[jjones@cerberus ~]$ /usr/local/python24/bin/ipython
Python 2.4 (#2, Nov 30 2004, 09:22:54)
Type "copyright", "credits" or "license" for more information.

IPython 0.6.6 -- An enhanced Interactive Python.
?       -> Introduction to IPython's features.
%magic  -> Information about IPython's 'magic' % functions.
help    -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]:

To exit IPython (from Linux), type Ctrl-D (which will request confirmation before editing) or either Exit or Quit (note the case), both of which exit without prompting.

Features

Magic

IPython has several "magic" keywords:

%Exit, %Pprint, %Quit, %alias, %autocall, %autoindent, %automagic,
%bookmark, %cd, %color_info, %colors, %config, %dhist, %dirs, %ed,
%edit, %env, %hist, %logoff, %logon, %logstart, %logstate, %lsmagic,
%macro, %magic, %p, %page, %pdb, %pdef, %pdoc, %pfile, %pinfo, %popd,
%profile, %prun, %psource, %pushd, %pwd, %r, %rehash, %rehashx, %reset,
%run, %runlog, %save, %sc, %sx, %system_verbose, %unalias, %who,
%who_ls, %whos, %xmode

IPython works by checking the command issued to it against its list of magic keywords. If the command is a magic keyword, IPython knows what to do with it. If it's not a magic keyword, it lets Python figure out to do with it. If automagic is on (the default), you don't need to prepend the % symbol to magic keywords. If automagic is off, you need to prepend them. Type magic at the shell prompt to see a list of all magic keywords and some words of help with their usage. Good documentation is a key feature of any piece of software, and between the IPython user manual and the inline documentation (%magic), IPython certainly does not lack in this area.

Tab Completion

A very powerful feature in IPython is tab completion. If you are familiar with Python, you may be thinking, The standard Python interactive shell has tab completion. All you have to do is:

[jjones@cerberus ~]$ /usr/local/python24/bin/python
Python 2.4 (#2, Nov 30 2004, 09:22:54)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rlcompleter, readline
>>> readline.parse_and_bind('tab: complete')
>>> 

That's correct. Both the standard Python interpreter and IPython support "regular" completion as well as menu completion. To use completion, type a pattern you would like the shell to match, followed by the Tab key. When in "regular" completion mode (the default), the Tab key will:

For example:

In [1]: import os

In [2]: os.po<tab>
os.popen   os.popen2  os.popen3  os.popen4

In [2]: os.popen

Typing os.po followed by the Tab key expands os.po out to os.popen (as shown at the In [2]: prompt) and displays all the modules, classes, and functions under os that begin with po, which happen to be popen, popen2, popen3, and popen4.

Menu completion has a slightly different behavior from regular tab completion. In order to change from the default tab completion behavior to menu completion, edit the ipythonrc config file located in $HOME/.ipython/. Comment out:

readline_parse_and_bind tab: complete

and uncomment:

readline_parse_and_bind tab: menu-complete

Rather than display a list of matches for the current command, menu-complete cycles through each matching item in the list with each press of the Tab key. For example:

In [1]: import os

In [2]: os.po<tab>

will result in:

In [3]: os.popen

Each subsequent press of the Tab key will cycle through to the next matching item in the remaining list of popen2, popen3, popen4, and back to po. To see a list of all items matching the specified pattern while in menu-complete mode, hit Ctrl-L:

In [2]: os.po<Control-l>
os.popen   os.popen2  os.popen3  os.popen4

In [2]: os.po

Introspection

Python has several built-in functions that facilitate introspection. IPython has access to all of the standard Python functions as well as some extras built into the shell. A typical introspective technique with the standard Python shell is using the built-in dir() function:

>>> import SimpleXMLRPCServer
>>> dir(SimpleXMLRPCServer.SimpleXMLRPCServer)
['__doc__', '__init__', '__module__', '_dispatch',
'_marshaled_dispatch', 'address_family', 'allow_reuse_address',
'close_request', 'fileno', 'finish_request', 'get_request',
'handle_error', 'handle_request', 'process_request',
'register_function', 'register_instance',
'register_introspection_functions', 'register_multicall_functions',
'request_queue_size', 'serve_forever', 'server_activate', 'server_bind',
'server_close', 'socket_type', 'system_listMethods',
'system_methodHelp', 'system_methodSignature', 'system_multicall',
'verify_request']

Well, that's nice. Very helpful, in fact. I have done just that for years now and have been very pleased with it. This is a nice list of all of SimpleXMLRPCServer.SimpleXMLRPCServer's methods, classes, modules, and so on. Because dir() is a built-in function, IPython can do the same thing, but its ? and ?? operators are more powerful:

In [1]: import SimpleXMLRPCServer

In [2]: ? SimpleXMLRPCServer.SimpleXMLRPCServer
Type:           classobj
String Form:    SimpleXMLRPCServer.SimpleXMLRPCServer
Namespace:      Interactive
File:           /usr/local/python24/lib/python2.4/SimpleXMLRPCServer.py
Docstring:
    Simple XML-RPC server.

    Simple XML-RPC server that allows functions and a single instance
    to be installed to handle requests. The default implementation
    attempts to dispatch XML-RPC calls to the functions or instance
    installed in the server. Override the _dispatch method inherited
    from SimpleXMLRPCDispatcher to change this behavior.

Constructor information:
Definition:     SimpleXMLRPCServer.SimpleXMLRPCServer(self, addr, 
                     requestHandler=<class
				SimpleXMLRPCServer.SimpleXMLRPCRequestHandler at 
				     0xf6b2c5cc>, logRequests=1)

If long strings are present, the ? operator will trim them. The ?? will not trim long strings, and it will display syntax highlighted source code if the source is available.

History

After interactively entering commands, statements, and so on into the IPython shell like this:

In [1]: a = 1

In [2]: b = 2

In [3]: c = 3

In [4]: d = {}

In [5]: e = []

In [6]: for i in range(20):
   ...:     e.append(i)
   ...:     d[i] = b
   ...:

you can quickly view everything you typed like this:

In [7]: hist
1: a = 1
2: b = 2
3: c = 3
4: d = {}
5: e = []
6:
for i in range(20):
    e.append(i)
    d[i] = b

To view the history without the input numbers (here, 1 through 6), use hist -n:

In [8]: hist -n
a = 1
b = 2
c = 3
d = {}
e = []
for i in range(20):
    e.append(i)
    d[i] = b

Using hist -n makes it easier to paste commands into a text editor. To search the history, type a pattern to search for and press Ctrl-P. When something matches, a subsequent Ctrl-P will search backward in your history, and Ctrl-N will search forward.

Edit

When testing an idea at a Python prompt, it is sometimes helpful to edit (and, more importantly, to reedit) some lines of source code with a text editor. Type edit from an IPython prompt to bring up the editor defined by the $EDITOR environment variable, or vi on Unix and Notepad on Windows if you don't have $EDITOR defined. To return to the IPython prompt, exit the editor. Saving and exiting will execute, in the current namespace, the code entered into the editor. If you do not want IPython to execute the code automatically, use edit -x. To reedit the last code that you edited, type edit -p. In the previous feature, I mentioned hist -n making it easier to paste code into an editor. An even easier way of putting code into an editor is using edit with Python list slice syntax. Suppose hist yields:

In [29]: hist
1 : a = 1
2 : b = 2
3 : c = 3
4 : d = {}
5 : e = []
6 :
for i in range(20):
    e.append(i)
    d[i] = b

7 : %hist

To export lines 4, 5, and 6 into an editor, type:

edit 4:7

Debugger Access

Another feature within IPython is its access to the Python debugger. Type the pdb magic word from the IPython shell to toggle automatic debugging upon hitting an exception. With automatic pdb calling enabled, the Python debugger will start automatically when Python encounters an unhandled exception; the current line in the debugger will be the line of code on which the exception occurred. The IPython author states that sometimes when he wants to debug something at a certain line of code, he will put 1/0 at the point he wants to begin debugging, enable pdb, and run the code in IPython. When the interpreter hits the 1/0 line of code, it raises a ZeroDivisionError exception and drops him into a debugging session at that particular line.

Run

Sometimes, when in an interactive shell, it is helpful to execute the contents of a Python source file. Issuing the run magic command followed by a Python source file will run the file in the IPython interpreter (for example, run <run options> <python source file> <options>). The following run options are available:

Macros

Macros allow a user to associate a name with a section of Python code so the code can be run later by referring to the name. As with the edit magic word, the list slice notation also works with macro definitions. For example, for a history such as:

In [3]: hist
1: l = []
2:
for i in l:
    print i

you can define a macro with:

In [4]: macro print_l 2
Macro `print_l` created. To execute, type its name (without quotes).
Macro contents:
for i in l:
    print i

Execute it via:

In [5]: print_l
Out[5]: Executing Macro... 

In this case, the list l was empty, so it did not print anything. However, and here is a powerful feature of macros, binding the list l to something and then executing the macro again produces a different result:

In [6]: l = range(5)

In [7]: print_l
Out[7]: Executing Macro...
0
1
2
3
4

It is as if you retyped and executed the code contained in the macro print_l when calling the macro again. It had access to the new binding of the variable l. While macros are absent from Python syntax (and probably always will be), it is certainly a useful feature in an interactive shell.

Profiles

As mentioned earlier, IPython installs multiple configuration files for several different profiles. The configuration files have a naming convention of ipythonrc-<profile name>. In order to start IPython with a specific profile, execute IPython with:

ipython -p <profile name>

One method of creating your own profile is to create an IPython configuration file in the $HOME/.ipython directory named ipythonrc_<your profile> where <your profile> is the name with which you will refer to your profile. This can be useful if you have several projects you work on and each project requires the use of specific, different libraries. You can create a profile for each project and import the modules you frequently use for that project in the configuration file for each project.

System Shell Access

In the default IPython profile, the Unix shell commands (on Unix, of course) cd, pwd, and ls all work like they do from a bash shell. To execute any other shell commands, prepend a ! or !! to them. Use the %sc and %sx magic words to capture the output from shell commands.

The pysh profile is intended as a shell replacement. Starting IPython with a -p pysh flag will cause IPython to accept and execute any commands in the user's $PATH, while at the same time allowing access to all Python modules as well as all Python keywords and built-in functions. For example, to create 500 directories named d_0_d through d_500_d, start IPython with a -p pysh and do something like this:

jjones@cerberus[foo]|2> for i in range(500):
                    |.>     mkdir d_${i}_d
                    |.> 

This will create 500 directories:

jjones@cerberus[foo]|8> ls -d d* | wc -l
500

Notice the mix of the Python range function and the Unix mkdir command.

Note, however, that while ipython -p pysh can provide a powerful shell replacement, it lacks proper job control. Pressing Ctrl-Z while performing some long-running task will stop the IPython session rather than the running subprocess.

Gotchas

While the Python replacement shell is excellent overall, two things provided a small amount of trouble for me. To the credit of the IPython developers, both items are configurable with clear documentation.

The first item was the coloring. On one of my systems, I use xterms with a white background. When requesting information from an object or module with the ? and ?? operators, the object definition line appeared, but it looked like the arguments were missing. That was because the arguments in the constructor displayed in white by default. I resolved this by entering colors LightBG at the IPython shell.

The second item was the combination of autoindent and pasting code. With autoindent enabled, IPython double-indented a section of code I pasted that already had indentation. For example, the following code:

for i in range(10):
    for j in range(10):
        for k in range(10):
            pass

became:

for i in range(10):
        for j in range(10):
                    for k in range(10):
                                    pass

which really was not a problem in this case, because the indentation was consistent among itself. In other circumstances (examples of which elude me just now), it may present a real problem. Invoking the autoindent magic word toggles autoindent so it will not add extra indents--similar to set paste in vim.

Conclusion

IPython is not revolutionary, nor is it entirely novel. Tab completion, searchable history, profiles, and config files have existed in other shells for years, and Python has had levels of introspection for quite some time now. However, IPython has unified some of the most powerful features of mature Unix shells, the Python standard shell, and the Python language into one utility. The result is an unbelievably powerful performance-enhancing tool that I will likely use for years to come. To paraphrase Archimedes, give me a powerful and flexible text editor (vim), interactive shell (IPython), and language (Python), and I can move the world.

Learning Python

Related Reading

Learning Python
By Mark Lutz, David Ascher

Jeremy Jones is a software engineer who works for Predictix. His weapon of choice is Python.


Return to the Python DevCenter.

Copyright © 2009 O'Reilly Media, Inc.