Preserving Backward Compatibilityby Garrett Rooney
When a project hits a certain point in its life cycle, the unpleasant issue of backward compatibility begins to rear its ugly head. All of a sudden the changes introduced in a new release of the software have a dark side to them; they hold hidden possibilities that will break something one of your users depends on. This is true in both open and closed source projects, but in the open source world it seems that the community has spent less time worrying about it than in the closed source world.
That is understandable; the closed source world has paying customers who complain loudly when they upgrade to a new version of a piece of software and something breaks, but the same issues occur in open source software as well. For the world to take open source programs seriously, you must deal with the issue of backward compatibility.
In order to better prepare you, the average open source hacker, for dealing with this problem, I'd like to share some of the experiences we've had with backward compatibility in the Subversion project. With luck, you'll be able to apply some of the lessons we've learned to your own projects. Everyone--developers, redistributors of your project, and, most of all, your users--will benefit in the long run.
Our example project
Before talking about backward compatibility, I want to explain the context of the examples, because without the context surrounding the project you can't make any real decisions about backward compatibility. What might be appropriate for one project can be totally wrong for another. This article draws examples from the Subversion project, a version control system designed to replace CVS, the current de facto standard in open source version control systems.
Subversion provides a client-server architecture that uses several separate network protocols that provide access to the Subversion server. The core Subversion libraries are written in C, with a command-line client providing the primary user interface. Additionally, bindings are provided that let you make use of the C libraries in various other languages. In several cases, non-Subversion clients speak the same protocol Subversion does, so there is a need to provide backward compatibility with users of the Subversion libraries and with programs that simply implement the protocols themselves.
Subversion has declared itself to be at a 1.0 level of quality and has a large user community. The project is now in a position where any break in backward compatibility will likely bring with it significant pain for users. Multiply this by the fact that a popular use of Subversion is to store source code, the most precious asset for a software developer, and so anything that makes it impossible for a user to access that data is even more critical a problem than you are likely to see in an average project.
Backward compatibility is really a series of promises to your users as to what they can expect when they upgrade to a new version of your software. Those promises often fall into three categories.
First is the promise that users can move both forward and backward in versions of a software package without incompatibilities. This is the strongest kind of promise and is very hard to maintain. As a result, many projects use it only for small upgrades. Subversion provides this kind of compatibility promise for patch releases: for example, the change between version 1.0.0 and 1.0.1. That means I can install version 1.0.0 of Subversion, use it for a while, upgrade to version 1.0.1, use it for a while, and then revert to 1.0.0, all with no ill effects.
Next is the promise that users can move forward to new versions of the software, though they may not be able to move backward again once they do. Subversion makes this promise for minor version changes, so people can move from any of the 1.0.x series of releases to any of the 1.1.x series of releases without any problems. However, they cannot necessarily move back. It requires some discipline to provide this kind of backward compatibility, but it's infinitely easier than the previous kind.
Third is the case in which there is absolutely no promise whatsoever. For obvious reasons, you want to avoid this kind of situation, because it means that the process of moving to a new version of the software is harder. Any difficulty in moving to a new version makes it more likely that people will just continue to use the old version. There's little point in releasing a new version of your project if nobody will move to it. Subversion reserves this kind of promise (or lack of promise, really) for major version number changes. For example, the move from Subversion version 1.x.x to 2.x.x will most likely have no promise of backward compatibility between the two versions.
Kinds of Compatibility
Once you've thought about the level of compatibility you want to promise to your users, the next step is to think about the actual places your project needs to worry about specific kinds of compatibility problems.
The first kind of compatibility most people think about is API compatibility. This means that subsequent versions of a library provide the same API that previous versions do, so programs written against the previous version will still be able to compile and run with the new version. In addition to actually leaving the same functions around, this also implies that those functions all do the same thing in the newer version that they did in the older ones. Of course, there is some flexibility here, as at some point you have to be able to say, "This behavior is a bug, so we changed it"; otherwise, there is little point in releasing new versions of your software.
Second is Application Binary Interface, or ABI, compatibility. This means that backward compatibility is preserved at the level of the binary object code produced when you compile the library. There is usually some overlap between API and ABI compatibility, but there are important differences. To maintain ABI compatibility, all you have to do is ensure that your program exports all of the same symbols. This means all the same functions and globally accessible objects need to be there, so that programs linked against the prior version will still be able to run with the new version. It's possible to maintain ABI compatibility while breaking API compatibility. In C code, leave symbols in the C files but remove them from the public headers, so new code that tries to access the symbols will fail to compile, while old code that users compiled against the previous version will continue to run.
If your program communicates over a network, it has to deal with a third form of compatibility, client-server protocol compatibility. This means that a client using the version of the network protocol provided in the older releases will continue to function when faced with a newer server, and that newer client programs will continue to work with an older server. Of course, in some cases this is impossible. A new feature might require support on both the client and server side of the network. In general, though, existing functionality should continue to work with various combinations of client and server code.
Finally, if your program stores data somewhere, be it in a database or in files on disk or wherever, there is data format compatibility. Newer versions of the code need to be able to work with data files written out by older versions, and vice versa. Ideally you should also be able to build some forward compatibility into data formats. If your file-handling routines can ignore and preserve unrecognized fields, then new functionality can modify data formats in ways that do not break older versions. This is one of the most critical kinds of compatibility, simply because users become very upset when they install a new version of a program and suddenly cannot access their old data.
If your project provides code libraries that other programs make use of, you will almost certainly have to worry about API compatibility. If you work in a language that produces object code other programs link against, you will either have to worry about ABI compatibility or require your users to recompile their code for every upgrade. If you make use of a network, you probably need to consider client-server protocol compatibility; and if you store any data at all, you will have to worry about data format compatibility. A nontrivial project like Subversion has to worry about all four of these issues.
Ways to Maintain Compatibility
With the various types of compatibility in mind, it's now time to delve into the actual techniques for maintaining them. This is not an exhaustive list by any means. Not every technique mentioned may apply to your particular project, but they're all worth learning about.
Don't throw anything away
The cardinal rule of maintaining API and ABI compatibility is that you must never remove anything from your interface. As soon as you expose a new function or data structure to your users in a public release, you have committed yourself to supporting it until your compatibility rules allow you to change or remove it, probably in your next major revision.
That means that if you want to make modifications to an API, you need to retain the existing version and add a new one alongside it. The old version remains intact, both in the public interface (such as in the header files for a C or C++ program) to ensure source-level API compatibility and in the binary object files to ensure link-level ABI compatibility.
Subversion has used this technique several times, specifically when we've needed
to add new arguments to existing functions. For example, in Subversion 1.1.0
svn export command gained the
argument, which allows you to specify your platform-specific end-of-line
character sequence. This allows you to simulate the effect of exporting a
project on a platform that has a different end-of-line sequence than the
platform you are running Subversion on. Under the hood, this required creating
svn_client_export2 function, which is identical to the
svn_client_export function but with an
additional argument for specifying the native end-of-line style. The existing
svn_client_export function continues to exist but now simply
svn_client_export2 with a
style. This allows the project to add new functionality while still enabling
third-party code that makes use of our APIs to continue to compile and run
Hide the details
Perhaps the simplest way to avoid problems when making changes to an interface is to not expose that interface to the users. This is difficult to do when you're talking about a function, because to make use of the API the caller really needs to know its name and signature. When you're talking about data structures, it often becomes a viable option.
In a C program, exposing the definition of a structure to your client means much the same thing as exposing the declaration of a function, but with one additional trick. A structure's definition includes not only its name and its contents, but also its size. If you want to maintain ABI compatibility, you cannot add or remove fields from the structure because a client might depend on that size.
For example, if you define a struct that contains two integers and put that definition in a public header file, there's nothing to keep your clients from declaring an instance of that struct on the stack in their own code. Suppose that you later add a third integer to the structure and a client drops in a new version of your library without recompiling the code. It's likely that the client's code will pass a pointer to the structure it declared on the stack into your code, and then you're in trouble, because your code compiled with the knowledge that the structure is three integers large, but the client allocated only two integers' worth of space for it, so any of your code that tries to access the third integer in the structure will access random memory somewhere, which can only lead to problems.
Other programming languages often have more elegant ways to deal with these issues. For example, Java and C++ support various means for restricting access to the internals of an object, and many languages require that all access to objects go through pointers, so the size issue is less of a problem. In raw C you really have to deal with it yourself, though, as the language provides little help.
You have a few ways to avoid these kind of problems in C. First, you can simply not give the client access to the definition of the structure. Instead, forward-declare the structure so that the client can pass around pointers to them, leaving the actual definition inside your library. All access to the structure must go through functions that you define in your library, allowing you to change them should the internals of the structure ever change. This is the opaque pointer technique, and it's probably the best way to solve these sort of issues.
Within the Subversion libraries, several places make use of opaque pointers.
One of the most prolific is the working copy library's access baton object,
svn_wc_adm_access_t. The access baton's forward definition is in
svn_wc.h. The client creates them by way of the
function (or, in versions of Subversion later than 1.0.x, the
svn_wc_adm_open2 function, another example of providing a new
function in a backward-compatible manner). All the public functions in
libsvn_wc simply accept opaque pointers to the structure.
Internally, the only place that actually has access to the definition of the
structure is the file subversion/libsvn_wc/lock.c. Note that the same
technique that keeps client code from poking around in the internals of the
access baton also keeps the rest of the Subversion libraries from doing so,
thus making it easier to modify the structure without modifying other parts of
If you absolutely need to leave the definition of a structure in public header files, but you still want to preserve the ability to change the structure later, there is a way to do it. Before doing so, keep in mind that it's much more fragile than just using an opaque pointer in the first place, mainly because it requires that your clients "do the right thing" without using any technical means to enforce that they do so. The trick is simply to provide a function that allocates the structure for you and to document that the only valid way to obtain access to an instance of the structure is to use that function. Because only your library will ever allocate the structure, you can be sure that there will always be enough memory to hold the entire thing, even if you increase its size later.
This technique has a few more gotchas. First, you can add only new fields to the structure, and you must add them to the end of the structure because clients compiled using previous versions of the definition will assume the offset of old fields within the structure are the same as they were in previous versions. Second, when adding fields to the structure you must ensure that code using the fields can deal with the new fields' not being initialized; otherwise, providing compatibility for old clients that know nothing about the fields is futile. Finally, as I already explained, there's nothing that enforces the use of the constructor functions, so it's still possible for clients to shoot themselves in the foot with this technique. If that bothers you, an opaque pointer is almost certainly a better solution.
For an example of how to provide a constructor function for a publicly
defined structure, see the
svn_client_create_context function in
svn_client.h. Please be cautious with this technique, though, as it
really is dangerous to rely on your users to do the right thing. In retrospect
(and I can say this because I wrote the client context code in Subversion and
made it a publicly defined structure in the first place), I think going with an
opaque pointer would have been worth the extra effort of creating the necessary
accessor functions. That's because I have encountered users of this code who jump right past the documentation that states that you must use
svn_client_create_context and allocate their own instance on the
stack, defeating the purpose of the constructor function and setting themselves
up for pain if and when we finally add new elements to the client context
Pages: 1, 2