Published on (
 See this if you're having trouble printing code examples

Software Engineering for Everyone

by Dennis E. Hamilton

Learning to program is easier than ever before with excellent, freely available tools like Python and a community of contributors willing to help newcomers. However, most employers want more than just programmers, they want software engineers. It takes consistent discipline to deliver complex systems into an imperfect world. Whether for commercial air travel, distributing electrical power, operating the postal service, visiting a comet, or giving depositors reliable access to their own accounts via automated teller machines anywhere in the world, we rely on the fruits of engineering discipline daily. Even when developing simple systems, you'll encounter some of the same difficulties an engineer confronts in developing large-scale software.

The elements that help software engineers win at their game can help you. You can develop engineering discipline as you go about learning the craft of programming by keeping in mind three elements of engineering discipline: predictability, collegiality, and accountability. Working closely with engineers in my early days as a programmer taught me the importance of these elements. Together, they form the basis of what I call Software Engineering for Everyone.

Predictability: make it so, keep it so

The key principle of predictability is divide-and-conquer. To divide, create interface agreements. With interface agreements in hand you can break down implementations into modules, making parallel and independent effort possible. This is the most difficult aspect of engineering discipline that you will ever encounter. To succeed you must first define and organize programs and the work to produce them without having already done it.

In 1975, I provided system architecture for a nationwide on-line processing system. The programming team was short-handed. They gave me the opportunity to develop real-time display interface software for minicomputers to be installed in 100 business offices across the country. It was a critical element. We were outgrowing an initial release and urgently needed a new system. We rewrote all of the software, increasing performance and function all at once.

Applying the key principle of predictability, I redefined the application interfaces for interacting with display terminals first. I turned one ill-defined interface with a complex set of parameters into a set of simple interfaces. I introduced one interface per operation. I gave each operation a single well-defined function clearly defining all behavior, operation-by-operation. It was like handcrafting an object-oriented implementation. Designing in an informal dialect of Algol, with the implementation in assembly language, our adherence to an object model came entirely out of disciplined use of the available lower-level tools.

I was nervous about introducing this change in place of something already familiar. However, the simplicity of debugging and verifying correct operation with cleaner, separate interfaces was too valuable to pass up. It also made the inspection and verification of my real-time implementation much easier. Because I could build the new interfaces atop the old implementation using a simple shim layer, I had a working prototype immediately. We began testing the prototype on a system that was already known to be reliable.

Since I had stable software to work with on the other side of my interfaces I could easily avoid finger pointing, and I could get at defects of my new code when it didn't behave the way the shim already did. This became critical as my software became later and later, running far behind the original plan. I had an incremental, unit-level test plan that I was determined to follow rather than do what my boss wanted which was to turn on the whole thing at once and see what happened. Although that was temporarily career-limiting, I have never regretted it.

With a stable interface and a working shim, we quickly caught, repaired, or worked around defects in the prototype. In the end, integration and confirmation testing went off without a hitch. Only one bug was uncovered and that was in a new administrative interface having no counterpart in the old system.

I would love to offer this as a complete divide-and-conquer success story, but there is more, an unexpected lesson: when hardware developers start programming, it is easy to stop thinking like an engineer. The new software was so fast we uncovered timing problems in the firmware of the terminal hardware. Because the new interfaces completely hid the hardware, I was able to add special communication delays in places where the firmware needed more time to complete its operations. Application modules weren't touched. Still, some problems never went away. The terminal hardware would lock up from time to time. When the firmware became autistic, up to 16 displays simply stopped talking, and my software quietly timed out, shutting down user sessions that appeared to have simply gone away.

Hardware technicians pleaded for a way to diagnose the terminals from the minicomputer. I gave them a way to analyze the data my software had about all of the displays, but it was useless. The hardware interface didn't provide the kind of information that the technicians needed. We were dealing with a computer program hidden in firmware where none of us could get to it, let alone fix it. Well-known industry-standard peripheral interfaces didn't have these limitations, but the engineers who designed this equipment thought it was more important to use an inexpensive, limited interface. No amount of add-on software could compensate for that unfortunate point of inscrutable failure. When I had been at a mainframe manufacturer a decade earlier, you couldn't build, let alone ship, a computer system if you couldn't show people how to troubleshoot and repair it. That principle didn't transfer over so well when software started becoming part of everything.

Predefining and sticking to your interfaces allows modules to be brought together -- and substituted -- without surprises. Preserving predictability across many cycles of alteration and refinement is important. When you study the interfaces used by different software packages, you'll notice that some interfaces work better than others, and you'll begin to see why. It is the result of a wonderful paradox: freedom for design and innovation is carved out of the space created by agreeing to constraints that will never be violated. You must first set up the rules of the game so there is room to play, and to play again, and then to win.

Collegiality: community accomplishment

Another crucial aspect of engineering is collegiality. By collegiality, I mean being a willing, self-conscious participant in a communal activity. Software development engages participants distributed across space and over time. Teamwork is part of collegiality. Scholarship is another. The key thing to recognize is computer software is a community accomplishment. Everyone who contributes, in any way, brings something that wouldn't otherwise have been there.

There are two simple practices for developing collegiality:

In short, borrow, add, and give it away.

I grew up on a steady diet of pre-Sputnik science fiction. I say that Robert Heinlein taught me to read. In all of that reading, I thought of being on the moon or going to Mars as something that would be a personal, individual act. The reality of space flight and the magnitude of the enterprise that it took just wasn't the way I dreamt it would be. Today, I'm moved to tears by the magnificence of that undertaking and the contributions that so many people made to bring space flight to reality. Every detail and every contribution mattered.

Human activities of any scale are cooperative activities. As a young software developer, I had this conceit that I could do it all myself, relying on my innate creativity, and if those other jerks would get out of my way it would all be perfect. Now, I know that's hogwash, but I'm still sometimes caught wasting time before consulting someone else for assistance. I want the work I share to be perfect.

Work doesn't have to be perfect before it can be shared with others. The constructive observations of others will provide focus on essentials that are easily overlooked by someone immersed in a project. Explaining a program design to someone else provides insights into what I am doing that I wouldn't have had, even when the person I am having a walkthrough with doesn't say a word. I don't know why that is. It works so often that I simply trust in it.

A good way to share your work is to write about it. That includes sharing your writings as imperfect work in-progress. In my first dedicated programming job in 1959, ("Clerk Typist A" since they hadn't invented student programmer positions) the faculty member I worked for was creating a handbook of software. I was constantly frustrated in my struggles to write intelligible documentation for the programs we were gathering. Sticking with it, I learned what I have heard repeatedly since: the way to learn to write is to start writing.

Nowadays I incorporate documentation as an inseparable part of the design of programs. Assuring that a program is explicable is my primary test for conceptual economy of the software itself. Even when I am not building software for anyone else to use, I preserve the hard-won habit of documenting what I am doing as if it is intended for others to be able to use without having written it themselves. Truthfully, I don't ever think otherwise, because that someone else is often my forgetful future self.

Mastering programming is not a solitary activity, no matter how much we go through it individually. For it to work, we must be willing to submit our work to the adaptation and refinement of others. When I first met Donald Knuth, he spoke about some of the most beautifully crafted programs he had ever read and that inspired work he would later be renowned for. Early, handcrafted software became the inspiration for our generation of programmers because earlier programmers made their work available to study and adapt.

Accountability: What happened when?

The third element of software engineering is accountability. Accountability is providing an accounting of what happened, how it happened, and even why it happened. It's easy to overlook this common element of engineering and scientific disciplines.

When I dropped out of college in 1958, freshman calculus plus high-school drafting classes got me a job as an engineering aide at Boeing. I created graphs and charts from the output of computer models of aircraft behavior. Everything was checked, signed, bound, and filed. The engineers wrote everything down. Everything. The lead engineer and I studied FORTRAN together. This was my first contact with a power user. Rather than suffering with difficult-to-digest output that the software group was willing to provide, he demanded program results in the exact form needed for the analyses and engineering reports we were producing,. Although I resented the work's tediousness, I never feared riding in the Boeing 720 when it was later produced. This experience taught me the discipline of accountability.

In Watts Humphrey's Introduction to the Personal Software Process, students record their time and keep a log and journal very early. This becomes a progressive historical account of actual effort, including the ideas that were tried, the problems that were discovered, and how the course of a project evolves over time. It also provides insight to the engineer, scientist, or student about where activity is spent and how long it takes to accomplish things in a genuine, not imagined, workday.

The critical instrument of accountability is keeping records of what you are doing while you are doing it. Practice and strengthen the discipline of accountability by keeping notebooks and journals. And stick to it.

Failure: winning isn't always easy.

Accountability, collegiality, and predictability are the elements that help engineers succeed, but engineers don't always win. It's important to have room to fail. Predictability, for example, is a hard won skill. I may fail repeatedly before having predictability. It is not something you or I already know. Taking on a new area of technology or changing my tools puts that deficiency in my face. When I'm afraid of being incompetent, procrastination sets in, compounding the problem. I am awestruck at how incompetent today's great software engineers must have been willing to be, so that they could have the experiences that gave them mastery at predictability.

Today, open source projects carried out over the Internet can provide direct experience in all aspects of predictability, including what happens when you lack it. There is no safer or more satisfying place available to the willing newcomer. Find an Internet study group or volunteer for a simple piece of a group project on the Internet. Most open source projects need people to practice installing and using the tools, provide information about problems, and provide documentation. All contributions are appreciated. Open source development makes the perfect public laboratory. As you learn to program with others, experiment and observe your own development of the crucial elements of engineering. Remember to log both your successes and failures. You will become a better programmer for doing so.

Dennis E. Hamilton is an international consultant on the creation of document processing systems.

Return to the Python DevCenter.

Copyright © 2009 O'Reilly Media, Inc.