The Engineering Approach to Social Processes

The computer revolution is a revolution in the way we think and in the way we express what we think. The essence of this change is the emergence of what might best be called procedural epistemology--the study of the structure of knowledge from an imperative point of view, as opposed to the more declarative point of view taken by classical mathematical subjects.

-- H. Abelson and G. Sussman (in The Structure and Interpretation of Computer Programs)

The Open Organizations Framework is the product of several different kinds of thinking. One of them is a relative newcomer in the field of social theory; because of recent historical developments, it is now an influence in many projects for social change. I believe it fills a major gap that has crippled many movements for social transformation, and therefore deserves a closer look.

This way of thinking can be called the engineering approach. It is not an attempt to fulfil the dream of some 19th-century thinkers, whose naive optimism about the human sciences led to the belief that the principles of human nature would soon be discovered, allowing theorists to identify the structures that should be imposed on society. Instead, today's engineering approach is a way for a group of people to engineer their own working practices, by means of a dialogue between needs and principles, between practice and theory, on equal terms. It therefore offers a way to close the age-old gap between academic social theory and the practice of social movements.

Engineering in general involves a close interaction between theory and practice compared to many other academic disciplines, for several reasons:

• It requires a great deal of theory in order to be conceivable at all.

• But it also has to work. All details and interdependencies must be accounted for; if anything is glossed over, the whole system may fail.

In software engineering, there are additional reasons for this interaction:

• The uses to which software is put, and the environments in which it is used, are constantly changing. This means that by the time academia has developed a solid theoretical basis for solving one set of problems, software developers are faced with very different problems. As a result, practitioners must act as theorists.

• Practitioners and theorists are also users.

The historical event that has made software engineering relevant to social change is the appearance of free software. In recent years, free software has become a flourishing international movement, whose practice of openness and cooperation has had far-reaching effects beyond the world of computers. The participants in free software include programmers, translators, graphic designers and journalists, as well as users who report bugs, write documentation, test new versions of software and suggest improvements. These participants are contributing to a vast body of tools for meeting concrete needs. The widespread use of copyleft reflects a commitment to the sharing of knowledge on a very large scale, guaranteeing the public ownership of these tools and of the ideas embodied in them. The result is the development of a strategic collective subject.

These participants are ideologically diverse, but there is a broad consensus on a certain number of working practices, both technical and social, that have evolved through an dialogue between theory and practice. Theory, in this case, includes the academic field of computer science, which provides ways of analysing technical problems; technical wisdom that has grown out of experience; ideas about human interaction, including project management and development processes; and ethical principles that guide commitments such as copyleft. Practice includes the experience of building effective teams and communities as well as useful software, and of solving the innumerable problems that arise in both realms, as well as in the interactions between the two.

Because the free software community is largely composed of inquisitive people who habitually try to understand how things work, it has been natural for them to study their own working processes from an engineering point of view, and to use engineering principles to improve those processes.

It may be tempting to point out that social interaction is far more complex than anything a computer program can deal with. This is true in the general case: computer programs cannot react to emotional realities or cope with the irrational, cannot be tactful or fall in love. However, there is an important subset of social interaction that very closely corresponds to what we use computers for: the coordination of production processes, which includes all practical communication.

Before the widespread use of computers, engineering encompassed everything people knew about how to make useful physical things. These physical things (and the interactions between them) became increasingly complex, until people reached the limit of their ability to keep track of all the details. We now use computers to make those details manageable, because computers have better attention to detail than we do. In order to tell the computers what we want them to do, we write programs, which are like recipes that embody our best insights about how practical things should be done.

Programming involves solving many of the sorts of problems that arise in other types of production: a result must be obtained with limited resources. In the world of software engineering, those resources are time, processing power, memory, disk space, network bandwidth... and programmers. As in other fields, trade-offs are possible between one kind of cost and another. A program works in an environment, which consists of human beings, physical machines and other computer programs; its interdependencies with its environment impose constraints on its operation. Users expect programs to respond promptly. A program cannot be allowed to pollute its environment any more than a factory can; nor can it be allowed to starve other programs by using all the available resources.

However, programming is a particularly difficult type of production to get right. The tiniest imperfection in a computer program can cause it to fail completely, possibly with disastrous results. And as Richard Stallman points out, it is possible for software to reach a degree of complexity that far exceeds what can be done in any other field of engineering.

In other fields, people have to cope with the perversity of matter. Of physical objects. Matter does what it is going to do. You can try to model it and if the actual behavior doesn't fit the model then tough on you, because the challenge is to make physical objects that really work.

If I wanted to put an 'If' statement in a 'While' statement, I don't have to worry about whether the 'If' statement will oscillate at a certain frequency and rub against the 'While' statement and eventually they will fracture.... I don't have to worry that if i run this program in a salt water environment that the salt water may get in between the 'If' statement and the 'While' statement and cause corrosion.... I don't have to worry when I write the program, about how I am going to physically assemble each copy and whether I can manage to get access to put that 'If' statement inside the 'While' statement.... So many problems that we don't have to worry about in software.... It is fundamentally easier to write a program than to design a physical object that's going to work. This may seem strange because you have probably heard people talking about how hard software is to design and how this is a big problem and how we are going to solve it. They are not really talking about the same question as I am. I am comparing physical and software systems of the same complexity, the same number of parts. I am saying the software system is much easier to design than the physical system. But the intelligence of people in these various fields is the same, so what do we do when we are confronted with an easy field? We just push it further! We push our abilities to the limit. If systems of the same size are easy, let's make systems which are ten times as big, then it will be hard! That's what we do! We make software systems which are far bigger in terms of number of parts than physical systems. A physical system whose design has a million different pieces in it is a mega project. A computer program whose design has a million pieces in it, is maybe 300,000 lines, a few people will write that in a couple of years. That is not a particularly giant program. GNU Emacs now has several million pieces in its design I think. It has a million lines of code. This is a project done with essentially no funding whatsoever. Mostly done by people in their spare time.

There is another big saving. If you have designed a physical product, the next thing you have to do is design the factory to make it. To build this factory may cost millions or tens of millions whereas to make copies of the program, you just have to type 'copy'. The same copy command will copy any program. You want copies on CD then fine. You burn a master CD and send it off to a CD plant. They will use the same equipment which will copy any contents on a CD. You don't have to build a factory to make this product. There is tremendous simplification and tremendous reduction in costs of designing things....

The result is software really is different from other fields. Because we are working with mathematical stuff, designing something is far, far easier and the result is that we regularly make systems which are much, much larger and do so with just a few people.

It is difficult to make something that is both very complex and perfect. Many if not most programs do a job so complex that no single person can remember the whole thing in detail. Just thinking through everything a typical program does, step by step, would take years. Programmers have always been acutely aware of the danger of runaway complexity, of not being able to understand their own work. Moreover, the environment in which programs work changes constantly. Computer programs have to be designed to allow for this change.

A great deal of work has therefore gone into strategies for managing complexity and change. These strategies have two aims. The first is to make the most of our knowledge and skills, in order to minimise the chances of making costly mistakes. The second is to organise practical processes themselves -- the work that programs must do -- in such a way that we are able to grasp them, and so they can adapt to change without having to be redesigned from scratch.

One way to make a complex body of knowledge easier to deal with is to write it all down. How you write it down, of course, makes a big difference. Accordingly, one class of strategies for dealing with complexity involves the use of language and communication.

It is often said that computer science, like many academic disciplines, is not relevant enough to the practical problems that practitioners face. As in other fields, there is a certain amount of distrust between those who are mainly practitioners and those who are mainly theorists. However, one area of fruitful cooperation has been language.

Natural languages are not suitable for writing computer programs, because they leave too much room for interpretation. Since computers cannot think for themselves, they need to be told exactly what to do. This forces the programmer to have an absolutely clear notion of what is to be accomplished and how; no details can be glossed over.

The vocabulary of early computer languages only allowed people to describe minute details of a process, one after the other. It was therefore easy to miss the forest for the trees. Nowadays, this statement (also from The Structure and Interpretation of Computer Programs) is a truism:

Programs must be written for people to read, and only incidentally for machines to execute.

Like any good work of nonfiction, a good program should be organised in a way that makes the main points clear, and makes it easy to find details of interest.

Clarity is essential for the program's original authors, so that they can check their own work and assure themselves that it makes sense. A successful program nearly always ends up being maintained by people other than its original authors; these maintainers need to be able to understand the program well enough to fix bugs and add new features. Clarity is therefore essential for them, too.

It is mainly this need for clear structure that has motivated the evolution of computer languages. As people have devised concepts that enable our human minds to understand complex, changing processes, they have invented programming languages in which these insights could be expressed.

For example, one problem of early computer programs was that sometimes, the details of every part of a program depended on the details of every other part. The result was unmanageable complexity: nothing could be changed without changing everything. In order to make change easier, people started dividing programs into more independent sections called modules or components, each of which performs a discrete task. Each component is given an interface, which is a description of the ways in which other parts of the program can interact with it; it describes what the component does, without specifying how it does it. This has several benefits. First, it means that the component can be tested separately from everything else. If testing shows that it keeps the promises expressed in its interface, the programmers can then treat it as a 'black box'; they can forget about how it works, and turn their minds to other problems. Second, if the users' needs change, the component can be replaced with a new one that has the same interface, and the rest of the program need not be changed. Third, different components can be written independently by different people, who don't have to understand each other's work; this allows for a division of labour. Because of the clear benefits of this approach, as languages have evolved, they have provided increasingly explicit ways of describing interfaces, of distinguishing between the what and the how.

All programs must be able to respond to errors, i.e. failures in their environment. A user may enter nonsensical data, a network connection may fail, a hard disk may run out of space. Early programming languages had no particular way of describing errors or ways of handling them. When one reads programs in those languages, it can be difficult to find the error-handling behaviour and to make sure that all possible errors are actually handled. Programmers realised that this was making it hard for them to manage the complexity of all the things that could go wrong, so they invented languages that provide clearer, more rigorous ways of describing errors.

In the very early days of programming, people realised that no matter how clearly a program is structured, it inevitably embodies decisions that will not be obvious to someone who is looking at it for the first time. There have been three remedies to this problem: (1) adding comments for human readers throughout the program, (2) writing articles explaining how it works and why, and (3) drawing diagrams to give people an overview of its structure. The latter two kinds of explanation are called design documentation. Many people feel that every program should be accompanied by all three, in order to maximise the chances that someone unfamiliar with the program will be able to understand it.

Good programmers tend to feel that if a program is not clear, it's probably because its authors didn't think through the problem well enough. A good engineer finds the simple solution hiding in apparently complex problems. Once that simplicity has been discovered, writing the program becomes simpler as well.

There is one more way that language is used to make programs easier to develop and maintain; it concerns not the way the program is written or explained, but what the program itself says about what it is doing. When a program is unable to do what it is supposed to do, it should say so. We have all seen programs that fail in spectacularly uninformative ways, giving us no hope of knowing how to fix the problem. Although you may curse programmers when this happens to you, it happens to them much more often. By the time the average user gets to use a program, its authors have spent countless hours trying to understand why it didn't work, and fixing their mistakes. Computer scientist Maurice Wilkes discovered this phenomenon in 1949:

As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.

In order to understand why a program didn't work correctly, you need to know what it was doing when it failed. One way to get this information is to make the program keep a log of everything it does. Then when something goes wrong, you can look at the log and reconstruct what was happening when the problem occurred. If the program was interacting with other programs, you can look at their logs and see what they thought was happening at that time. Logs often provide precious clues for diagnosing bugs.

We can sum up these lessons about language and communication as follows:

• Write down a clear recipe for producing the results you want. In order to do this, you need to find the right concepts and vocabulary.

• If your description of a solution is hard to understand, it's usually a sign that you don't understand the problem well enough.

• Separate the what from the how. Interactions between processes must be defined by clear interfaces, so that interdependencies can be strictly limited.

• You have not described a practical process unless you have also described how it will handle all the things that could go wrong.

• Processes must be documented so that everyone involved can understand them.

• Keep detailed records so that when things go wrong, you can reconstruct what happened.

Within the proprietary software industry, the difficulties of managing complexity have led a few exceptionally perceptive people to identify basic truths about about project management and working practices. One of the best-known such efforts is The Mythical Man-Month, written by Fred Brooks in 1975.

Cost does indeed vary as the product of the number of men and the number of months. Progress does not. Hence the man-month as a unit for measuring the size of a job is a dangerous and deceptive myth. It implies that men and months are interchangeable.

Brooks's point is that progress will vary according to the amount of communication required between the participants, the difficulty of dividing the task between them, and the complexity of the interrelationships between their work. He is also known for Brooks's Law:

Adding manpower to a late software project makes it later.

There are three reasons for this: repartitioning jobs disrupts work and creates more work in itself, training new people takes time, and communication and decision-making are more time-consuming in a larger group.

Most of these lessons were already familiar to good programmers by the time significant numbers of people started participating in free software. When sharing of source code started to happen on a large scale, programmers realised that this was a type of communication that had previously unimagined benefits. For the first time, it became commonplace to find and correct bugs in programs that you hadn't written and didn't normally work on. If a program didn't quite suit your needs and you had the source code, you could modify it to do what you wanted. Perhaps most importantly, you could learn the craft of programming by studying the work of the great masters and even by apprenticing, as it were, on their projects.

As the corpus of free software grew, it became more and more likely that, when you had a problem to solve, you could see how someone else had solved it, and perhaps even use their solution verbatim.

Much of the design philosophy of free software has its origins in the Unix operating system; as a result, many people have been exposed, through free software, to what is known as the Unix philosophy. Here is a paraphrase of some of its tenets, those which are clearly applicable outside the world of software:

• Small is beautiful. Strive for simplicity. Simple processes are easy to understand, easy and inexpensive to maintain, and easy to combine with other processes.

• Make each process do one thing well. Avoid duplication of effort.

• Test your assumptions as early as possible.

• Write down everything you learn in a form that is easily accessible to others.

• Copy good processes from other people.

• Save trees. Keeping data online allows it to be used in a greater variety of ways.

• Look for the 90% solution. Solving the next 5% probably costs more than the previous 90%; solving the next 5% after that costs more still.

Given this commitment to openness, it's not surprising that free software projects have chosen to publish not only their source code online, but their entire development process. Typically, everything is publicly accessible, including rough drafts of new programs, mailing lists where a project's participants discuss their work in progress, and bug tracking systems on which users report problems they've had and programmers post attempts at fixing them. Moreover, participation is open to anyone who has something to contribute, and large projects like KDE have come to include many non-programmers. There are simple practical reasons for this. When processes are open, it is more likely that problems will be spotted before they become costly. It is also more likely that, given a problem, someone will come along with the right skills to solve it.

A corollary to this principle, first pointed out by Eric Raymond in The Cathedral and the Bazaar, is the maxim 'Release early, release often.' Translated into non-programming terms, it means:

• When you're developing a new process, let people try it out and contribute to it as soon as you have even a partial rough draft.

The need to publish and organise knowledge so that others can use and improve it has led to the development of new kinds of software, such as Wiki. The commitment to openness, and to the ability to reuse other people's knowledge and work, found a formal expression in the idea of copyleft, which has inspired non-software projects such as Indymedia and Wikipedia. Like free software developers, the participants in these communities seem to view their social processes as part of the project they're working on. It is worth comparing this approach to that of not-for-profit groups that have not had this contact with free software, such as charities, NGOs, trade unions and many activist groups. Such groups typically adopt a traditional organisational structure at the outset, and rarely question it during the course of their work. In organisations that have been strongly influenced by free software, the opposite seems to be true: it is taken for granted that organisational processes ought to be thought through collectively like any engineering problem, and periodically adjusted to meet changing needs. It is also taken for granted that anyone in the group can work on improving its organisational processes.

The Open Organizations project has, in part, been motivated by a desire to adapt this engineering approach, and the organisational processes used successfully in free software, to a broad range of organisations, as well as to the production of goods other than knowledge.

-- BenjaminGeer

---

APPENDIX (to be refactored) - Patterns

One definition of pattern is: "a named nugget of insight that conveys the essence of a proven solution to a recurring problem within a certain context amidst competing concerns". This idea was applied first in architecture and then in other areas such as pedagogy, ecology, and specially in software.

Since a pattern is presented in a structured form, one can easily realize what aspect is missing or needs more work, and one can compare it with other similar patterns.

For instance, the compact pattern form consists of these sections: Context, Problem, Forces, Solution, Resulting context. If we apply this idea to OpenOrg modules, processes or functional rules, for Framework restructuration in the way proposed in this meeting and elaborated here (i.e. separating WHY, WHAT, HOW), we have that:

• Context, Problem and Forces relate to the WHY, the description of a recurring tension between the current conditions (be material or not), interdependencies, core values and, let's say, the anti-values (i.e. competition, concentration of power, etc.)

• Solution relates to the HOW, the proposal of a proven technique or process the OpenOrg Framework offers.

• Resulting context relates to the WHAT, the desired goal or final product, which may have advantages and disadvantages, but the former should outweigh the latter.

-- RicardoStuven

---

Two things to keep in mind about this material:

• What is its ethical status? Not much. It's not value-free, but principles such as simplicity can certainly be used for a variety of incompatible ethical goals. The engineering approach is about how to get the results you're aiming for, not about what to aim for. For the latter, you need ethics.
• Not all of these principles can be used in all situations. For example, the 90% solution isn't acceptable if you're making a radiation therapy device.

-- BenjaminGeer

---

This work is licensed under a Creative Commons Licence. To contact us, please use the openorg-dev mailing list.

---