(R) evolutionary changes

When developing software you are dealing with changes. Changes in what your creations have to do; changes in the way you create your software. The process of change sometimes seems to be a goal on itself. We are almost beaten to death by both the enormous amount of new tools and techniques to create software. To some of our customers changing specifications has become a form of survival on itself. All too often the words revolutionary or evolutionary come by to stress the importance of change. In this post I want to take at change from the perspective of my original studies as an evolutionary biologist. I do know biological evolution can be laden topic, not just in the US. I do take evolution as a given part of biological science. Developing software is a science as well; which deals with observation and repeatable experiments. Developing software can even be a form of art but it does not answer any questions on the meaning of life. I'm quite happy discussing the non scientific aspects of evolution with you, but not here on CodeBetter.

On revolutionary changes I can be short. In the days I went to university kicking against anything available was all part of social life. But I don't think revolution taken too seriously leads to much good. Looking at history shows so many well intended fresh ideas which ended in a situation worse than the one they tried to overthrow. CodeBetter used to be the basis of a revolutionary blogger who considered the state of software development to be in status quo which required an uprising of the coding masses. In his struggle he alienated quite a few people from a cause which was on itself not bad at all.

Let's now take a look at some aspects of evolution and see what happens when you try to apply them to software development.

At the basis of evolution lies variation. Not all offspring has the same qualities, those whose qualities are best fit in their environment are selected as parents for the next generation. There is also a huge variation in the tools to develop software. Those tools which produce the most working software are most likely to be selected for the next projects. It is a survival of the fittest. Note that this selection happens at time of reproduction. A strong parent which carries a lethal disease which does not expose itself until a later age is not selected against. A software tool which produces a shiny project which was built with a lot of wizardry is likely to be selected for the next project. At a later age of the project, when it has turned out to be un-maintainable, the tool has already spread.

So this selection which happens at the time of reproduction is like taking a snapshot. In the next generation, under circumstances which are different, selection can favor something completely different and push change into an opposite direction. In many an old textbook on evolution you will see progressive evolutionary lines. Step by step the small rabbit-like mammal evolves in a swift racing horse. Far more famous is the line from a chimpanzee to a modern human being. The lines suggest a line of progress. A less biased study of the fossil record and the already mentioned observation of selection as a snapshot do show that evolution is in fact a wide spreading tree. With many branches which end in extinction. When you follow one specific path in this tree, jumping from branch to branch, you can construct a classic progressive evolutionary line; but it is still one of the many paths possible. The same applies to software tools. Does anybody remember languages like Algol-68 or Logo? At the time they looked like the starts of very promising lines. But they have vanished into oblivion and given place to languages with far older ancestors. Ancestors who only served a niche market for quite some time. This zig-zag behavior can also be observed within a line. Notorious are the many, many versions of Delphi. Version 5 is still alive in many a location, but version 6 is almost extinct. After that the .NET environment has lead to an adaptive radiation of Delphi versions. Some are far more successful than others. Yet another analogy is living fossils like Cobol. That language has never changed that much over the years and is still fit enough to survive.

(Image nicked here)

So far we have only observed that there are changes in life forms and in software tools. A very interesting aspect of evolution is the time scale over which these changes happen. Up till recently it was believed that evolutionary changes happen gradually over time. In every generation there is a variation in the offspring and selection will cause the generations to differ from their ancestors. When scientist actually started to track and measure this gradual change they found out that the species studied actually did not change at all over a long period of time and space. Except for small isolated populations which rapidly changed after which these changes spread over a large area. This was formulated in the theory of punctuated equilibria by Stephen Jay Gould and Niles Eldredge. (You may know Gould as the great essay writer who has reached a large audience; his major profession was that of an evolutionary biologist) This pattern of punctuated equilibria can also be observed in software. I remember myself trying to convince the Delphi crowd of the merits of .NET. Not long after that I had to explain what Delphi was to the .NET people. TDD has been the hobby horse of a minority of evangelists, but the way it is now integrated in asp.mvc looks like a rapid main stream acceptance.

So far I have named some parallels in biological and software evolution. To me the major lessons are that there is no such thing as a constant linear progress. Many a path taken can lead to a dead end; there are periods of fast changes and there are periods of standstill. There is also no such thing as a plan or a direction. There is no intelligent design; there is no direction things will go in the future. All we can do is observe what has happened up till this very moment.

But there is one very big difference between biological and software evolution. It is human culture. Biological variation is dictated by the genes. Changing these is something which only happens in Science Fiction. But many more complicated life forms, like human beings, also pass information to next generations using other coding mechanism. Like stories, books or the internet. Any attempt to capture cultural evolution using the theory of biological evolution so far has lead to stories which might look appealing but have a very weak scientific base.

So take these words with some grains of salt and do concentrate on your human skills to learn and teach to keep the process of change under control.