Large and complex projects make a large and complex mess?

I have written already earlier about the commonalities between building houses (maybe it would be better to generalise it by saying construction projects) and computer systems. It so happens that Tekniikka&Talous writes about the results of a survey made by Independent Project Analysis (IPA), of which IPA's head of Europe and Middle-East Mary Ellen Yarossi talked in a seminar held by Bentley in London in the end of October. I couldn't find any international news coverage on the event, and the full survey results likely are not available freely, but I'll refer here some of the highlights published by T&T on it.

IPA had analysed almost 17 000 construction projects (world-wide) of which 500 were classified mega-projects. Of those 500, two out of three had failed when schedule or budget overflowed over 25% or there still were unresolved issues after two years of official project ending. The survey also concluded that large projects fail more often than small ones, and projects which use new technology, fail more often than the rest.

Now, why does that not sound surprising? And from projects with new technology it's pretty easy to draw an analogy to software projects. I mean, new technology is introduced pretty often on this trade (construction business, in contrast, is usually quite conservative, but the direction is toward ever more challenging environments and larger structures which require new technologies).

Curiously enough, the survey also concluded that a common problem was that the objectives set for the project are not clear to all parties, or the objectives are not understood the same way, or they are in conflict (analogous to common issues with software requirements). Also they noted that in long projects the slipping of schedule in one phase is thought to be possible to compensate by shortening later phases, although in reality that only introduces more problems (as far as I know, testing tends to often get this kind of treatment in non-agile SW projects, which was also brought up in the "Obamacare" case lately).

As the seminar was held by a software house it should be no surprise that there seem to be common factors with software business, but I find these pretty striking and central. Thus, it might be that it is possible to learn from the mistakes of the other trade what comes to these things - and it's not just (us) software folks who are not that good with large and/or complex projects (it is us humans who are miserable in sticking to good practices and proper processes).

Coding for the kids

There has been some discussion lately here and also abroad (see Oct 2013 Wired and the president of Estonia for BBC news) on whether kids should be taught computer programming already during the first years at elementary school, or even in kindergarten. I don't know about the latter, but definitely programming languages should be easier to grasp than any natural language due to the more limited syntax and vocabulary, so if a 7 year-old can figure out the basics of a foreign language, programming shouldn't be any harder.

Myself I was introduced to coding with Sinclair ZX81, and since my dad had bought only the computer and not any programs, the only option was to either type in a program from a magazine or book - or to write it yourself. I was about nine years old at that time, I think. Come on, writing BASIC is like writing English, which I was introduced to at school around the same time.

Of course not all kids (neither when they are young nor when they grow up) need to get deep into coding, but I have to agree with writer Elina Lappalainen and Ilkka Paananen of Supercell (both in Talouselämä 38/2013) on that knowing the basics of how computers are made to work is something that can be of incredible benefit later on as dealing with computers has become so everyday thing that it would take an effort to not to use any for even a single day. I've long been thinking that the fact that buyers in software projects don't actually know anything about what software really is (or how it is made) is one big reason for why software projects go as they tend to go (i.e. badly). If they had had a peek at how things operate within computers in school, things might have been different later on.

After all, I think kids are still today taught in Finnish elementary schools the basics of handicraft (both with wood and textiles) and cooking, not to mention sports and arts, why shouldn't we be teaching them also something about computers - it would be potentially just as important for their future.

Of house and systems building

It happened that one evening I was watching the episode of British TV series Grand Designs which was portraying a usually overly-optimistic and imaginative house building project in Exeter (you may be able to view the episode via youtube). The gotcha of the series is that all the projects are focused on truly grand designs, leading almost always to a doubled budget and severely stretched schedule. Now, if that doesn't sound enough like an average system development project, let me add that quite many of the building projects also have legacy components (all kinds of conversion and extension projects) and the plans are modified along the road to accommodate to the failing budget and/or schedule - or the changing vision of the project owner(s), and the project is not necessarily led by an professional of building industry or architecture.

This specific episode was somehow unique in the sense that this time the plans really were changed remarkably in key points multiple times as the project went on, and the host of the show looked really doubtful of the chances of the owners ever getting a decent house. As usual in the series, the end result indeed was a decent house, unique for sure but not in a weird way, and definitely not ugly to look at (I wonder if they just leave out all the miserable failures or if the house builders just are so darn lucky).

Software or system development projects have been compared to all kinds of more traditional projects in the history, and software industry also has tried to take influence from other industries (like lean that originates in car industry). The inherent differences between software and its comparison points that might make the metaphor lame are debatable, as cars and houses (and maybe even more so the huge projects like bridges, dams etc, not to mention nuclear power plants) really are quite complex and technical these days.

The text book recipe for success is to have a rigorous working process that is not abandoned when difficulties arise, to avoid touching legacy parts when possible and to have professionals that are knowledgeable on the task they've taken on all levels of the project organisation. Many times it just doesn't go like that, I think. Many times there will be a sort of a cowboy mentality, "agile" or perhaps better called anti-agile (as it does not have much anything to do with the definition the signees of Agile Manifest had in mind) way of working. Many times it is indeed the legacy parts that need to be touched, which is harder, more expensive and more error-prone. Many times the key persons might be professionals of some other trades than software (no-one really expects non-coders to code, but surely any person with a technical mindset can be an architect, and financial skills are enough to lead the project because then the project surely will be on budget, right?)

The success rate of software projects is not very flattering. I've not got any recent figures at hand, but Robert Kraut and Lynn Streeter referred in 1995 in their work "Coordination in Software Development" to Robert Blazer's statement dating as far back as 1975 (!) and said that software still is unreliable, delivered late, does not respond to changes, doesn't hit the performance target and is expensive. Well, has that changed since then? I cited that paper in my MSc thesis in 2007 and my conclusion is still the same: that does still happen in large numbers.

Looking at the news here, I'd say that the success rate for building projects is not very good, either, especially the ones that are not large and demanding enough to clearly require the best practices and the best people. Just like the resulting software is likely to have security issues, the resulting buildings (at least in this country) are likely to have issues with moisture and/or unhealthy athmosphere inside the building. Neither of those would be hard to avoid, though, either by thinking security already in the design phase (for software) or applying basic building physics in the design phase and carefully sticking to the plan through the building phase (in construction work). But like in the TV series, when asked about the expected end time, it can be found out that there is no detailed schedule, and the project owners have the habit of changing the design as they go along (each change perhaps leading to other changes), and the project lead might be technical, but not exactly from the right branch of science...

As for the couple that featured in the TV show, I have absolutely nothing to say to how they dealt with the house they were building for themselves - it is solely in each one's own decision and risk. After all, they were amateurs when they started the project (as much as I am what comes to building a complete house), and should thus be allowed to learn as they go. However, the same mentality and way of working would be very strongly objectible if it occurs in the context of professional work.

So, what all this babble boils down to, I assume, is that some software projects are either led or completely run by a bunch of amateurs, or a bunch of people acting like amateurs, which is sad and unfortunate (both for the industry itself and the users of the software). I'm not into poking fingers at people, and I must confess that I've done plenty of amateurish mistakes so far and likely will discover new amateurishness in my actions in the future. Live and learn, or should I say fail and learn...

Since my sources are a bit dated already, I did some quick googling on the current situation:
Why Projects Fail - Facts and Figures
The Failed Record of the Software Industry

2010 IT Project Success Rates

It would seem that there, indeed, has been some improvement, and the agile movement really seems to be able to deliver what they've promised. However, the papers referred in the above sources still contain the same complaint and worry: that it is the people who make the projects fail. People, who do not act in a professional way.

From a certain angle computers creating their own software sounds like a good idea, even if it is quite sci-fi sort of an idea...


Post addendum

Some more survey results:
http://thisiswhatgoodlookslike.com/2012/06/10/gartner-survey-shows-why-projects-fail/
http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php


And then yet later on (13th Nov 2013):
A good example of a gigantic failure in a gigantic project is the US healthcare.gov project that has been under scrutiny lately.

The yoga of software engineering

Time to finalise this post, it has been brewing there already for some time now...

The following is an attempt to map raja / ashtanga yoga to software engineering. The connection between the two might seem a bit vague at first, but let's see if we can find something in there...

Ashtanga yoga, also know as raja yoga or classic yoga, contains eight limbs or pillars or steps or whatever you want to call them. They prescribe the students how to progress on the path to the ultimate goal, which in case of yoga is either union with the God or realisation of the God-like part in themselves. With relation to SWE the goal might be to build a piece of software that has maximum compliance with the requirements, performs cleanly and swiftly, has good user experience and is secure. So the yoga of software engineering is about developing oneself in the profession so that one is, some day, honored among both co-workers and customers for being an efficient worker producing quality code and known for not straying from the golden path of best practices.

The first two pillars are yama and niyama, yama meaning self-restraints (what one should not do) and niyama requirements or obligations (what one should do). They help building the ethical platform without which the rest of the path will most likely fail to lead to the desired result.

The five yamas are ahimsa (non-violence), satya (thruthfulness), asteya (non-stealing), brahmacharya (sexual abstinence) and aparigraha (non-covetousness).

1. Non-violence would mean (in addition to not abusing anything or anyone physically) in the context of SWE to not violating any interfaces or specifications etc. 
2. Thruthfulness is easy to see in the context of pretty much every aspect of working life, with SWE that would mean being truthful to both the team and the customer, not hiding bad code or compromised design decisions but confronting and solving them before they give rise to more serious issues, and telling the customer the real status of the project and not trying to cover things up. 
3. Non-stealing means one should not literally steal code (unless licensing permits such code-reuse) as well as not violate any licenses but perhaps also not stealing the time of ones co-workers. 
4. Abstinence from sex... well, that might not be required in SWE (even though a stereotypical coder might get intimate as rarely as a monk), but abstinence from activities that drain time and energy from the yoga of SWE might be a fitting interpretation. That could mean abstinence from fulfilling ones lusts, which might include things like hanging out in social media or gaming sites during working time, or even cheating on ones employer by doing some work for other parties (i.e. professional adultery, unless permitted by the employer). 
5. Non-covetousness (or non-greediness) can be seen to mean both not trying to gain personal glory and/or benefit and also referring to holding unto the KISS and YAGNI design principles. Also trying to include too many things/functions within a module/class should be avoided.

The five niyamas are shacha (purity both inside and outside), santosha (contentment), tapas (austerity), svadhyava (study of religious books and repetitions of mantras) and ishvarapranidhana (surrendering to God).

1. Purity is needed both in design, code and personal hygiene, so keep both your design and code neat and clean, keep your head clean so you can think straight and get a shower more than once a week.
2. Contentment is required to be able to adjust to the limitations so that no precious energy and time is used on fighting wind mills.
3. Now there is a sanskrit word with rather varying meanings, but it might be seen here as having to take certain amount of discomfort along the way of meeting the goal. If we're gonna advance, it will take practice over an over again. Some of the connotations of the word also can be interpreted as holding KISS and YAGNI dear.
4. Study of SWE books is, of course, necessary to improve one's professional skills on SWE, and it might even be helpful to pick up some good quotes (guidelines, one-liners etc) and remind oneself of those things repeatedly, thinking about what they really mean.
5. Total surrender to anyone might not be the best thing for a SW engineer to do as individual thinking and challenging the current design is a good way to improve the design and thus the resulting piece of software. But curiously enough, many experts of their trade are highly dedicated to the thing they are doing, and certainly that does also apply to SWE.

The third pillar is asana, which in yoga tradition means postures (which is the only part most  people ever know of yoga). Asana practice is meant to develop one's tool (the body) so that it is fit and properly diciplined for the challenges that follow. In SWE one needs also proper practice (just read Practicing Programming by Stevey Yegge if you feel like disagreeing). Coding itself is not enough to develop oneself, since there are always external constraints that dictate things like how much time there is for a given task - which prevents adopting new methods or tools that would need time to be learned. If there is always a hurry to get an adequate solution, when will one have time to implement good (or even excellent) solutions? From my experince I'd dare to answer "hardly ever".

The fourth pillar would be pranayama, which in yoga tradition means controlling the life force, but like this part is done in the yoga tradition, I'll leave that and the rest of the story for the private teachings passed from the SWE master to his or her disciples. Besides, I wouldn't even be competent to give any deep teachings - being in dire need for such myself, too.