Large and complex projects make a large and complex mess?

I have written already earlier about the commonalities between building houses (maybe it would be better to generalise it by saying construction projects) and computer systems. It so happens that Tekniikka&Talous writes about the results of a survey made by Independent Project Analysis (IPA), of which IPA's head of Europe and Middle-East Mary Ellen Yarossi talked in a seminar held by Bentley in London in the end of October. I couldn't find any international news coverage on the event, and the full survey results likely are not available freely, but I'll refer here some of the highlights published by T&T on it.

IPA had analysed almost 17 000 construction projects (world-wide) of which 500 were classified mega-projects. Of those 500, two out of three had failed when schedule or budget overflowed over 25% or there still were unresolved issues after two years of official project ending. The survey also concluded that large projects fail more often than small ones, and projects which use new technology, fail more often than the rest.

Now, why does that not sound surprising? And from projects with new technology it's pretty easy to draw an analogy to software projects. I mean, new technology is introduced pretty often on this trade (construction business, in contrast, is usually quite conservative, but the direction is toward ever more challenging environments and larger structures which require new technologies).

Curiously enough, the survey also concluded that a common problem was that the objectives set for the project are not clear to all parties, or the objectives are not understood the same way, or they are in conflict (analogous to common issues with software requirements). Also they noted that in long projects the slipping of schedule in one phase is thought to be possible to compensate by shortening later phases, although in reality that only introduces more problems (as far as I know, testing tends to often get this kind of treatment in non-agile SW projects, which was also brought up in the "Obamacare" case lately).

As the seminar was held by a software house it should be no surprise that there seem to be common factors with software business, but I find these pretty striking and central. Thus, it might be that it is possible to learn from the mistakes of the other trade what comes to these things - and it's not just (us) software folks who are not that good with large and/or complex projects (it is us humans who are miserable in sticking to good practices and proper processes).

Value in simple tools: psloggedon

Scenario: You notice a stream of requests ending up in a production server error log that clearly hint you of a misconfigured software client on a laptop (running Windows) in the company network. Checking the IP address against DNS gets you the computer name, but checking that against the company CMDB you only get a name of an former employee. The errors in the log are ugly and you want to get rid of them, but as you don't have admin access for laptops, you can't use the regular Windows admin tools to figure out who's using the darn thing. Asking for a workstation admin to get you the information might be one way, but there's also another way...

Solution: Grab psloggedon from pstools package (and nevermind if your copy is six years old, as mine was). Issue

pslooggedon \\hostname.yourdomain.com

on the command line and there you'll see all the accounts that are currently logged on, and can contact the user in question.


I acknowledge there might be plenty of other ways, too (myself I tried also using msg to message the user but that seemed to be blocked somehow, or maybe it was a conflict with Win7 vs. WinXP), and you're welcome to share similar stories in the comments.

Coding for the kids

There has been some discussion lately here and also abroad (see Oct 2013 Wired and the president of Estonia for BBC news) on whether kids should be taught computer programming already during the first years at elementary school, or even in kindergarten. I don't know about the latter, but definitely programming languages should be easier to grasp than any natural language due to the more limited syntax and vocabulary, so if a 7 year-old can figure out the basics of a foreign language, programming shouldn't be any harder.

Myself I was introduced to coding with Sinclair ZX81, and since my dad had bought only the computer and not any programs, the only option was to either type in a program from a magazine or book - or to write it yourself. I was about nine years old at that time, I think. Come on, writing BASIC is like writing English, which I was introduced to at school around the same time.

Of course not all kids (neither when they are young nor when they grow up) need to get deep into coding, but I have to agree with writer Elina Lappalainen and Ilkka Paananen of Supercell (both in Talouselämä 38/2013) on that knowing the basics of how computers are made to work is something that can be of incredible benefit later on as dealing with computers has become so everyday thing that it would take an effort to not to use any for even a single day. I've long been thinking that the fact that buyers in software projects don't actually know anything about what software really is (or how it is made) is one big reason for why software projects go as they tend to go (i.e. badly). If they had had a peek at how things operate within computers in school, things might have been different later on.

After all, I think kids are still today taught in Finnish elementary schools the basics of handicraft (both with wood and textiles) and cooking, not to mention sports and arts, why shouldn't we be teaching them also something about computers - it would be potentially just as important for their future.

Consumer services going corporate

Lately I've been involved in implementing Google's services in an enterprise setting. It has been rather amusing at times, and rather irritating at other times. Somehow I have had the feeling, that the enterprise service model is still so new that it hasn't made it's mark on the services yet, or maybe it is Google's willful decision to be primarily a consumer service company.

(image by Bartosz Kaszubowski)

I base my claim on the fact that everything is always tied to a personal account (and even though you can use an account in multiple devices or for multiple users, the policies forbid that kind of usage), so we eventually need to sync our Android user base to Google's cloud and manage them there as individuals (although grouping is possible). You also need to adjust to certain things like using a (personal) credit card to pay for e.g. the developer console (after which the card is attached to the account from time to eternity) which to an European feels awkward compared to invoicing, and when communicating with a retailer they - obviously - insist on using Google Hangouts which just doesn't work if you don't have an Google+ account - so you end up either using your personal account or creating a new one for that purpose. In fact, I have currently five Google accounts including the corporate one, I don't know if that is much but it is a ridiculous amount of accounts to have as a user for using the services of a single company. Of course that is partly due to my explicit wish to separate some things, like not have my personal account tied to my work celly (or maybe an Android celly of any kind, to make it easier not to accidentally sync loads of stuff in there). At least now I can ditch the one I created specifically for work purposes, since the corporate account replaces it.

Also, Google does seem to be pushing their services pretty hard, even though I wonder how many enterprise customers (and I'm talking about medium or large companies here) are interested in having corporate Gmail account enabled for their employees (or Google's calendar, address book, instant messaging, document management & file sharing etc). It was some work to adjust the 70+ services so that they didn't seem awfully wrong...

That being said, I must admit that at least Google is trying hard. They have a wealth of support documentation on-line, which seems to be pretty correct and helpful at least if you're using English language for your services. When we finally got the Google Apps for Business up and running, there was such a huge amount of stuff in there to adjust that it made some kind of an impression to at least this systems nerd. The web based admin UI is a bit slowish at times, and it is also complex as hell but then again so is the service offering.

We also ended up transferring the developer console (that we had opened already earlier not knowing we shouldn't do it quite yet) to the matching enterprise account and while the process was incredibly difficult from the user point of view (had to consult both the support docs and the retailer to get through it) and while it required paying another $25 for opening the enterprise developer console, it also allowed requesting a refund on the previous payment and moving all data between the accounts (even though most of the services didn't seem to support the automatic transfer, including dev console). I'd bet that not all similar services offer that possibility.

Currently we have entered the pilot phase with Google's enterprise offering, and it will be interesting to see how things go. At least there are plenty of possibilities to be reaped for the per device license fee... And the users seem to be interested on getting Android devices as their everyday tools.

Old SQL, new SQL or no SQL at all, now that's the question

(image by luana1985)

It's about time to blog something about NoSQL since that is on top of it's hype cycle according to Gartner (as of July 2013, and actually key-value DBs are already sliding fast into the Through of Disillusionment already). And boy, does it show? Last summer I dived in the vast depths of Youtube searching for interesting talks and presentations around this subject and at least most of the younger lads presenting their favourite product (which of course is totally awesome) acted like that is the key to everything. There had even been a "Battle of the Backends" in Google I/O 2012. Ok, have to admit, that was quite entertaining.

As a sidenote, I kind of hope that the confusing name NoSQL would soon be forgotten as some of the products anyway support some sort of structured query language (if not even the standard SQL) and the alternative name used by some, NewSQL, is not much better since the "old" (R)DBMSs are starting to gain features from the "new" products. Besides, the basic ideas are not all that new anyway, for example Lotus Notes had a document DB already way back in the 90s. And what comes to in-memory DBs, even Oracle has TimesTen that can act either as a memcache or as a independent in-memory (R)DBMS. I don't think it is even that useful to bundle together things as different as key-value stores, column-family stores, documents stores and graph DBs (and what else there is). So it's probably better to talk about them separately without using the buzzwords.

Back on the track... Marketing speeches aside, it's not likely that the new approaches are the key to everything (the older lads probably had already seen the "coming" of object DBs and knew it better). I don't know who conducted the MySQL-SenseiDB comparison tests published on SenseiDB website but if your solution somehow looks like 100x faster than a widely used solution, while running a most simple test scenario, it is quite likely that either your test setup is flawed or you're outright messing up the results. Well, maybe there is a reason why their web site is also still in the year 2012 now in Sept 2013...

I think I introduced myself with Youtube and some textual resources to at least MongoDB, RavenDB, Cassandra and Hadoop, with mostly more than one longer presentations plus a couple of clips on the current DBMS scenery in general and hey, I admit, many of those were just terrific talks and I did get enthusiastic about the subject. After the upsides of the non-RDBMS approaches were starting to dawn on me I could quickly think of two example cases where an RDBMS was not quite giving adequate performance. Unfortunately the other of them is tied to proprietary software and likely will support a non-RDBMS in production somewhere in the next decade - if ever - even though e.g. a document oriented approach might fit it very nicely. With the second case I'm free to try out everything myself, and inspired by some of the DB design related presentations I already did some changes to the DB design on the current platform (MySQL) with which I was able to get 50% off from a particular query that took very long. However, that was just plain old denormalisation, nothing new in that, and the initial design might have been bad in practice, anyway, even though it was kind of a textbook case of a relational DB model.

What I intend to do is to try out similar designs on both MySQL and a document DB (probably MongoDB), since the data lends itself quite well for document-oriented design. This will mean having to stuff arrays of things in a single field in MySQL but as I'm most interested in the DB performance that is ok (and subsequent parsing in the application shouldn't be much of a problem, either). The thing I'm most curious about is how easy it is to design the DB so that it still allows querying from different angles. At the moment my expectation is that I probably need to duplicate the data in some parts to allow that, but then again that is also required in reporting DBs (think of cubes).

Stay tuned (but don't hold your breath since I can't promise the next post will arrive before your brain cells die of asphyxiation)...

Polyglotism

Lately, I've been educating myself on the topic of what is commonly called NoSQL, and this nice presentation by Martin Fowler made me think about stuff I've already written about earlier. The same thing can also be found on a talk on data storage technologies employed in Craigslist.



The key phrase was polyglot persistence (@~51 mins in the video). Not that I wouldn't probably have heard it already earlier, but now the ultimate meaning finally dawned on me. It means, that to build a system - one that is highly performant, at least - you probably need to be able to take advantage of the different DBMSs that are around. The same way that multi-lingual developers push the idea that you need to use the language that is right for the job. So the diagram Fowler gave on polyglot persistence could be extended with programming languages with which the different parts of the system have been built with, and probably also with OSs the parts run on, and so on. There is a reason for dozens of DB brands and programming languages and whatnot existing, and it is the attempt to produce proper tools for specific problems. I can see this could also lead to a sprawl of diversifying computational environments, but I guess the constant popularity contest that goes on between products makes either the products converge or the niche options quietly fade away with the more popular ones gaining market share.

All this polyglotism further widens the options for a fresh wannabe ICT specialist. There certainly is a need for narrow deep expertise (to be able to know something like the Spring framework from top to bottom one really needs to spend considerable time with that alone), but I do not see that it would become very common for companies that are not actually huge in size and less than 100% technology oriented to hire bunch of specialists of different areas. Or they could, but then again they could hire them just as well as consultants since they are not likely to need each one of them equal amount of time constantly, and if the team members do not know much about each others areas of expertise they really can't be assigned to work on the same task together. My experience from the service sector is that there is a huge need for a team of people small enough that it can constantly be kept busy with the ongoing tasks and that you can throw at any kind of a problem and they can handle that based on their collective wide experience.

What I am after here is that even though I've been mostly using Oracle (from version 8 onward, I think) as the data back-end in my professional life,  the potential of interesting non-RDBMS encounters is highly rising. All along the road I've been running into cases where it is not feasible to stick to the good ol' 3NF of data and instead denormalising to get performance - something that the NoSQL folks are very fond of. This far it has been a convention that if an application needs a DB (bigger than can be reasonably embedded within the product on the same server), it supports one or more of the big traditional RDBMSs. Now I already see RHQ including Apache Cassandra in the package, and even though Cassandra comes bundled with the product, one basically needs to install it on a server of its own for practical use - so basically it can be said RHQ requires Cassandra in addition to one of the supported RDBMSs. Personally, this made me think of how I can sell the idea of requiring another server for application server monitoring to my superiors...

Your old sins will come to pay you a visit once more

Some time ago, I had to shut down both of my Linux boxes at home since I needed to cut off electricity from some rooms while installing a ceiling fan (remember working safety!). All was well with the fan after switching electricity back on but then I tried getting the older server box up. It said it didn't have any disks. Well, that is a sure way to pump up stress hormone levels, and it did work on me again. The first suspicion was that since the hardware is rather old and had not been off for a looong time maybe some electronics just had died. But no, it didn't see either of the attached disks. Since that box was serving also as DHCP, DNS and gateway for LAN, and it was getting late in the evening, I just hacked the needed services up on the desktop box which luckily had pretty much the same firewall config already. DHCP didn't still work, though, and since we were about to leave on a holiday trip I just left it that way (later on I discovered there was just a stupid mistake in the config but that's the way it always goes, right?). Professionally it felt bad to leave my lady's web site offline, but she didn't seem to mind the long SLA.

After returning from the trip I took another try on trying to get the issues sorted out. Trying out a live cd proved that the hardware was alive and well. I had already been suspecting something about udev earlier based on some googling and found some info on kernel options CONFIG_SYSVS_DEPRECATED and CONFIG_SYSVS_DEPRECATED_V2. Disabled those, recompiled the kernel and behold, all was well again! I have absolutely no recollection why those were enabled (probably for supporting some older utilities), they had been that way for years already. Seems like Debian had evolved their udev implementation so that it had become incompatible with those features somewhere after the last reboot (which on the server box was 1.5 years earlier as I found out from the uptime logger afterwards). Serves me right for keeping a feature that is clearly meant to be transitory enabled for so long...

Well, this week I noticed that I had no sound on the desktop. First I just thought it was due to Flash which was not working in Opera anyway, so I installed Gnash and its dependencies. Too bad there was still no sound. No sound from much anything, actually. I got a crash course on sorting out Pulseaudio issues, but the end result was that I only had a dummy sound device available and even though Alsa seemed to recognise the emu10k1 device, there was just no sound at all.
Then I got a flash of enlightnenment. Those damned kernel options... the kernels were identical excluding things related to differing HW on the boxes, so also the desktop had the same deprecated features enable. Needless to say, disabling them made sun shine on me again. They really do seem to mess up with udev in unpredictable ways.

Uh, I just need to start paying more attention on what these boxes have installed and configured and not let them deteriorate....

Hazards of sitting

Having been reading monthly occupational safety reports at work has increased my consciousness on safety related to work in general (very good for preventing some stupid home injuries) and electricity related risks (also very good thing to know if you are the kind of guy who doesn't fear to make some minor changes to home electricity, like installing a ceiling lamp that doesn't have the fancy and safe kind of a plug for connecting it). Other than that, I guess, for a guy who sits at the desk all day long the most likely accident at work is slipping on the office floor. And oh yes, I have once dropped a thick folder full of papers on my big toe, which resulted in a small bruise. Probably not reason enough to request for safety shoes with anti-slip soles and toe protection, though.

Those reports and subjective experience has also increased consciousness on the fact that long term sitting in front of a computer is a hazard to one's health. Not an acute one, though, and perhaps it is not that high profile on corporate health and safety agenda - except (hopefully) in software houses. Generally computer work is taken as such that it can be done in a condition where manual labour is not anymore possible (the most important muscle in a specialist is the brain, anyway). However, there is potential for many kinds of pain which might even be debilitating and thus affect productivity, so to me it looks a very real threat for working health.

I remember times when stiffness and tension caused by crouching over the keyboard was easily taken care of by some physical workout, stretching and a walk outdoors. By that time I didn't do yoga asanas, but I'm sure those would have worked, too.

I miss those times. Nowadays I'm happy if there is no notable tension and the stiffness is not painful. I've increased my awareness on the condition of my body as well as my posture, and I try to have short breaks during the day for doing some exercise and stretching at my desk. I do asana practice at least a couple of times a week, and have tried several kinds of treatments and therapies (different kinds of massage, acupuncture, reflexology...) and go to work by bike or foot. I ran into a good article on the subject and the theory given there on the reasons for the continued stiffness and discomfort seem very logical to me: the stiffness kind of works its way into the different tissues and thus in the end it's not just the muscles that behave badly.

So far I haven't used any occupational health care services since the situation is not outright bad, but I'm concerned of the directions of things for the sake of myself. I'm also pretty confident that the condition is indeed profession related (type of profession and condition do match).

The thing I'm happy about is that there is one common computer job related injury that I've managed to prevent so far - the carpal tunnel syndrome. I've been paying a lot of attention to hand ergonomy since pretty young which must have helped a lot. Maybe if I had been just as aware of the possible results of bad posture, too, things might be now different...

The reason I'm writing this is that during all these years I haven't met too many colleagues who would have indicated that they are really aware of these things. It's pretty common for office workers to go a gym regularly, and that is most likely a very good thing to do. Many might also have other hobbies that are good for the upper back area (I've heard that for example golf is great physical exercise in this sense). However, I fear that I'm not at all alone with this issue and it's worth the effort to bring up the subject.

Thanks for reading and now go do some stretching and make your body happy!

Of house and systems building

It happened that one evening I was watching the episode of British TV series Grand Designs which was portraying a usually overly-optimistic and imaginative house building project in Exeter (you may be able to view the episode via youtube). The gotcha of the series is that all the projects are focused on truly grand designs, leading almost always to a doubled budget and severely stretched schedule. Now, if that doesn't sound enough like an average system development project, let me add that quite many of the building projects also have legacy components (all kinds of conversion and extension projects) and the plans are modified along the road to accommodate to the failing budget and/or schedule - or the changing vision of the project owner(s), and the project is not necessarily led by an professional of building industry or architecture.

This specific episode was somehow unique in the sense that this time the plans really were changed remarkably in key points multiple times as the project went on, and the host of the show looked really doubtful of the chances of the owners ever getting a decent house. As usual in the series, the end result indeed was a decent house, unique for sure but not in a weird way, and definitely not ugly to look at (I wonder if they just leave out all the miserable failures or if the house builders just are so darn lucky).

Software or system development projects have been compared to all kinds of more traditional projects in the history, and software industry also has tried to take influence from other industries (like lean that originates in car industry). The inherent differences between software and its comparison points that might make the metaphor lame are debatable, as cars and houses (and maybe even more so the huge projects like bridges, dams etc, not to mention nuclear power plants) really are quite complex and technical these days.

The text book recipe for success is to have a rigorous working process that is not abandoned when difficulties arise, to avoid touching legacy parts when possible and to have professionals that are knowledgeable on the task they've taken on all levels of the project organisation. Many times it just doesn't go like that, I think. Many times there will be a sort of a cowboy mentality, "agile" or perhaps better called anti-agile (as it does not have much anything to do with the definition the signees of Agile Manifest had in mind) way of working. Many times it is indeed the legacy parts that need to be touched, which is harder, more expensive and more error-prone. Many times the key persons might be professionals of some other trades than software (no-one really expects non-coders to code, but surely any person with a technical mindset can be an architect, and financial skills are enough to lead the project because then the project surely will be on budget, right?)

The success rate of software projects is not very flattering. I've not got any recent figures at hand, but Robert Kraut and Lynn Streeter referred in 1995 in their work "Coordination in Software Development" to Robert Blazer's statement dating as far back as 1975 (!) and said that software still is unreliable, delivered late, does not respond to changes, doesn't hit the performance target and is expensive. Well, has that changed since then? I cited that paper in my MSc thesis in 2007 and my conclusion is still the same: that does still happen in large numbers.

Looking at the news here, I'd say that the success rate for building projects is not very good, either, especially the ones that are not large and demanding enough to clearly require the best practices and the best people. Just like the resulting software is likely to have security issues, the resulting buildings (at least in this country) are likely to have issues with moisture and/or unhealthy athmosphere inside the building. Neither of those would be hard to avoid, though, either by thinking security already in the design phase (for software) or applying basic building physics in the design phase and carefully sticking to the plan through the building phase (in construction work). But like in the TV series, when asked about the expected end time, it can be found out that there is no detailed schedule, and the project owners have the habit of changing the design as they go along (each change perhaps leading to other changes), and the project lead might be technical, but not exactly from the right branch of science...

As for the couple that featured in the TV show, I have absolutely nothing to say to how they dealt with the house they were building for themselves - it is solely in each one's own decision and risk. After all, they were amateurs when they started the project (as much as I am what comes to building a complete house), and should thus be allowed to learn as they go. However, the same mentality and way of working would be very strongly objectible if it occurs in the context of professional work.

So, what all this babble boils down to, I assume, is that some software projects are either led or completely run by a bunch of amateurs, or a bunch of people acting like amateurs, which is sad and unfortunate (both for the industry itself and the users of the software). I'm not into poking fingers at people, and I must confess that I've done plenty of amateurish mistakes so far and likely will discover new amateurishness in my actions in the future. Live and learn, or should I say fail and learn...

Since my sources are a bit dated already, I did some quick googling on the current situation:
Why Projects Fail - Facts and Figures
The Failed Record of the Software Industry

2010 IT Project Success Rates

It would seem that there, indeed, has been some improvement, and the agile movement really seems to be able to deliver what they've promised. However, the papers referred in the above sources still contain the same complaint and worry: that it is the people who make the projects fail. People, who do not act in a professional way.

From a certain angle computers creating their own software sounds like a good idea, even if it is quite sci-fi sort of an idea...


Post addendum

Some more survey results:
http://thisiswhatgoodlookslike.com/2012/06/10/gartner-survey-shows-why-projects-fail/
http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php


And then yet later on (13th Nov 2013):
A good example of a gigantic failure in a gigantic project is the US healthcare.gov project that has been under scrutiny lately.

All work makes Jack a dull boy?

Sometimes I've been pretty frustrated over the fact that I need to worry about my employer's business also outside the office hours. That's an unavoidable side-effect of administering a production system, I guess, and it does not diminish at all by lack of quality within the administered system. I mean, I do like my job, but I also do like the rest of the stuff I like to do when I'm off-duty, and most of all I do like to sleep at nights. Unfortunately the system has its best time window for service from 10 pm to 4 am, which completely overlaps my best time window for sleeping. And I do have trouble getting sleep after late night work.

Whining aside - since that is the point in this post, at least so far - I ended up seeing a documentary about the poor people of India called Nero's Guests (this was actually already last summer, but never mind, and I think everyone should educate themselves with that kind of documentaries but that, too, is not the point here). One of the people portrayed there was a single mom of I think 3 young kids who worked a long way from home. She woke up around 4 am, washed up, cooked breakfast for the family and went to the train station at 6 am. After the day of low-paying hard work she returned home on the train after sunset around 23 pm (her kids obviously had had to take care of their own dinner since they were already asleep by the time their mom got home) and went to bed. I don't remember the details anymore, but I'd guess that went on seven days a week (or maybe she was lucky and was able to have one day off from work per week). Compared to that, the discomfort of having to work 37.5 hours a week with occasional hours hitting outside of the oh-so-convenient 9 am to 5 pm window is minuscule.

That is not by far the only example. Like the farmers of yesterday, the poor of the world today do not have a separation of work time and free time. And I think farmers still have the same schedule that nature dictates (work when needed, rest when you can). Not exactly the most prominent role models of today, but what is it that they seem to actually cope better with their circumstances than me with my own?

Actually the habit of having to be at the office from 9 am to 5 pm was inherited from the days of industrialisation in the 19th century, and has not much to do with the current ICT work that is not tied to place or time. In a factory it was - and to some extent still - is crucial to have the people at the conveyor belt (or the monitoring console in a more modern factory) all the time, otherwise the production will likely suffer. Since that is not the case with modern information work, we have things like flextime (ok, that's rather old thing already), remote work and flex work. Basically the information workers of now can put in their effort at the time they feel most comfortable and sometimes even in the amount they feel comfortable with at that particular time. Working in cafés, park benches, sofas etc is kind of trendy these days, and even the less-trendy companies start allowing that kind of work.

I've been doing flexitime all my career, and I appreciate the freedom it offers. I also appreciate that my current employer does allow occasional remote work days when I need it (although I do have better ergonomy at the office, so working a full day at home is rather heavy physically). This clearly shows that I have absolutely nothing to whine about in my working circumstances - so far I even got the luxury of a private room at the office (remains to be seen how long) - something not so trendy but yet so beneficial for concentration.

I've been working with my mindset some time now to mend this mental issue of mine. As I'm interested in yoga I like to view it also from the karma yoga angle - some things just need to be done due to that being my duty (applies also to the multitude of things not so pleasant yet mandatory in life in general). If I was as enthusiastic about my work as some are, things would be easier. I wonder whether it is about having a "perfect" job for oneself or whether it is about the mindset, or both. Nevertheless, not all can have a "perfect" job. In fact, "perfect" is something that likely does not belong to the domain of material life, so it must be about the mindset. It is rather ironic that if one sees things in a positive way, everything is so much easier.

After all, I'm being pretty darn lucky.

The yoga of software engineering

Time to finalise this post, it has been brewing there already for some time now...

The following is an attempt to map raja / ashtanga yoga to software engineering. The connection between the two might seem a bit vague at first, but let's see if we can find something in there...

Ashtanga yoga, also know as raja yoga or classic yoga, contains eight limbs or pillars or steps or whatever you want to call them. They prescribe the students how to progress on the path to the ultimate goal, which in case of yoga is either union with the God or realisation of the God-like part in themselves. With relation to SWE the goal might be to build a piece of software that has maximum compliance with the requirements, performs cleanly and swiftly, has good user experience and is secure. So the yoga of software engineering is about developing oneself in the profession so that one is, some day, honored among both co-workers and customers for being an efficient worker producing quality code and known for not straying from the golden path of best practices.

The first two pillars are yama and niyama, yama meaning self-restraints (what one should not do) and niyama requirements or obligations (what one should do). They help building the ethical platform without which the rest of the path will most likely fail to lead to the desired result.

The five yamas are ahimsa (non-violence), satya (thruthfulness), asteya (non-stealing), brahmacharya (sexual abstinence) and aparigraha (non-covetousness).

1. Non-violence would mean (in addition to not abusing anything or anyone physically) in the context of SWE to not violating any interfaces or specifications etc. 
2. Thruthfulness is easy to see in the context of pretty much every aspect of working life, with SWE that would mean being truthful to both the team and the customer, not hiding bad code or compromised design decisions but confronting and solving them before they give rise to more serious issues, and telling the customer the real status of the project and not trying to cover things up. 
3. Non-stealing means one should not literally steal code (unless licensing permits such code-reuse) as well as not violate any licenses but perhaps also not stealing the time of ones co-workers. 
4. Abstinence from sex... well, that might not be required in SWE (even though a stereotypical coder might get intimate as rarely as a monk), but abstinence from activities that drain time and energy from the yoga of SWE might be a fitting interpretation. That could mean abstinence from fulfilling ones lusts, which might include things like hanging out in social media or gaming sites during working time, or even cheating on ones employer by doing some work for other parties (i.e. professional adultery, unless permitted by the employer). 
5. Non-covetousness (or non-greediness) can be seen to mean both not trying to gain personal glory and/or benefit and also referring to holding unto the KISS and YAGNI design principles. Also trying to include too many things/functions within a module/class should be avoided.

The five niyamas are shacha (purity both inside and outside), santosha (contentment), tapas (austerity), svadhyava (study of religious books and repetitions of mantras) and ishvarapranidhana (surrendering to God).

1. Purity is needed both in design, code and personal hygiene, so keep both your design and code neat and clean, keep your head clean so you can think straight and get a shower more than once a week.
2. Contentment is required to be able to adjust to the limitations so that no precious energy and time is used on fighting wind mills.
3. Now there is a sanskrit word with rather varying meanings, but it might be seen here as having to take certain amount of discomfort along the way of meeting the goal. If we're gonna advance, it will take practice over an over again. Some of the connotations of the word also can be interpreted as holding KISS and YAGNI dear.
4. Study of SWE books is, of course, necessary to improve one's professional skills on SWE, and it might even be helpful to pick up some good quotes (guidelines, one-liners etc) and remind oneself of those things repeatedly, thinking about what they really mean.
5. Total surrender to anyone might not be the best thing for a SW engineer to do as individual thinking and challenging the current design is a good way to improve the design and thus the resulting piece of software. But curiously enough, many experts of their trade are highly dedicated to the thing they are doing, and certainly that does also apply to SWE.

The third pillar is asana, which in yoga tradition means postures (which is the only part most  people ever know of yoga). Asana practice is meant to develop one's tool (the body) so that it is fit and properly diciplined for the challenges that follow. In SWE one needs also proper practice (just read Practicing Programming by Stevey Yegge if you feel like disagreeing). Coding itself is not enough to develop oneself, since there are always external constraints that dictate things like how much time there is for a given task - which prevents adopting new methods or tools that would need time to be learned. If there is always a hurry to get an adequate solution, when will one have time to implement good (or even excellent) solutions? From my experince I'd dare to answer "hardly ever".

The fourth pillar would be pranayama, which in yoga tradition means controlling the life force, but like this part is done in the yoga tradition, I'll leave that and the rest of the story for the private teachings passed from the SWE master to his or her disciples. Besides, I wouldn't even be competent to give any deep teachings - being in dire need for such myself, too.

How quality degrades

I by no means assume the following does not fall into some sort of logical pit along the way, but I hope my idea will be conveyed anyway.

On a layered architecture each level has its individual degree of quality, but the total quality is not a sum over the parts but more like a multiplication over them, meaning that adding a top quality layer does not improve the whole (if not makes it any worse, either) and adding a less-than-perfect one will drag the total quality down somewhat. Adding quality would actually mean fixing a bug in another layer, which is not a good idea to do anyway. The whole can be as bad as the worst part within it.

What does that mean, then?

This week I took my first run on a web-based service for creating, editing and publishing video content. All the pain of being able to produce a nice enough, coherent and clearly voiced material aside, I also ran into some problems with the tool itself. I do not know if the service itself had occasionally some trouble handling all the traffic, or was my network connection bad somehow (the office WAN connection is known to have latency issues), or was it the fault of the browser (tried both IE and Firefox, though). Nevertheless I ended up logging in and out and restarting the browser and lost two good audio takes due to all that, which is not adding any glory for the service or its provider.

How are regular users seeing that kind of things?  They point at the service provider and say "your service is not good" as they do not see the different pieces in the whole. Is the service provider at fault? Many times that might not be the case. Does the reputation of the service provider take hit? Yes. Does it take the user support person a great amount of patience to get through the idea that the service provider does not (and can not) be responsible in any way of what there is between the user and their network? Most likely yes.



Thus, it is really important to ensure top quality all over the stack, and also beware of the pitfalls of having your product being stacked with layers you have no control over, since it may affect your brand. There are of course ways to protect against things like network twitches etc by properly handling all error conditions and taking care of handling communication time-outs gracefully - all signs of good quality software!

On (mis)communication

Reading a district manager's blog post on (mis)communication on our intranet made me think. His point was that although everyone is complaining on being assaulted by too much information, there is not enough communication - or more precisely it does not reach the audience.

Today I have communicated by unstructured means (email, instant messaging, phone) and (semi)structured means (ticketing system, supplier wiki) and I must say that I love the structured way for just that, structure. There is too much unstructured information flying around. Even though IM and email are fine for short term use, IM discussions are just a mess from archival point of view and email... well... there's simply too much of it. From a well structured wiki or a ticketing system you can find the information easily and stay updated on the state of affairs, and they are maybe even easier for the author (no need to think of who should receive the message or to keep references to things said and done earlier).

But blogging, yes, I was going to say something about it... What I find a huge problem at work is to reach the users when I have some instructions or tips to share. The usual way would be to send email to a mailing list, but the same applies to those emails as what is the district manager's grief - the message does not reach the receivee (or at least it does not get fully through). If there is more than one sentence some people stop reading, and if the subject does not immediately feel important to them at that time, they ignore the message (and thus I need to repeat the same things over and over). Besides, passing instructions by email is bad just for the reasons email in general is bad (hard to find anything after some time has passed). Thus, having a blog on the intranet struck me as actually a good media to both share some simple tips and also to notify people on new instructions that might be stored somewhere else. If the blog gained followers, it might even work as a feedback channel for development of new features (or gathering knowledge on what the user base really would like to change). A colleague of mine actually has started a blog of his own on just the same purpose, spreading out the word on current issues and hot user tips.

The downside of adding a blog to the array of the existing communication channels is obvious - one more thing to update (and for the audience one more to follow). And it is a big if whether enough users really bother tinkering with the intranet to actually start following a blog. Then again I think the potential is great enough to take the chance and try it.