Large and complex projects make a large and complex mess?

I have written already earlier about the commonalities between building houses (maybe it would be better to generalise it by saying construction projects) and computer systems. It so happens that Tekniikka&Talous writes about the results of a survey made by Independent Project Analysis (IPA), of which IPA's head of Europe and Middle-East Mary Ellen Yarossi talked in a seminar held by Bentley in London in the end of October. I couldn't find any international news coverage on the event, and the full survey results likely are not available freely, but I'll refer here some of the highlights published by T&T on it.

IPA had analysed almost 17 000 construction projects (world-wide) of which 500 were classified mega-projects. Of those 500, two out of three had failed when schedule or budget overflowed over 25% or there still were unresolved issues after two years of official project ending. The survey also concluded that large projects fail more often than small ones, and projects which use new technology, fail more often than the rest.

Now, why does that not sound surprising? And from projects with new technology it's pretty easy to draw an analogy to software projects. I mean, new technology is introduced pretty often on this trade (construction business, in contrast, is usually quite conservative, but the direction is toward ever more challenging environments and larger structures which require new technologies).

Curiously enough, the survey also concluded that a common problem was that the objectives set for the project are not clear to all parties, or the objectives are not understood the same way, or they are in conflict (analogous to common issues with software requirements). Also they noted that in long projects the slipping of schedule in one phase is thought to be possible to compensate by shortening later phases, although in reality that only introduces more problems (as far as I know, testing tends to often get this kind of treatment in non-agile SW projects, which was also brought up in the "Obamacare" case lately).

As the seminar was held by a software house it should be no surprise that there seem to be common factors with software business, but I find these pretty striking and central. Thus, it might be that it is possible to learn from the mistakes of the other trade what comes to these things - and it's not just (us) software folks who are not that good with large and/or complex projects (it is us humans who are miserable in sticking to good practices and proper processes).

Value in simple tools: psloggedon

Scenario: You notice a stream of requests ending up in a production server error log that clearly hint you of a misconfigured software client on a laptop (running Windows) in the company network. Checking the IP address against DNS gets you the computer name, but checking that against the company CMDB you only get a name of an former employee. The errors in the log are ugly and you want to get rid of them, but as you don't have admin access for laptops, you can't use the regular Windows admin tools to figure out who's using the darn thing. Asking for a workstation admin to get you the information might be one way, but there's also another way...

Solution: Grab psloggedon from pstools package (and nevermind if your copy is six years old, as mine was). Issue

pslooggedon \\hostname.yourdomain.com

on the command line and there you'll see all the accounts that are currently logged on, and can contact the user in question.


I acknowledge there might be plenty of other ways, too (myself I tried also using msg to message the user but that seemed to be blocked somehow, or maybe it was a conflict with Win7 vs. WinXP), and you're welcome to share similar stories in the comments.

Coding for the kids

There has been some discussion lately here and also abroad (see Oct 2013 Wired and the president of Estonia for BBC news) on whether kids should be taught computer programming already during the first years at elementary school, or even in kindergarten. I don't know about the latter, but definitely programming languages should be easier to grasp than any natural language due to the more limited syntax and vocabulary, so if a 7 year-old can figure out the basics of a foreign language, programming shouldn't be any harder.

Myself I was introduced to coding with Sinclair ZX81, and since my dad had bought only the computer and not any programs, the only option was to either type in a program from a magazine or book - or to write it yourself. I was about nine years old at that time, I think. Come on, writing BASIC is like writing English, which I was introduced to at school around the same time.

Of course not all kids (neither when they are young nor when they grow up) need to get deep into coding, but I have to agree with writer Elina Lappalainen and Ilkka Paananen of Supercell (both in Talouselämä 38/2013) on that knowing the basics of how computers are made to work is something that can be of incredible benefit later on as dealing with computers has become so everyday thing that it would take an effort to not to use any for even a single day. I've long been thinking that the fact that buyers in software projects don't actually know anything about what software really is (or how it is made) is one big reason for why software projects go as they tend to go (i.e. badly). If they had had a peek at how things operate within computers in school, things might have been different later on.

After all, I think kids are still today taught in Finnish elementary schools the basics of handicraft (both with wood and textiles) and cooking, not to mention sports and arts, why shouldn't we be teaching them also something about computers - it would be potentially just as important for their future.

Consumer services going corporate

Lately I've been involved in implementing Google's services in an enterprise setting. It has been rather amusing at times, and rather irritating at other times. Somehow I have had the feeling, that the enterprise service model is still so new that it hasn't made it's mark on the services yet, or maybe it is Google's willful decision to be primarily a consumer service company.

(image by Bartosz Kaszubowski)

I base my claim on the fact that everything is always tied to a personal account (and even though you can use an account in multiple devices or for multiple users, the policies forbid that kind of usage), so we eventually need to sync our Android user base to Google's cloud and manage them there as individuals (although grouping is possible). You also need to adjust to certain things like using a (personal) credit card to pay for e.g. the developer console (after which the card is attached to the account from time to eternity) which to an European feels awkward compared to invoicing, and when communicating with a retailer they - obviously - insist on using Google Hangouts which just doesn't work if you don't have an Google+ account - so you end up either using your personal account or creating a new one for that purpose. In fact, I have currently five Google accounts including the corporate one, I don't know if that is much but it is a ridiculous amount of accounts to have as a user for using the services of a single company. Of course that is partly due to my explicit wish to separate some things, like not have my personal account tied to my work celly (or maybe an Android celly of any kind, to make it easier not to accidentally sync loads of stuff in there). At least now I can ditch the one I created specifically for work purposes, since the corporate account replaces it.

Also, Google does seem to be pushing their services pretty hard, even though I wonder how many enterprise customers (and I'm talking about medium or large companies here) are interested in having corporate Gmail account enabled for their employees (or Google's calendar, address book, instant messaging, document management & file sharing etc). It was some work to adjust the 70+ services so that they didn't seem awfully wrong...

That being said, I must admit that at least Google is trying hard. They have a wealth of support documentation on-line, which seems to be pretty correct and helpful at least if you're using English language for your services. When we finally got the Google Apps for Business up and running, there was such a huge amount of stuff in there to adjust that it made some kind of an impression to at least this systems nerd. The web based admin UI is a bit slowish at times, and it is also complex as hell but then again so is the service offering.

We also ended up transferring the developer console (that we had opened already earlier not knowing we shouldn't do it quite yet) to the matching enterprise account and while the process was incredibly difficult from the user point of view (had to consult both the support docs and the retailer to get through it) and while it required paying another $25 for opening the enterprise developer console, it also allowed requesting a refund on the previous payment and moving all data between the accounts (even though most of the services didn't seem to support the automatic transfer, including dev console). I'd bet that not all similar services offer that possibility.

Currently we have entered the pilot phase with Google's enterprise offering, and it will be interesting to see how things go. At least there are plenty of possibilities to be reaped for the per device license fee... And the users seem to be interested on getting Android devices as their everyday tools.

Old SQL, new SQL or no SQL at all, now that's the question

(image by luana1985)

It's about time to blog something about NoSQL since that is on top of it's hype cycle according to Gartner (as of July 2013, and actually key-value DBs are already sliding fast into the Through of Disillusionment already). And boy, does it show? Last summer I dived in the vast depths of Youtube searching for interesting talks and presentations around this subject and at least most of the younger lads presenting their favourite product (which of course is totally awesome) acted like that is the key to everything. There had even been a "Battle of the Backends" in Google I/O 2012. Ok, have to admit, that was quite entertaining.

As a sidenote, I kind of hope that the confusing name NoSQL would soon be forgotten as some of the products anyway support some sort of structured query language (if not even the standard SQL) and the alternative name used by some, NewSQL, is not much better since the "old" (R)DBMSs are starting to gain features from the "new" products. Besides, the basic ideas are not all that new anyway, for example Lotus Notes had a document DB already way back in the 90s. And what comes to in-memory DBs, even Oracle has TimesTen that can act either as a memcache or as a independent in-memory (R)DBMS. I don't think it is even that useful to bundle together things as different as key-value stores, column-family stores, documents stores and graph DBs (and what else there is). So it's probably better to talk about them separately without using the buzzwords.

Back on the track... Marketing speeches aside, it's not likely that the new approaches are the key to everything (the older lads probably had already seen the "coming" of object DBs and knew it better). I don't know who conducted the MySQL-SenseiDB comparison tests published on SenseiDB website but if your solution somehow looks like 100x faster than a widely used solution, while running a most simple test scenario, it is quite likely that either your test setup is flawed or you're outright messing up the results. Well, maybe there is a reason why their web site is also still in the year 2012 now in Sept 2013...

I think I introduced myself with Youtube and some textual resources to at least MongoDB, RavenDB, Cassandra and Hadoop, with mostly more than one longer presentations plus a couple of clips on the current DBMS scenery in general and hey, I admit, many of those were just terrific talks and I did get enthusiastic about the subject. After the upsides of the non-RDBMS approaches were starting to dawn on me I could quickly think of two example cases where an RDBMS was not quite giving adequate performance. Unfortunately the other of them is tied to proprietary software and likely will support a non-RDBMS in production somewhere in the next decade - if ever - even though e.g. a document oriented approach might fit it very nicely. With the second case I'm free to try out everything myself, and inspired by some of the DB design related presentations I already did some changes to the DB design on the current platform (MySQL) with which I was able to get 50% off from a particular query that took very long. However, that was just plain old denormalisation, nothing new in that, and the initial design might have been bad in practice, anyway, even though it was kind of a textbook case of a relational DB model.

What I intend to do is to try out similar designs on both MySQL and a document DB (probably MongoDB), since the data lends itself quite well for document-oriented design. This will mean having to stuff arrays of things in a single field in MySQL but as I'm most interested in the DB performance that is ok (and subsequent parsing in the application shouldn't be much of a problem, either). The thing I'm most curious about is how easy it is to design the DB so that it still allows querying from different angles. At the moment my expectation is that I probably need to duplicate the data in some parts to allow that, but then again that is also required in reporting DBs (think of cubes).

Stay tuned (but don't hold your breath since I can't promise the next post will arrive before your brain cells die of asphyxiation)...

Polyglotism

Lately, I've been educating myself on the topic of what is commonly called NoSQL, and this nice presentation by Martin Fowler made me think about stuff I've already written about earlier. The same thing can also be found on a talk on data storage technologies employed in Craigslist.



The key phrase was polyglot persistence (@~51 mins in the video). Not that I wouldn't probably have heard it already earlier, but now the ultimate meaning finally dawned on me. It means, that to build a system - one that is highly performant, at least - you probably need to be able to take advantage of the different DBMSs that are around. The same way that multi-lingual developers push the idea that you need to use the language that is right for the job. So the diagram Fowler gave on polyglot persistence could be extended with programming languages with which the different parts of the system have been built with, and probably also with OSs the parts run on, and so on. There is a reason for dozens of DB brands and programming languages and whatnot existing, and it is the attempt to produce proper tools for specific problems. I can see this could also lead to a sprawl of diversifying computational environments, but I guess the constant popularity contest that goes on between products makes either the products converge or the niche options quietly fade away with the more popular ones gaining market share.

All this polyglotism further widens the options for a fresh wannabe ICT specialist. There certainly is a need for narrow deep expertise (to be able to know something like the Spring framework from top to bottom one really needs to spend considerable time with that alone), but I do not see that it would become very common for companies that are not actually huge in size and less than 100% technology oriented to hire bunch of specialists of different areas. Or they could, but then again they could hire them just as well as consultants since they are not likely to need each one of them equal amount of time constantly, and if the team members do not know much about each others areas of expertise they really can't be assigned to work on the same task together. My experience from the service sector is that there is a huge need for a team of people small enough that it can constantly be kept busy with the ongoing tasks and that you can throw at any kind of a problem and they can handle that based on their collective wide experience.

What I am after here is that even though I've been mostly using Oracle (from version 8 onward, I think) as the data back-end in my professional life,  the potential of interesting non-RDBMS encounters is highly rising. All along the road I've been running into cases where it is not feasible to stick to the good ol' 3NF of data and instead denormalising to get performance - something that the NoSQL folks are very fond of. This far it has been a convention that if an application needs a DB (bigger than can be reasonably embedded within the product on the same server), it supports one or more of the big traditional RDBMSs. Now I already see RHQ including Apache Cassandra in the package, and even though Cassandra comes bundled with the product, one basically needs to install it on a server of its own for practical use - so basically it can be said RHQ requires Cassandra in addition to one of the supported RDBMSs. Personally, this made me think of how I can sell the idea of requiring another server for application server monitoring to my superiors...

Your old sins will come to pay you a visit once more

Some time ago, I had to shut down both of my Linux boxes at home since I needed to cut off electricity from some rooms while installing a ceiling fan (remember working safety!). All was well with the fan after switching electricity back on but then I tried getting the older server box up. It said it didn't have any disks. Well, that is a sure way to pump up stress hormone levels, and it did work on me again. The first suspicion was that since the hardware is rather old and had not been off for a looong time maybe some electronics just had died. But no, it didn't see either of the attached disks. Since that box was serving also as DHCP, DNS and gateway for LAN, and it was getting late in the evening, I just hacked the needed services up on the desktop box which luckily had pretty much the same firewall config already. DHCP didn't still work, though, and since we were about to leave on a holiday trip I just left it that way (later on I discovered there was just a stupid mistake in the config but that's the way it always goes, right?). Professionally it felt bad to leave my lady's web site offline, but she didn't seem to mind the long SLA.

After returning from the trip I took another try on trying to get the issues sorted out. Trying out a live cd proved that the hardware was alive and well. I had already been suspecting something about udev earlier based on some googling and found some info on kernel options CONFIG_SYSVS_DEPRECATED and CONFIG_SYSVS_DEPRECATED_V2. Disabled those, recompiled the kernel and behold, all was well again! I have absolutely no recollection why those were enabled (probably for supporting some older utilities), they had been that way for years already. Seems like Debian had evolved their udev implementation so that it had become incompatible with those features somewhere after the last reboot (which on the server box was 1.5 years earlier as I found out from the uptime logger afterwards). Serves me right for keeping a feature that is clearly meant to be transitory enabled for so long...

Well, this week I noticed that I had no sound on the desktop. First I just thought it was due to Flash which was not working in Opera anyway, so I installed Gnash and its dependencies. Too bad there was still no sound. No sound from much anything, actually. I got a crash course on sorting out Pulseaudio issues, but the end result was that I only had a dummy sound device available and even though Alsa seemed to recognise the emu10k1 device, there was just no sound at all.
Then I got a flash of enlightnenment. Those damned kernel options... the kernels were identical excluding things related to differing HW on the boxes, so also the desktop had the same deprecated features enable. Needless to say, disabling them made sun shine on me again. They really do seem to mess up with udev in unpredictable ways.

Uh, I just need to start paying more attention on what these boxes have installed and configured and not let them deteriorate....