Friday, July 3, 2015

The joys and griefs of having a hardware keyboard with a smartphone

My awaited TOHKBD arrived among the first batch that was sent out, just before I started my early summer vacation in mid-June. The package actually arrived before the software had even passed QA at Jolla, so I had to wait for a couple of days before being able to actually use the keyboard - that certainly was a chilling factor. Another chill was the M key which I noticed didn't have any tactile response, and I assumed it is faulty. That was confirmed when the SW was made officially available and I was able to test the keyboard for real, the keypress was registered only after applying much force. I joined the few others with similar issues by whining about it on TMO forum, also reporting it by email to FunkyOtherHalf. Dirk van L, the one responsible for the coordination of HW development, responded and asked to wait and see if it settles with use - OK, fair enough, getting a replacement keypad won't be easy anyway for a gadget that has been manufactured in a small volume, strictly based on orders. As I'm not typing this post with my TOHKBD it is obvious that things have improved a lot, although there's still huge difference in the response and the required force. There's also another key which needs more force but has normal response (if any Os are missing from this post, there's the reason) and that's perhaps even more distracting.

There are still SW issues open even after the couple of updates by Kimmo L, mostly prominent being the lack of other layouts than American. Support for other layouts requires changes in Sailfish, and it'll take some more time until Jolla releases an update with those changes.


Despite all that, I'm happy for spending the money in the project. There's no way I could (and would) have written text this long with the on-screen keyboard. I'm also confident that the SW issues get resolved sooner or later. Even now it's a delight to use the terminal app, and writing in English is fine already. Seems like not all apps support all keys (tab, home, end, page up/down being the most prominent), but the community will fix that, too, eventually. Jolla, together with the community, will also eventually enable landscape mode everywhere (definitely something wait for from Sailfish 2.0). The greatest remaining issue is the faulty keys. According to a Kickstarter project page update, all keypads should have been manually tested, but it is evident that if that really was done for all batches (or even to this batch mine came from), it was done badly. I'd hope that there would eventually be some sort of a compensation with replacement keypads for those who need them. Hoping for the best!

Oh, one more "grief": Now the originally slim-fit phone has a rather bulky look which certainly will cause comments like "is that from the 90s or what" (hey, it is still thinner than my previous trusted phone, the E90!). Due to that, also my Mapbagdrag case don't fit anymore. Well, maybe someone will, some day, make a nice case for sturdier TOHs, too.


submit to reddit Delicious

Friday, January 9, 2015

Really unlike - state of affairs with the Jolla phone 1.5 years after launch party

 It's roughly 1.5 years after the event in which Jolla launched their Sailfish OS powered phone, and a bit over one year after the first devices were sent out to people who pre-ordered. Now, as I've been using my Jolla for a couple of weeks, I want to write up something - not just to make noise of the company and their product but also hopefully giving some useful points of view for anyone considering buying into Sailfish ecosystem. After all, I did wait for myself, too, although I was waiting for the The Other Half physical keyboard - which I'm now waiting eagerly...

To be honest, Jolla is still a bit a geek-phone, made by geeks, supported by a geek community. I switched from the old'n'trustworthy Nokia E90 and I'm in heaven, but coming from Samsung Android or Apple iOS might be quite different (I've been using Samsung at work for some years now). The basics are there (of course it depends on what is "basic" to any given user), but one might still miss some things. For one, the app market is still small, even though with Android app support there's potentially quite a few apps available. Edit: The most annoying thing about the Android support actually might be that Aptoide and Yandex don't carry the same apps that are available at Google Play, or you might for some other reason get a much older version - I have reddit sync v10 from Google Play for the Android 4.4 phone and got v5 from Aptoide for Jolla (which has Android 4.1).
Of course, one must note that it is a product created by a very small team of people (comparing to the teams working with mobile devices at Google, Samsung and Apple), and it takes some time to get everything built and fine-tuned. Personally, taking everything into account, I think the result has this far been awesome, and if they keep up the same pace at Jolla, this is to become something remarkable.

There are, however. some things that are sub-optimal or downright missing at the moment, for example:
  • Android apps can't access low-level device data such as cellular and wi-fi stats, making certain geek apps not working. Edit: Cell ID stuff is missing at least, but then again it is hard to compare since you might not get the same version installed to Jolla.
  • Some Android apps also can't seem to access location data, even though at least GPS data is available. This might of course be due to a non-standard way of accessing location data in the app.
  • There are many things that cannot be done in the Sailfish UI, but require enabling developer mode and accessing the command line. For me, and the other geeks, this is just fine, but I can't imagine everyone to think the same.
  •  The out-of-memory killer is rather trigger happy in the latest Sailfish version (1.1.1.27/Vaarainjärvi), which somewhat nullifies the benefits of multi-tasking.

Some other things I think are pretty remarkable:
  • There are many neat things that can be done via the developer mode CLI, like making certain apps start up automatically, customising virtual keyboard layout etc.
  • The multitasking really works. A non-geek example of this is that I can have my favourite music playing from Youtube in the background while doing something else on the phone - and even being able to see the live thumbnail of the video on the home screen!
  • User interfaces are always subjective to judge, but coming from Android / Samsung TouchWiz I think the UI of Sailfish works at least as well, and it didn't take too long to learn it, in fact I noticed this week I was trying to swipe an app to background on Android :)
  • There's pretty good support for also running Android apps if there are no native apps for a purpose. One could possibly even install the whole Google Play stack on Sailfish. Now, I dare you to try the same in any other ecosystem!

One thing to note here is also that unlike the big ones, Jolla does listen to their community and work together to make the product more pleasing. Working communication with a supplier techies (and a good community overall) is something I've learnt to appreciate.

All in all, I think the Jolla phone is a good device, just as long one doesn't have too high expectations to get a sleek and thoroughly finalised device. The potential - in my opinion - is huge, so stay tuned...

Update: (10.1.2014 9:30 GMT+2) added clarifications on the Android support.
submit to reddit Delicious

Monday, October 27, 2014

Things that you might think don't influence the buying decision - yet they do

Recently I've been reviewing APM products for the Java-based system I run for my pay. After some research online I ended up with two choices; one based on the recorded demos I'd seen earlier, and another one just since it was very easy to get for a trial period. If there are two choices, and to be able to see what one of them really does I'd first have to expose all kinds of details about my company and my position while the other is freely downloadable for a 30 day trial by just entering name, email address and company name, I'll choose the latter.

The one with an easy self-service trial period (let's call it product A from now on) was a breeze to install, both the server and the agents, and the GUI looked very sleek and intuitive - it took me roughly an hour to get as far as looking at some analysis of slow transactions from a test server. They clearly had made an effort to polish the product and make it such that one doesn't need to be über geek to be able to install and use it. I am rather geek myself, but I still appreciate the effort.

The supplier of the other one (product B), of which I'd seen some really nice demos, had a peculiar requirement: I had to schedule for a live demo before getting the download. Had I not already made my mind that I wanted to have this product for a trial run, I might have bailed out (as the demo required much more of my time and I'd already seen much of the stuff they could demo for me). However, I made the personal sacrifice of attending to a session outside of the office hours (due to the awkward eight hour time difference). During the demo it become clear that the product actually didn't officially support the version of the monitoring software that we had and to which product B was to be integrated to, but they had a beta to which they immediately added the support and asked me if I'd like to wait for releasing it or have it immediately as a beta. Original estimate for the release was "a few weeks", but it eventually was changed to "next week". I waited, happy for such a swift response for the need.

As the promised time came, I received a download link. I grabbed the installation packages and went on with the instructions. However, I got stalled for roughly an hour in trying to get the agent installed successfully, as the only automated installation was for the case when the application server is started with a batch file and there was only some notes about setting things up when it is installed as a service. I finally figured out the parameter I needed to add to the service wrapper to get it going, and so finally got everything installed. Then I went for the GUI to get some readings on how the system is running, just as I did with product A. This time that was not a breeze, either, despite of the demo I'd seen just a week earlier. I had to look things up in the manual to navigate in the GUI and get what I wanted.

Needless to say, at this point I had rather strong preference for the product A for its ease of set up and use.

I emailed my experience with the product B (with some very honest critique, pointing also out the contrast with the product of their competitor) to the account manager I had been in contact with. I was ok with having the technical issues forwarded to their support team and after two days I received email from the product manager who asked for some clarifications, mentioned that some of the issues I noted were known and gave some workarounds, and promised to get back with the rest of the issues after they had looked into them more closely. The next day I noticed they had actually opened support cases for many of my issues as I got a more detailed status report and some more assistance.

One and a half weeks later, after catching up with some other things in the meantime, I emailed some further info and questions on some of the issues, for which I received again very prompt and helpful response. The next day - directly after installing the agent in production - I had yet another issue (due to having messed up some config while trying to work around on my own an issue with the agent installer), and next morning I had enough info to remedy the situation.

At this point my preference had changed again - despite the product B being hard to install (well, I'd already installed it on every server I needed) and having some bugs and limitations (that were now known to me) and inferior GUI and user experience overall.

Why?

I had seen that they had a support team and attitude that is not that common in the industry. I knew, that if I ever encounter an issue with their product, I'll have a solution in my inbox promptly. Besides, having a GUI with a steep learning curve is not an issue as I've got used to things that way (that's what I do here, figure out things) and I prefer a "professional" UI over an easy and simple one that can get restricting when you want something that is not a very common use case. Last but not least, the off-the-shelf license cost of the product B in our case was lower than the price of the product A even after applying all the possible discounts their sales person could come up with.

So, even if your product is polished and fancy, you might still get beaten by someone who has the right attitude (not to forget a more realistic price tag). Also, if you don't offer an easy-to-start trial period, you might also get ignored, unless you've already earlier succeeded in making a very good impression.
submit to reddit Delicious

Saturday, June 21, 2014

Sometimes it is better to have a fresh start than to work on what has been around for ages

I finally got to upgrade my ancient desktop at home. The box was originally built around 7-8 years ago, with some minor upgrades along the road, and now it had Athlon XP 2500+ with 1.5 GB RAM. Seriously, it still could run light-weight Linux desktop pretty fine, with even two user sessions. The merit goes much to the U160 SCSI server grade disk system that only got outdated in the rather recent years by the SATA architecture. The origins of that box go still further into the history, as it is basically the direct offspring of my first Linux box from the days of RH 4.2 (and I mean RH before the days of RHEL or even Centos, literally in the last century).

I'd rather build new systems alongside of the current ones, so that there's time to slowly work on the new box to get everything going fine and dandy, but this was to be a "old junk out, new junk in" style of operation. Regarding the HW part of it it was pretty simple, only the main board with CPU and memory needed to be replaced and the SSD installed, since PSU was fairly recent and had enough wattage for the new setup, mostly since I'm doing fine with the integrated GPU in the new i7, and even the CPU was low-power model. Just plug back in the SCSI adapter and the 2nd NIC and power back on. It was a no-brainer to get the system to boot back to Debian 7.5, and everything was running so fast and smoothly already - but it was running all 32-bit.

Since I have experience of both doing fresh installs and major upgrades, I've noticed that I like more the upgrade path since there's no need to rebuild configuration piece-by-piece afterwards, but rather get everything in place in a big but process-wise simple effort. First, I intended to upgrade to 64-bit kernel with SMP support to get all CPUs and full RAM into use. I've not done cross-compiling earlier and after some futile efforts to get the Debian kernel build tools to produce a working 64-bit kernel (I might have succeeded without using .deb build but I've learnt to like the concept too much to diverge) I went with a stock kernel from the repository. Great, it supported all the important hardware out-of-the-box (and yay, it was the first time I'm running stock kernel in many years). But what about the rest of the software? A 64-bit kernel can run it just fine with IA32 emulation, but isn't it a bit dumb to only be able to get some of the performance of the shiny HW... That's what I thought, and went googling about converting the system from 32-bit to 64-bit.

That's where the crux of the story comes in.

There are plenty of pages giving some instructions and also some warnings (e.g. this askubuntu.com Q&A which says 'it is very complicated' and is absolutely correct about it). Debian wiki has a great article on architecture migration which I decided to follow. I got pretty far with it (that's why I called it a great article), but eventually I couldn't get all packages to reinstall, and aptitude was acting pretty confused (initially it didn't list any packages under Installed packages even though it did admit that I had packages in installed state, and even after I told it to rethink it (with forget new packages + update) it still was in denial about the overall state of affairs. What was more worrying, there were quite a lot of errors about libs being of wrong architecture. I got some individual packages fixed using the same manual procedure as for resolving conflicts during the mass-upgrade/migration, but eventually I felt that this is not going to end up with a wholly working system. Looking at it now, I did some mistakes in the process which at least made things harder, so I think it still should be possible to follow the article successfully, but definitely it is not an easy path and requires knowledge on resolving package conflicts (doing a couple of dist-upgrades when the Debian team releases new stable release is a good prerequisite). After all apt is a terrific package management system, IMO, and can do all kinds of stunts in the hands of an expert.

So, after wasting many hours down the upgrade path I gave up, downloaded an install ISO and started over. After approximately the same amount of work I spent on the upgrade attempt I now have pretty much all major things in place and working and after all, now I know that if something is off, I just need to compare things with the old state of affairs that can be found on the old disks and migrate the changes. It is also a good chance to re-learn some things like setting up DAV on Apache for use with Subversion.

What is to be learnt from all this? At least for the kinds of me who prefer building on the old, it is good to learn to consider the benefits of a fresh start, and the downsides of carrying the load of the past over the new platform. What might help with this is having some sort of a configuration management system that could be used to restore at least part of the customisation that has to be re-done. Having that at home might still be an over-kill, though...
submit to reddit Delicious

Sunday, March 23, 2014

Mining not for average Joe (for long at least)

I'll put an end to my adventures in cryptocoin mining. The most obvious reason is that my current hardware is not really too good for it (and as the summer is coming there would be heating issues), but now that there are ASICs coming in also for Scrypt coins, the difficulty will go on the rise making it even worse for people like me. There are even cloud mining rigs that one could rent (I'm waiting for someone soon publish an article that states "nn% of all computing power used for cryptocurrencies")...

I've seen it claimed that Satoshi Nakamoto had intended mining to be something that also average folks could profit from, but given the fierce competition to build ever more powerful mining rigs, that is not happening, and likely it will never be happening at large as long as mining doesn't include some sort of a human component (without it making mining a full-time job for the person), or unless new algorithms are introduced often enough to keep the slow ASIC development at bay (but even then people who are both able and willing to buy $1000 USD worth gear every two years will be gaining the most).

Despite my decision, I'll be keeping my eye on the subject, after all I do have some fractions of LTC and some DOGE.

Edit (some 3 hours later):  Yet it is not so simple. CPU mined coins are vulnerable to botnets, and on the other hand ASICs are terrific on hashing vs. power efficiency (i.e. green mining). And yet, in the lack of clean, renewable and cheap source of electricity, it's not a good idea globally if people started buying rigs that draw 1kW constantly...
submit to reddit Delicious

Sunday, March 16, 2014

Things to consider for profitable cryptocurrency mining

There is a looong discussion on Reddit on whether Dogecoin mining is profitable or not. I do not claim I'd had the stamina to read all the way through it, but a theme seems to get repeated ("yes it is" - "no it isn't").

The opinions also vary widely on the usefulness of so called altcoins (meaning anything other than Bitcoin). Surely any coin (I think I'll from now on use "coin" instead of the more tedious "cryptocurrency") that is not accepted widely as payment isn't really useful as a token of exchange, i.e. payment, but both for miners and traders they might prove useful. However, as these coins come and go it's essential to assess if the value of a given coin is expected to stay or increase in the future to prevent losses.

Trading aside (since that's not my cup of tea), once one has established adequate level of trust on a given coin, there are things to consider (after considering the efficiency of your mining hardware):
  • Do you want to take the exchange risk involved in holding on a coin for more than a day? If you do, do you see a given coin increasing in exchange value in the near future?
  • If you're going for low exchange risk and will immediately sell what you mine, does the lower risk factor counter the daily transaction and exchange fees? Also, what coin is the most profitable today regarding difficulty, reward, network hash rate and the target coin/currency?
  • Even if you're willing to take the exchange risk, it's worth checking the profitability based on difficulty, reward and network hash rate.
There are a number of mining profitability calculators around the Net. There are CoinWarz, Dustcoin, CrabCoins and whatnot (please don't get offended if your favourite one is not listed, those are just random ones I ended up to). I got interested in how those calculators actually estimate (as for that it is, estimation, since random events are in play) the profitability - and I think everyone who's using the calculators should be interested as well. CoinWarz does the calculation on the server so I couldn't check their code, but both Dustcoin and CrabCoins reveal their formula in the page source. Both also use pretty much the same formula:

 time [s] x hashrate [H/s] x reward
------------------------------------
         difficulty x A

where A = 0x100010001h at Dustcoin and A = 2^32 at CrabCoins. Since those values are close to being the same, the sites give almost exactly the same results.

At this point I started looking up more calculators. CoinSelect, Where to Mine and Criptovalute seemed to give the same results so I guess they use the same formula, too. And hey, the formula does make sense: The longer the time, the greater your hash rate or the greater the reward, the greater will be the profit - and the greater the difficulty, the smaller will be the profit. What bugs me is that coefficient A as I don't know where it is derived. The actual code at Dustcoin uses two other constants in place of it, but they are likewise as cryptic to me. I'd be glad if someone pointed me to an explanation for the coefficient.

One thing to note about the calculators: Always check if they are using the correct up-to-date data. Difficulty might be off from the current one, as well as exchange rates, and even the reward (but that would mean their data is really stale). It makes sense to check at least two sources that you trust to eliminate the risk of deciding based on incorrect data.

It is very likely that an extremely favourable mining situation will not go on for long, as also other miners will come to mine thus raising the hash rate which in turn will make difficulty rise. So, there will be constant ebb and flow which also means one should automate pool/coin switching based on estimated profit to continuously adapt to the changing situation. There is already software for that, quick googling brought up CryptoSwitcher and I remember having seen others as well. E.g. cgminer has API that allows centralised remote controlling of miners and when you add automatic decision making based on network data, your miners should be always after the largest profits, or at least staying away from the least profitable coins.

It should be quite easy to implement home-brewn mining automation since e.g. CoinWarz offers an API that one could use to directly access their profitability data. The free version allows 25 calls in 24 hours, meaning the situation could be checked once an hour, which should be quite enough when you're not doing this too seriously. The lack of real-time network data can be compensated by steering away from the most volatile coins. There will still be the risk of sudden large exchange rate changes, but those should be rare enough to keep the risk relatively small.

Finally, for those who want it to be extremely easy, there are mining pools that automatically do the switching for you. If you trust their algorithm does a good work, one of those is the easiest way to get the benefits of coin switching.
submit to reddit Delicious

Sunday, February 2, 2014

Adding sharing buttons for Pinterest, Reddit, Delicious, Stumbleupon and LinkedIn to a Blogger blog

My better half wanted to add a Pin It button to her blog and needed my help, so I did some googling around it to find out how to do it. As a result I figured I could also add support for some other sharing sites in my own blog as well. It was not as straight-forward as many blog posts and support docs on the subject do claim, so I'll tell here what I did in order to get them all nicely lined up. I hope this makes life easier for somebody else.

This was the resulting row of buttons,
in case I end up changing the layout later on...


Pinterest

There is a rather good blog post at bloggercentral.com on adding Pin It button to Blogger, and it also tells the basics about the blog template.

However, I had very little luck with the embedded template editor (after trying two browsers including Chromium, saving the template after changes didn't work), and found out it is way better to take the XML backup of the template and use a good text editor to modify that. Remember to save your changes on a different name! On Linux e.g. kedit and gedit work fine, but on Windows it was tougher since Notepad doesn't understand Unix style line breaks, and Wordpad isn't really a text editor so it is not guaranteed to preserve formatting. I downloaded Notepad++ Portable for the task (since it does not require system-wide installation), but any decent text editor that supports UTF-8 encoding and Unix style line breaks should do.

The other thing that didn't go as the instructions said was the placement of the button code - in both of the blogs it was required to add the button at the second occurrence of <data:blog.post/>. I should study more of the template structure to figure out in what cases the other occurrences are being used, but anyway I made the addition to all of them.

Reddit

The support doc on reddit.com shows many options for the button, all with sample HTML code, but there is a catch in using them with Blogger. All the samples would more or less work on a single blog post, but they wouldn't work on the blog home page which has many posts. The advanced options down the page show a way to go with some of the buttons (those that use a script tag), but in all cases the page reference needs to be modified to suit Blogger. Below is what I use.
<a expr:href='&quot;http://www.reddit.com/submit?url=&quot; + data:post.url'>
 <img src='http://www.reddit.com/static/spreddit7.gif' alt='submit to reddit' border='0'/> </a>

Noteworthy things:
  1. Usage of expr prefix on href attribute tells Blogger that it needs to interpret the attribute value which contains references to layout data tags (e.g. data:post.url).
  2. Also quoting needs some tuning, since the whole expression to be interpreted requires quotes around it, and the static text part in it also needs to be enclosed in quotes, thus the two occurrences of &quot;.
If you'd be using those buttons that have just a script tag in the example, just look at the example given under Interactive button advanced settings and change the values of reddit_url and reddit_title to point to data:post.url and data:post.title, respectively. However, reddit seems to figure out the title from the URL if title is not given, which is nice since I didn't find a way to make Blogger understand multiple URL parameters (although it should be possible and I do know how to format a GET request with multiple parameters).

Delicious

Also delicious.com shows a working sample of their button, but as mentioned just above with reddit, it didn't work that well with multiple parameters on the URL in Blogger template. However, it seems to be enough to put just the url parameter there. Also in this case I moved the request URL to href attribute even though it is not as neat as hiding it in the onClick handler. So, here's what I use:
<a expr:href='&quot;http://del.icio.us/post?url=&quot; + data:post.url' target='_blank'>
  <img border='0' alt='Delicious' title='Del.icio.us' src='https://delicious.com/img/logo.png' height='16' width='16' />
</a>

Stumbleupon

The badge creator at stumbleupon.com looks rather fancy, but at that point I had grown a bit tired of all fancy things which are hard to put into the template, and so I took the easy route of peeking at a page with a working badge and extracting the URL and the icon from there. Not quite as recommended, but it seems to work, too:
<a class='logo' target='_blank' expr:href='&quot;http://www.stumbleupon.com/submit?url=&quot; + data:post.url'>
  <img border='0' alt='Stumbleupon' src='http://cdn.stumble-upon.com/i/badges/badgeLogo18x18.png?v5' height='18' width='18' />
</a>

LinkedIn

There is a share plugin generator on developer.linkedin.com which gives the necessary code for the share button, and the data-url attribute with added expr prefix gets its value from data:blog.url just like in all the above.

The final touch

All of the above would work just fine alone, but putting them together took some additional effort for the result to look nicely aligned. From the Pinterest sample code I took the enclosing div, and put all the rest within that, too. However, the icons ended up aligned pretty bad, and so I added vertical alignment.

That solved all but Pin It  and inShare buttons, which have enforced styles from the accompanying Javascript code. For Pinterest it is possible to just ditch the Javascript and go with plain link+icon, but LinkedIn has made it more complex, so I ended up adjusting the styling of the enclosing div and adding some spacers to add some space around the buttons.

So here is my template addition as a whole:
<style type='text/css'> 
  #sharing-wrapper {margin:10px 0 0 0; text-align:left; vertical-align:baseline !important; padding:0px !important;}
  #sharing-wrapper img {padding: 0px !important;}
  #sharing-wrapper .spacer {padding-left: 8px;}
</style> 

<div id='sharing-wrapper'>

<!-- pinterest start -->
  <a data-pin-config='none' data-pin-do='buttonPin' expr:href='&quot;http://pinterest.com/pin/create/button/?url=&quot; + data:post.url'>
    <img src='//assets.pinterest.com/images/pidgets/pin_it_button.png'/>
  </a>
  <span style='margin-left:-44px;'>
    <a data-pin-config='none' data-pin-do='buttonBookmark' href='//pinterest.com/pin/create/button/' style='outline:none;border:none;'/>
  </span>
  <script src='http://assets.pinterest.com/js/pinit.js' type='text/javascript'/> 
<!-- pinterest end -->

<span class='spacer'/>

<!-- reddit.com start -->
<a expr:href='&quot;http://www.reddit.com/submit?url=&quot; + data:post.url'>
  <img src='http://www.reddit.com/static/spreddit7.gif' alt='submit to reddit' border='0'/> </a>
<!-- reddit.com end -->

<span class='spacer'/>

<!-- del.icio.us start -->
<a expr:href='&quot;http://del.icio.us/post?url=&quot; + data:post.url' target='_blank'>
  <img border='0' alt='Delicious' title='Del.icio.us' src='https://delicious.com/img/logo.png' height='16' width='16' />
</a>
<!-- del.icio.us end -->

<span class='spacer'/>

<!-- stumbleupon start -->
<a class='logo' target='_blank' expr:href='&quot;http://www.stumbleupon.com/submit?url=&quot; + data:post.url'>
  <img border='0' alt='Stumbleupon' src='http://cdn.stumble-upon.com/i/badges/badgeLogo18x18.png?v5' height='18' width='18' />
</a>
<!-- stumbleupon end -->

<span class='spacer'/> 
<!-- linkedin start -->
<script src='//platform.linkedin.com/in.js' type='text/javascript'></script>
<script type='IN/Share' expr:data-url='data:post.url'></script>
<!-- linkedin end -->

</div>
I put that right after <data:blog.post/> tag so that it appears after the post body text. It would be even nicer to have it on the same row as the built-in share buttons, but that would really require figuring out the templating more deeply than I am willing to do right now.

In addition, put the following right before </body>, since repeating it for every post messes up the positioning of Pin It button on blog home page:
<script src='http://assets.pinterest.com/js/pinit.js' type='text/javascript'/>

(oh, and since this blog is visually not that fancy, I think I'll drop Pinterest out, I doubt anyone would pin from this anyway...)

Addendum (5.2.2014): Digg and Tumblr

Later I also added Digg and Tumbr sharing.

Digg was simple, although the current incarnation doesn't seem to provide any official share button or widget.Thus I just made a link to their submit URL and used their favicon for the icon.

Tumblr was a bit harder, since the official JavaScript version only works on single post pages (not on the blog home page) and when using a simple GET request URL, the shared URL needs to be encoded - which Blogger template API can't handle (also they don't seem to fetch the page title automatically so that, too, must be encoded and included in the link). So I made my own inline JS to create the link the way I want it to be.

Here are the additions to the above, placed just before ending the </div>:
<span class='spacer'/>

<!-- digg start -->
<a class='logo' expr:href='&quot;http://digg.com/submit?url=&quot; + data:post.url' target='_blank'>
  <img alt='Digg' border='0' height='18' src='http://digg.com/static/images/digg_favicon.png' width='18'/>
</a>
<!-- digg end -->

<span class='spacer'/>

<!-- tumblr start -->
<script type='text/javascript'>
  var strPostUrl = "<data:post.url/>";
  var strPostTitle = "<data:post.title/>";
  document.write("&lt;a href='http://www.tumblr.com/share/link?url="
    +encodeURIComponent(strPostUrl)+"&amp;name="+encodeURIComponent(strPostTitle)
    +"' target='_blank' title='Share on Tumblr'&gt;&lt;img src='http://platform.tumblr.com/v1/share_3.png' width='129' height='20'/&gt;&lt;/a&gt;");
</script>
<!-- tumblr end -->
submit to reddit Delicious

Monday, January 20, 2014

Green mining?

Green server room was a hot topic a couple of years ago. The cost of energy plays a large part in the total operating cost of any computer system today, and this might indeed have been the largest incentive for the industry to go green (in addition of the PR value). The energy requirements are also worth noticing with private computer use.

In the previous post I talked about cryptocurrencies.  Mining is the most reliable way of gaining wealth with them, trading at exchange being more risky (yet also having greater potential profits). Mining is computationally very intensive, though, and thus it is also energy intensive (quite like traditional mining industry), and there are two main points about energy - the economical cost and the ecological cost. The economical cost is largely due to cost of energy, and ecological cost comes from production of the energy. The economical efficiency of a cryptocurrency miner is dependent - in addition to the price of energy - on the kilohash/s per kilowatt ratio of the mining equipment. The specialised mining rigs build for the single purpose of mining Bitcoins have pretty good kH/s per kW ratio, and it is a constant race to enhance the efficiency even more. Still, I'd also guess that most operators try to get their electricity at the lowest possible price, which many times means power mills burning coal. Coal (or either oil) is not that good for the ecological efficiency, which I'm going to measure in kH/s per CO2 (metric) ton. CO2 is being used due to simplicity although it represents only a fraction of all ecological effects (e.g. enviromental effects of mining and refining of the fuel, other emissions from the production of energy, and building of the facility are somewhat harder to take into account).

My initial idea with the small project that was described in the previous post was to take advantage of my work laptop while I'm not working, more precisely to borrow the hardware in a way no different to if I was surfing in Youtube over the weekend or trading stocks after office hours, using my own Internet connection and the electricity I pay myself (thus, I feel I'm not being taking advantage of my employer). I chose laptop not for the computing power (which is not that great) but for the energy efficiency of mobile equipment. The max output of the charger of the laptop is 65 W - a mere desktop processor of roughly the same specs can have max TDP of 95 W - not to mention the consumption of the rest of the hardware.


Calculating efficiency ratios

Let's toss some numbers around:

The laptop topped 14.1 khash/s - although that might not be the best that can be got out of the hardware due to only utilising the CPU - with measured power draw of 47 W, which means kH/s per kW ratio of 300. With 3 threads the hash rate decreased to 11.9 kH/s with power draw of 43 W, which actually means the efficiency ratio dropped, too, to 277. With only 2 threads (one per physical core) the rate came down to 9.7 khash/s with the power draw of 40 W, which means efficiency ratio of 243. A good example why it is worth measuring and calculating these things, since at least to me the result was not evident from the mere numbers. I would have also expected the power consumption to come down a bit more. No wifi or bluetooth was enabled, but I haven't touched the PM settings either - probably there might be some things to tweak.

There are not many (if any) LTC mining rigs (at least such that would compare to the BTC rigs) on the market due to the nature of the LTC hashing algorithm. However, I found a tutorial for building an LTC mining rig which quotes some specs to assess the efficiency. The rig ought to reach 1940 kH/s with power draw of 720 W when ideally tuned, which makes 2694 kH/s per kW - much better than the laptop.

LTC and BTC mining rates are not directly comparable, and since also the exchange rates are different and vary all the time, it is quite impossible to do comparison with the BTC rigs. However, just out of curiosity, let's peek at the first ASIC powered BTC mining rig I found with specs on the power consumption. It also happens to be the best you can get from the market at the moment. The specs say it can do 2 Thash/s at nominal power of 1650 W, so the efficiency ratio here is 1,212,121. That is mighty lot of hashing power (even though the figure is not comparable with the ones above). No wonder there is a market for such an ultra-expensive gear that can do exactly one thing and due to the continuous rise of the mining difficulty will get obsolete in less than six months. With that kind of computing gear it starts to become necessary to have proper A/C in place even in the most arctic climate (since computer gear don't like to get too cool, either), and the average figure for server rooms is that it takes at least as much power to get the heat out than to run hardware that produced the heat, which would practically halve the mining-power efficiency ratio. Having just one of those should be quite enough for keeping part of a small house warm during the cold season.


The cost of electricity and return of investment

The total price of electricity where I live during the last year was on average 0.125 EUR/kWh (based on the pricing for a small single house with estimated consumption of 18 MWh/year). Running the 2 TH/s BTC rig all around the year (14.5 MWh) would cost roughly 1800 EUR. With the current obscenely high exchange rate of BTC it would take 2.92 BTC to cover the electricity cost (and 7.15 BTC to pay back the rig of 5999 USD list price).

With the same electricity price, the home-built LTC rig would drain 6.3 MWh in a year, costing roughly 788 EUR (currently equivalent of 45.3 LTC), and to pay back the price of the parts (1356 USD) would take additional 57.6 LTC.

According to a Bitcoin calculator, the above mentioned BTC rig would have paid itself back on the third week, but assuming mining difficulty increase of 30%, the electricity price as above and the current exchange rates and other constants as they are now, the profit would have dropped to mere 100 EUR/week in just 190 days. At that point the return of investment would, however, be over 10,000 EUR, so it wouldn't be a bad investment as such. Adding some delivery and maintenance fees, the profit would likely be around 8000 EUR. It is worth noting, that the calculator listed some recent rigs that will never break even (assuming the current situation and forecast), let alone gain any profit.

In contrast to that, it would take 200+ days for the LTC rig to pay itself back, even if assuming the recent stability of LTC mining difficulty. It would take roughly 560 days to get about the same sized profit relative to the price paid as above (and here I didn't even throw in any maintenance fees yet). As it is unlikely that difficulty will keep at the current level that long and also electricity price is likely to go up, it might take a year just to break even. So, either BTC value is vastly bloated to make BTC mining so profitable, or specialised HW just beats commodity gear in efficiency.


The carbon dioxide foot-print

From the ecological side, generating 14.5 MWh of electricity would produce something like 13.9 tons of CO2 if generated by coal-fired thermal power, 10.9 tons of CO2 (oil-fired thermal power) or 7.4 tons of CO2 (natural gas combined). Also other sources of energy cause some CO2 emissions indirectly (building and operation of facilities etc), from 0.5 tons (solar power) to less than 0.2 tons (water power).
(Sources: Hitachi, U.S. Energy Information Agency. Values shown here calculated from averages between the sources)

Even though there are skeptics who claim climate change due to human actions is a myth, there is a strong scientific evidence that CO2 emissions alter the climate. There's also no question about the adverse ecological effects of all sorts of large scale industrial actions, or the fact that fossil fuels are going to end some day (or more precisely, it will become too expensive to extract them from the ground). So, in the ever-continuing absence of fusion power plants, I think it is well worth thinking where we are using the energy that is produced, and what kinds of production methods should be preferred.


Summary

After such a rant it is always hard to draw things to a closing, but I'll try...

Clearly it is best to have the equipment you're running match the requirements as closely as possible to be as efficient as possible. Home computers are bad at that since they usually try to offer a balanced but wide set of features. Servers are built with a different mind-set, they have a target task that needs to be filled with no extras.

Financially, the best you can probably do with a high-end gaming GPU (or an array of such) is to put it mining for cryptocurrency when you're not gaming. That will pay at least some of the next generation gear you'll likely want to buy at some point. If possible, aim for renewable energy sources for the electricity you buy, to get some of that greeniness into your personal IT, too. It can be argued that the heat dissipation of home electronics reduces the energy needed for heating, but that applies only to the cold season, having A/C to bust out the excess heat in the summer would just put the profits down the drain and be a waste of energy.

Also, if you have boxes that are on 24/7, it's worth trying a CPU miner with a low priority (in Windows) / high nice value (in Linux). That way you'd get something out of those otherwise idle hours, to compensate the base cost of having a box on (you have measured the power consumption of your computers when they idle, haven't you?).

However, one thing is sure: If you're going to make an investment, especially for financial purposes, be sure to calculate beforehand if the investment will pay itself back in a reasonable time.
(Uh, that might be the first financial statement I have ever made :D )
submit to reddit Delicious

Sunday, January 12, 2014

Employing random hardware for Litecoin mining

Cryptocurrencies are a prominent trend at the moment, with Bitcoin (BTC) leading the way. In fact, BTC seems to already have passed the limits of hype and boomed also financially to the extent that mining of BTC is hardly anymore profitable unless you have a state of the art mining rig (and there is a chance the rig won't pay for itself as the mining difficulty rises). For the ones who want to try the ever-lucrative miracle of getting bucks out of thin air (or more precicely, out of CPU/GPU cycles), there are fortunately many alternatives left. Litecoin (LTC) is maybe currently the most prominent alternative for BTC (at least measured by market capitalisation) that can be mined (source: coinmarkercap.com). Ripples is ahead of LTC in market cap but it works differently to the likes of BTC.

That's it for the financial part of this post. I'll give whatever I will get mined to charity, and with the hardware I have there's no risk of getting rich in this business. Anyway, I got interested of the possibility to easily harness the processing power of unused hardware along the lines of plug it in and leave it there to crunch numbers. The same idea could be employed to anything that uses massively distributed processing.

I selected a live Linux distro on a USB stick as the OS for the experiment. Linux, since it doesn't require much resources, is easy to tweak and free to use, and I'm more familiar with it than Windows. Grml Live Linux was the one I chose to use, mostly because of the small footprint (good fit for old 512MB sticks).

There were two things in this project that I hadn't done before: customising a live distro (to add miner software) and re-producing a bootable ISO image. The first part ended up being pretty trivial, even though Grml uses squashed file system so I had to unsquash it first and squash it up again after modifications. Since I won't need to retain any run-time changes in the filesystem between reboots squashfs is just fine for the purpose.


Customising of the live image

Mount the original Grml ISO and copy it somewhere for modifications:

> sudo mount /var/tmp/grml32-full_2013.09.iso -o loop /media/cdimage
> cp /media/cdimage /var/tmp

Unsquash the root filesystem, modify it and put it back together:

> mkdir /var/tmp/grml32-custom
> cd /var/tmp/grml32-custom
> unsquashfs /var/tmp/cdimage/live/grml32-full/grml32-full.squashfs 


[copy stuff and adjust to run on the target setup]

> cd ..
> mksquashfs grml32-custom/squashfs-root/ \
  /var/tmp/grml32-custom_2013.09.squashfs -b 262144
> cp grml32-custom.squashfs /var/tmp/cdimage/live/grml32-full/

I ended up using the same block size for the squashfs as the original (thus "-b 262144"), but likely that doesn't matter much other than reduced overhead for large files since the default block size mksquashfs uses is pretty small.

Finally, to save space, I deleted the original .squashfs file from live/grml32-full/ and modified live/grml32-full/filesystem.module to point to the customised version.


Re-building the ISO

This took me a while to solve, since there were all kinds of instructions on the net how to produce a bootable ISO, but none worked. With genisoimage I got as far as being able to boot with the image file using QEmu, but after copying the image on the USB stick it wasn't working anymore. The issue seemed to be that the image didn't contain proper partition table (checking with cfdisk showed no partitions). Of course, one could just have the contents of the root partition in the image and then install boot loader manually after installing the image, but I preferred to produce a neat and simple image that does everything the original one does, too.

I finally ended up using xorriso as instructed somewhere:

> xorriso -as genisoimage -r -o grml32-custom_2013.09.iso \
  -b boot/isolinux/isolinux.bin -c boot/isolinux/boot.cat \
  -no-emul-boot -boot-info-table -boot-load-size 4 \
  -V "GRML_custom" -iso-level 3 -partition_offset 16 \
  -isohybrid-mbr /usr/lib/syslinux/isohdpfx.bin /var/tmp/cdimage/

Finally, just I just cat'd the image on the USB drive.

This required me to install xorriso and syslinux and their dependencies from the Debian repository, but that was all extra that was needed.


The aftermath

I used cpuminer in this experiment because that is what I use on my boxes anyway (couldn't get cgminer to compile, and I doubt my low-end GPUs would provide much extra power anyway). I had built init.d script for minerd which makes it rather easy to construct a plug-in-and-mine solution.
The nice thing here was that Grml had all the required libraries already so I had just to copy over the executable, scripts and config files. I don't know how much it affects that I am now running an executable built for old AMD Athlon XP on a Intel Core i5, but unless there is good evidence that compiling for a newer target CPU would do any good, I don't bother. At least this way the executable should run on pretty much all hardware that is worth trying - the old Athlon XP here gets about 0.8 khash/sec which is pretty near at not worth trying...

Interestingly, running 4 miner threads on the Core i5 chip (which advertises 4 cores due to hyper-threading) gained about 14.1 khash/sec, but since that also meant having the laptop fan blow at top rpm constantly, I changed to running only 2 miner threads, still gaining about 9.7 khash/sec. This is a good example on the limitations of shared-something CPU architectures, but probaly also an example on how the turbo boost technology can compensate by reaping the headroom left in TDP due to only using part of the chip.

(edit: after measuring the actual power draw with both 4 and 2 mining threads it was obvious that the power vs. hashing efficiency is much better with 4 cores)
submit to reddit Delicious

Wednesday, November 27, 2013

Large and complex projects make a large and complex mess?

I have written already earlier about the commonalities between building houses (maybe it would be better to generalise it by saying construction projects) and computer systems. It so happens that Tekniikka&Talous writes about the results of a survey made by Independent Project Analysis (IPA), of which IPA's head of Europe and Middle-East Mary Ellen Yarossi talked in a seminar held by Bentley in London in the end of October. I couldn't find any international news coverage on the event, and the full survey results likely are not available freely, but I'll refer here some of the highlights published by T&T on it.

IPA had analysed almost 17 000 construction projects (world-wide) of which 500 were classified mega-projects. Of those 500, two out of three had failed when schedule or budget overflowed over 25% or there still were unresolved issues after two years of official project ending. The survey also concluded that large projects fail more often than small ones, and projects which use new technology, fail more often than the rest.

Now, why does that not sound surprising? And from projects with new technology it's pretty easy to draw an analogy to software projects. I mean, new technology is introduced pretty often on this trade (construction business, in contrast, is usually quite conservative, but the direction is toward ever more challenging environments and larger structures which require new technologies).

Curiously enough, the survey also concluded that a common problem was that the objectives set for the project are not clear to all parties, or the objectives are not understood the same way, or they are in conflict (analogous to common issues with software requirements). Also they noted that in long projects the slipping of schedule in one phase is thought to be possible to compensate by shortening later phases, although in reality that only introduces more problems (as far as I know, testing tends to often get this kind of treatment in non-agile SW projects, which was also brought up in the "Obamacare" case lately).

As the seminar was held by a software house it should be no surprise that there seem to be common factors with software business, but I find these pretty striking and central. Thus, it might be that it is possible to learn from the mistakes of the other trade what comes to these things - and it's not just (us) software folks who are not that good with large and/or complex projects (it is us humans who are miserable in sticking to good practices and proper processes).
submit to reddit Delicious

Tuesday, November 12, 2013

Value in simple tools: psloggedon

Scenario: You notice a stream of requests ending up in a production server error log that clearly hint you of a misconfigured software client on a laptop (running Windows) in the company network. Checking the IP address against DNS gets you the computer name, but checking that against the company CMDB you only get a name of an former employee. The errors in the log are ugly and you want to get rid of them, but as you don't have admin access for laptops, you can't use the regular Windows admin tools to figure out who's using the darn thing. Asking for a workstation admin to get you the information might be one way, but there's also another way...

Solution: Grab psloggedon from pstools package (and nevermind if your copy is six years old, as mine was). Issue
pslooggedon \\hostname.yourdomain.com
on the command line and there you'll see all the accounts that are currently logged on, and can contact the user in question.


I acknowledge there might be plenty of other ways, too (myself I tried also using msg to message the user but that seemed to be blocked somehow, or maybe it was a conflict with Win7 vs. WinXP), and you're welcome to share similar stories in the comments.
submit to reddit Delicious

Tuesday, October 29, 2013

Coding for the kids

There has been some discussion lately here and also abroad (see Oct 2013 Wired and the president of Estonia for BBC news) on whether kids should be taught computer programming already during the first years at elementary school, or even in kindergarten. I don't know about the latter, but definitely programming languages should be easier to grasp than any natural language due to the more limited syntax and vocabulary, so if a 7 year-old can figure out the basics of a foreign language, programming shouldn't be any harder.

Myself I was introduced to coding with Sinclair ZX81, and since my dad had bought only the computer and not any programs, the only option was to either type in a program from a magazine or book - or to write it yourself. I was about nine years old at that time, I think. Come on, writing BASIC is like writing English, which I was introduced to at school around the same time.

Of course not all kids (neither when they are young nor when they grow up) need to get deep into coding, but I have to agree with writer Elina Lappalainen and Ilkka Paananen of Supercell (both in Talouselämä 38/2013) on that knowing the basics of how computers are made to work is something that can be of incredible benefit later on as dealing with computers has become so everyday thing that it would take an effort to not to use any for even a single day. I've long been thinking that the fact that buyers in software projects don't actually know anything about what software really is (or how it is made) is one big reason for why software projects go as they tend to go (i.e. badly). If they had had a peek at how things operate within computers in school, things might have been different later on.

After all, I think kids are still today taught in Finnish elementary schools the basics of handicraft (both with wood and textiles) and cooking, not to mention sports and arts, why shouldn't we be teaching them also something about computers - it would be potentially just as important for their future.
submit to reddit Delicious

Consumer services going corporate

Lately I've been involved in implementing Google's services in an enterprise setting. It has been rather amusing at times, and rather irritating at other times. Somehow I have had the feeling, that the enterprise service model is still so new that it hasn't made it's mark on the services yet, or maybe it is Google's willful decision to be primarily a consumer service company.

(image by Bartosz Kaszubowski)
I base my claim on the fact that everything is always tied to a personal account (and even though you can use an account in multiple devices or for multiple users, the policies forbid that kind of usage), so we eventually need to sync our Android user base to Google's cloud and manage them there as individuals (although grouping is possible). You also need to adjust to certain things like using a (personal) credit card to pay for e.g. the developer console (after which the card is attached to the account from time to eternity) which to an European feels awkward compared to invoicing, and when communicating with a retailer they - obviously - insist on using Google Hangouts which just doesn't work if you don't have an Google+ account - so you end up either using your personal account or creating a new one for that purpose. In fact, I have currently five Google accounts including the corporate one, I don't know if that is much but it is a ridiculous amount of accounts to have as a user for using the services of a single company. Of course that is partly due to my explicit wish to separate some things, like not have my personal account tied to my work celly (or maybe an Android celly of any kind, to make it easier not to accidentally sync loads of stuff in there). At least now I can ditch the one I created specifically for work purposes, since the corporate account replaces it.

Also, Google does seem to be pushing their services pretty hard, even though I wonder how many enterprise customers (and I'm talking about medium or large companies here) are interested in having corporate Gmail account enabled for their employees (or Google's calendar, address book, instant messaging, document management & file sharing etc). It was some work to adjust the 70+ services so that they didn't seem awfully wrong...

That being said, I must admit that at least Google is trying hard. They have a wealth of support documentation on-line, which seems to be pretty correct and helpful at least if you're using English language for your services. When we finally got the Google Apps for Business up and running, there was such a huge amount of stuff in there to adjust that it made some kind of an impression to at least this systems nerd. The web based admin UI is a bit slowish at times, and it is also complex as hell but then again so is the service offering.

We also ended up transferring the developer console (that we had opened already earlier not knowing we shouldn't do it quite yet) to the matching enterprise account and while the process was incredibly difficult from the user point of view (had to consult both the support docs and the retailer to get through it) and while it required paying another $25 for opening the enterprise developer console, it also allowed requesting a refund on the previous payment and moving all data between the accounts (even though most of the services didn't seem to support the automatic transfer, including dev console). I'd bet that not all similar services offer that possibility.

Currently we have entered the pilot phase with Google's enterprise offering, and it will be interesting to see how things go. At least there are plenty of possibilities to be reaped for the per device license fee... And the users seem to be interested on getting Android devices as their everyday tools.
submit to reddit Delicious

Thursday, September 5, 2013

Old SQL, new SQL or no SQL at all, now that's the question

(image by luana1985)
It's about time to blog something about NoSQL since that is on top of it's hype cycle according to Gartner (as of July 2013, and actually key-value DBs are already sliding fast into the Through of Disillusionment already). And boy, does it show? Last summer I dived in the vast depths of Youtube searching for interesting talks and presentations around this subject and at least most of the younger lads presenting their favourite product (which of course is totally awesome) acted like that is the key to everything. There had even been a "Battle of the Backends" in Google I/O 2012. Ok, have to admit, that was quite entertaining.

As a sidenote, I kind of hope that the confusing name NoSQL would soon be forgotten as some of the products anyway support some sort of structured query language (if not even the standard SQL) and the alternative name used by some, NewSQL, is not much better since the "old" (R)DBMSs are starting to gain features from the "new" products. Besides, the basic ideas are not all that new anyway, for example Lotus Notes had a document DB already way back in the 90s. And what comes to in-memory DBs, even Oracle has TimesTen that can act either as a memcache or as a independent in-memory (R)DBMS. I don't think it is even that useful to bundle together things as different as key-value stores, column-family stores, documents stores and graph DBs (and what else there is). So it's probably better to talk about them separately without using the buzzwords.

Back on the track... Marketing speeches aside, it's not likely that the new approaches are the key to everything (the older lads probably had already seen the "coming" of object DBs and knew it better). I don't know who conducted the MySQL-SenseiDB comparison tests published on SenseiDB website but if your solution somehow looks like 100x faster than a widely used solution, while running a most simple test scenario, it is quite likely that either your test setup is flawed or you're outright messing up the results. Well, maybe there is a reason why their web site is also still in the year 2012 now in Sept 2013...

I think I introduced myself with Youtube and some textual resources to at least MongoDB, RavenDB, Cassandra and Hadoop, with mostly more than one longer presentations plus a couple of clips on the current DBMS scenery in general and hey, I admit, many of those were just terrific talks and I did get enthusiastic about the subject. After the upsides of the non-RDBMS approaches were starting to dawn on me I could quickly think of two example cases where an RDBMS was not quite giving adequate performance. Unfortunately the other of them is tied to proprietary software and likely will support a non-RDBMS in production somewhere in the next decade - if ever - even though e.g. a document oriented approach might fit it very nicely. With the second case I'm free to try out everything myself, and inspired by some of the DB design related presentations I already did some changes to the DB design on the current platform (MySQL) with which I was able to get 50% off from a particular query that took very long. However, that was just plain old denormalisation, nothing new in that, and the initial design might have been bad in practice, anyway, even though it was kind of a textbook case of a relational DB model.

What I intend to do is to try out similar designs on both MySQL and a document DB (probably MongoDB), since the data lends itself quite well for document-oriented design. This will mean having to stuff arrays of things in a single field in MySQL but as I'm most interested in the DB performance that is ok (and subsequent parsing in the application shouldn't be much of a problem, either). The thing I'm most curious about is how easy it is to design the DB so that it still allows querying from different angles. At the moment my expectation is that I probably need to duplicate the data in some parts to allow that, but then again that is also required in reporting DBs (think of cubes).

Stay tuned (but don't hold your breath since I can't promise the next post will arrive before your brain cells die of asphyxiation)...
submit to reddit Delicious

Wednesday, July 31, 2013

Polyglotism

Lately, I've been educating myself on the topic of what is commonly called NoSQL, and this nice presentation by Martin Fowler made me think about stuff I've already written about earlier. The same thing can also be found on a talk on data storage technologies employed in Craigslist.



The key phrase was polyglot persistence (@~51 mins in the video). Not that I wouldn't probably have heard it already earlier, but now the ultimate meaning finally dawned on me. It means, that to build a system - one that is highly performant, at least - you probably need to be able to take advantage of the different DBMSs that are around. The same way that multi-lingual developers push the idea that you need to use the language that is right for the job. So the diagram Fowler gave on polyglot persistence could be extended with programming languages with which the different parts of the system have been built with, and probably also with OSs the parts run on, and so on. There is a reason for dozens of DB brands and programming languages and whatnot existing, and it is the attempt to produce proper tools for specific problems. I can see this could also lead to a sprawl of diversifying computational environments, but I guess the constant popularity contest that goes on between products makes either the products converge or the niche options quietly fade away with the more popular ones gaining market share.

All this polyglotism further widens the options for a fresh wannabe ICT specialist. There certainly is a need for narrow deep expertise (to be able to know something like the Spring framework from top to bottom one really needs to spend considerable time with that alone), but I do not see that it would become very common for companies that are not actually huge in size and less than 100% technology oriented to hire bunch of specialists of different areas. Or they could, but then again they could hire them just as well as consultants since they are not likely to need each one of them equal amount of time constantly, and if the team members do not know much about each others areas of expertise they really can't be assigned to work on the same task together. My experience from the service sector is that there is a huge need for a team of people small enough that it can constantly be kept busy with the ongoing tasks and that you can throw at any kind of a problem and they can handle that based on their collective wide experience.

What I am after here is that even though I've been mostly using Oracle (from version 8 onward, I think) as the data back-end in my professional life,  the potential of interesting non-RDBMS encounters is highly rising. All along the road I've been running into cases where it is not feasible to stick to the good ol' 3NF of data and instead denormalising to get performance - something that the NoSQL folks are very fond of. This far it has been a convention that if an application needs a DB (bigger than can be reasonably embedded within the product on the same server), it supports one or more of the big traditional RDBMSs. Now I already see RHQ including Apache Cassandra in the package, and even though Cassandra comes bundled with the product, one basically needs to install it on a server of its own for practical use - so basically it can be said RHQ requires Cassandra in addition to one of the supported RDBMSs. Personally, this made me think of how I can sell the idea of requiring another server for application server monitoring to my superiors...
submit to reddit Delicious

Sunday, July 21, 2013

Your old sins will come to pay you a visit once more

Some time ago, I had to shut down both of my Linux boxes at home since I needed to cut off electricity from some rooms while installing a ceiling fan (remember working safety!). All was well with the fan after switching electricity back on but then I tried getting the older server box up. It said it didn't have any disks. Well, that is a sure way to pump up stress hormone levels, and it did work on me again. The first suspicion was that since the hardware is rather old and had not been off for a looong time maybe some electronics just had died. But no, it didn't see either of the attached disks. Since that box was serving also as DHCP, DNS and gateway for LAN, and it was getting late in the evening, I just hacked the needed services up on the desktop box which luckily had pretty much the same firewall config already. DHCP didn't still work, though, and since we were about to leave on a holiday trip I just left it that way (later on I discovered there was just a stupid mistake in the config but that's the way it always goes, right?). Professionally it felt bad to leave my lady's web site offline, but she didn't seem to mind the long SLA.

After returning from the trip I took another try on trying to get the issues sorted out. Trying out a live cd proved that the hardware was alive and well. I had already been suspecting something about udev earlier based on some googling and found some info on kernel options CONFIG_SYSVS_DEPRECATED and CONFIG_SYSVS_DEPRECATED_V2. Disabled those, recompiled the kernel and behold, all was well again! I have absolutely no recollection why those were enabled (probably for supporting some older utilities), they had been that way for years already. Seems like Debian had evolved their udev implementation so that it had become incompatible with those features somewhere after the last reboot (which on the server box was 1.5 years earlier as I found out from the uptime logger afterwards). Serves me right for keeping a feature that is clearly meant to be transitory enabled for so long...

Well, this week I noticed that I had no sound on the desktop. First I just thought it was due to Flash which was not working in Opera anyway, so I installed Gnash and its dependencies. Too bad there was still no sound. No sound from much anything, actually. I got a crash course on sorting out Pulseaudio issues, but the end result was that I only had a dummy sound device available and even though Alsa seemed to recognise the emu10k1 device, there was just no sound at all.
Then I got a flash of enlightnenment. Those damned kernel options... the kernels were identical excluding things related to differing HW on the boxes, so also the desktop had the same deprecated features enable. Needless to say, disabling them made sun shine on me again. They really do seem to mess up with udev in unpredictable ways.

Uh, I just need to start paying more attention on what these boxes have installed and configured and not let them deteriorate....
submit to reddit Delicious

Sunday, July 14, 2013

Hazards of sitting

Having been reading monthly occupational safety reports at work has increased my consciousness on safety related to work in general (very good for preventing some stupid home injuries) and electricity related risks (also very good thing to know if you are the kind of guy who doesn't fear to make some minor changes to home electricity, like installing a ceiling lamp that doesn't have the fancy and safe kind of a plug for connecting it). Other than that, I guess, for a guy who sits at the desk all day long the most likely accident at work is slipping on the office floor. And oh yes, I have once dropped a thick folder full of papers on my big toe, which resulted in a small bruise. Probably not reason enough to request for safety shoes with anti-slip soles and toe protection, though.

Those reports and subjective experience has also increased consciousness on the fact that long term sitting in front of a computer is a hazard to one's health. Not an acute one, though, and perhaps it is not that high profile on corporate health and safety agenda - except (hopefully) in software houses. Generally computer work is taken as such that it can be done in a condition where manual labour is not anymore possible (the most important muscle in a specialist is the brain, anyway). However, there is potential for many kinds of pain which might even be debilitating and thus affect productivity, so to me it looks a very real threat for working health.

I remember times when stiffness and tension caused by crouching over the keyboard was easily taken care of by some physical workout, stretching and a walk outdoors. By that time I didn't do yoga asanas, but I'm sure those would have worked, too.

I miss those times. Nowadays I'm happy if there is no notable tension and the stiffness is not painful. I've increased my awareness on the condition of my body as well as my posture, and I try to have short breaks during the day for doing some exercise and stretching at my desk. I do asana practice at least a couple of times a week, and have tried several kinds of treatments and therapies (different kinds of massage, acupuncture, reflexology...) and go to work by bike or foot. I ran into a good article on the subject and the theory given there on the reasons for the continued stiffness and discomfort seem very logical to me: the stiffness kind of works its way into the different tissues and thus in the end it's not just the muscles that behave badly.

So far I haven't used any occupational health care services since the situation is not outright bad, but I'm concerned of the directions of things for the sake of myself. I'm also pretty confident that the condition is indeed profession related (type of profession and condition do match).

The thing I'm happy about is that there is one common computer job related injury that I've managed to prevent so far - the carpal tunnel syndrome. I've been paying a lot of attention to hand ergonomy since pretty young which must have helped a lot. Maybe if I had been just as aware of the possible results of bad posture, too, things might be now different...

The reason I'm writing this is that during all these years I haven't met too many colleagues who would have indicated that they are really aware of these things. It's pretty common for office workers to go a gym regularly, and that is most likely a very good thing to do. Many might also have other hobbies that are good for the upper back area (I've heard that for example golf is great physical exercise in this sense). However, I fear that I'm not at all alone with this issue and it's worth the effort to bring up the subject.

Thanks for reading and now go do some stretching and make your body happy!
submit to reddit Delicious

Friday, June 7, 2013

Of house and systems building

It happened that one evening I was watching the episode of British TV series Grand Designs which was portraying a usually overly-optimistic and imaginative house building project in Exeter (you may be able to view the episode via youtube). The gotcha of the series is that all the projects are focused on truly grand designs, leading almost always to a doubled budget and severely stretched schedule. Now, if that doesn't sound enough like an average system development project, let me add that quite many of the building projects also have legacy components (all kinds of conversion and extension projects) and the plans are modified along the road to accommodate to the failing budget and/or schedule - or the changing vision of the project owner(s), and the project is not necessarily led by an professional of building industry or architecture.

This specific episode was somehow unique in the sense that this time the plans really were changed remarkably in key points multiple times as the project went on, and the host of the show looked really doubtful of the chances of the owners ever getting a decent house. As usual in the series, the end result indeed was a decent house, unique for sure but not in a weird way, and definitely not ugly to look at (I wonder if they just leave out all the miserable failures or if the house builders just are so darn lucky).

Software or system development projects have been compared to all kinds of more traditional projects in the history, and software industry also has tried to take influence from other industries (like lean that originates in car industry). The inherent differences between software and its comparison points that might make the metaphor lame are debatable, as cars and houses (and maybe even more so the huge projects like bridges, dams etc, not to mention nuclear power plants) really are quite complex and technical these days.

The text book recipe for success is to have a rigorous working process that is not abandoned when difficulties arise, to avoid touching legacy parts when possible and to have professionals that are knowledgeable on the task they've taken on all levels of the project organisation. Many times it just doesn't go like that, I think. Many times there will be a sort of a cowboy mentality, "agile" or perhaps better called anti-agile (as it does not have much anything to do with the definition the signees of Agile Manifest had in mind) way of working. Many times it is indeed the legacy parts that need to be touched, which is harder, more expensive and more error-prone. Many times the key persons might be professionals of some other trades than software (no-one really expects non-coders to code, but surely any person with a technical mindset can be an architect, and financial skills are enough to lead the project because then the project surely will be on budget, right?)

The success rate of software projects is not very flattering. I've not got any recent figures at hand, but Robert Kraut and Lynn Streeter referred in 1995 in their work "Coordination in Software Development" to Robert Blazer's statement dating as far back as 1975 (!) and said that software still is unreliable, delivered late, does not respond to changes, doesn't hit the performance target and is expensive. Well, has that changed since then? I cited that paper in my MSc thesis in 2007 and my conclusion is still the same: that does still happen in large numbers.

Looking at the news here, I'd say that the success rate for building projects is not very good, either, especially the ones that are not large and demanding enough to clearly require the best practices and the best people. Just like the resulting software is likely to have security issues, the resulting buildings (at least in this country) are likely to have issues with moisture and/or unhealthy athmosphere inside the building. Neither of those would be hard to avoid, though, either by thinking security already in the design phase (for software) or applying basic building physics in the design phase and carefully sticking to the plan through the building phase (in construction work). But like in the TV series, when asked about the expected end time, it can be found out that there is no detailed schedule, and the project owners have the habit of changing the design as they go along (each change perhaps leading to other changes), and the project lead might be technical, but not exactly from the right branch of science...

As for the couple that featured in the TV show, I have absolutely nothing to say to how they dealt with the house they were building for themselves - it is solely in each one's own decision and risk. After all, they were amateurs when they started the project (as much as I am what comes to building a complete house), and should thus be allowed to learn as they go. However, the same mentality and way of working would be very strongly objectible if it occurs in the context of professional work.

So, what all this babble boils down to, I assume, is that some software projects are either led or completely run by a bunch of amateurs, or a bunch of people acting like amateurs, which is sad and unfortunate (both for the industry itself and the users of the software). I'm not into poking fingers at people, and I must confess that I've done plenty of amateurish mistakes so far and likely will discover new amateurishness in my actions in the future. Live and learn, or should I say fail and learn...

Since my sources are a bit dated already, I did some quick googling on the current situation:
Why Projects Fail - Facts and Figures
The Failed Record of the Software Industry
2010 IT Project Success Rates

It would seem that there, indeed, has been some improvement, and the agile movement really seems to be able to deliver what they've promised. However, the papers referred in the above sources still contain the same complaint and worry: that it is the people who make the projects fail. People, who do not act in a professional way.

From a certain angle computers creating their own software sounds like a good idea, even if it is quite sci-fi sort of an idea...


Post addendum

Some more survey results:
http://thisiswhatgoodlookslike.com/2012/06/10/gartner-survey-shows-why-projects-fail/
http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php


And then yet later on (13th Nov 2013):
A good example of a gigantic failure in a gigantic project is the US healthcare.gov project that has been under scrutiny lately.
submit to reddit Delicious

Sunday, April 14, 2013

All work makes Jack a dull boy?

Sometimes I've been pretty frustrated over the fact that I need to worry about my employer's business also outside the office hours. That's an unavoidable side-effect of administering a production system, I guess, and it does not diminish at all by lack of quality within the administered system. I mean, I do like my job, but I also do like the rest of the stuff I like to do when I'm off-duty, and most of all I do like to sleep at nights. Unfortunately the system has its best time window for service from 10 pm to 4 am, which completely overlaps my best time window for sleeping. And I do have trouble getting sleep after late night work.

Whining aside - since that is the point in this post, at least so far - I ended up seeing a documentary about the poor people of India called Nero's Guests (this was actually already last summer, but never mind, and I think everyone should educate themselves with that kind of documentaries but that, too, is not the point here). One of the people portrayed there was a single mom of I think 3 young kids who worked a long way from home. She woke up around 4 am, washed up, cooked breakfast for the family and went to the train station at 6 am. After the day of low-paying hard work she returned home on the train after sunset around 23 pm (her kids obviously had had to take care of their own dinner since they were already asleep by the time their mom got home) and went to bed. I don't remember the details anymore, but I'd guess that went on seven days a week (or maybe she was lucky and was able to have one day off from work per week). Compared to that, the discomfort of having to work 37.5 hours a week with occasional hours hitting outside of the oh-so-convenient 9 am to 5 pm window is minuscule.

That is not by far the only example. Like the farmers of yesterday, the poor of the world today do not have a separation of work time and free time. And I think farmers still have the same schedule that nature dictates (work when needed, rest when you can). Not exactly the most prominent role models of today, but what is it that they seem to actually cope better with their circumstances than me with my own?

Actually the habit of having to be at the office from 9 am to 5 pm was inherited from the days of industrialisation in the 19th century, and has not much to do with the current ICT work that is not tied to place or time. In a factory it was - and to some extent still - is crucial to have the people at the conveyor belt (or the monitoring console in a more modern factory) all the time, otherwise the production will likely suffer. Since that is not the case with modern information work, we have things like flextime (ok, that's rather old thing already), remote work and flex work. Basically the information workers of now can put in their effort at the time they feel most comfortable and sometimes even in the amount they feel comfortable with at that particular time. Working in cafés, park benches, sofas etc is kind of trendy these days, and even the less-trendy companies start allowing that kind of work.

I've been doing flexitime all my career, and I appreciate the freedom it offers. I also appreciate that my current employer does allow occasional remote work days when I need it (although I do have better ergonomy at the office, so working a full day at home is rather heavy physically). This clearly shows that I have absolutely nothing to whine about in my working circumstances - so far I even got the luxury of a private room at the office (remains to be seen how long) - something not so trendy but yet so beneficial for concentration.

I've been working with my mindset some time now to mend this mental issue of mine. As I'm interested in yoga I like to view it also from the karma yoga angle - some things just need to be done due to that being my duty (applies also to the multitude of things not so pleasant yet mandatory in life in general). If I was as enthusiastic about my work as some are, things would be easier. I wonder whether it is about having a "perfect" job for oneself or whether it is about the mindset, or both. Nevertheless, not all can have a "perfect" job. In fact, "perfect" is something that likely does not belong to the domain of material life, so it must be about the mindset. It is rather ironic that if one sees things in a positive way, everything is so much easier.

After all, I'm being pretty darn lucky.
submit to reddit Delicious