I work for realestate.com.au and recently we launched a new section of our site called Share. Previously, it used to exist on a physical data centre as a Perl application but now exists as a Rails app in the cloud using AWS. The re-platform was achieved in mere 7 weeks and was a direct result of the hack day culture at realestate.com.au. But thats not what I am, at this moment, going to talk about.

One of the features of the re-platform was the build pipeline that we constructed. A suite of tests which led to building an RPM; which gets deployed to an EC2 out of which we bake an AMI which then forms the basis of our active and inactive production environments. But the main beauty of this pipeline was that a commit from the development environment (which in our case is our/my laptop) goes straight to the inactive production. And since anyone, and I mean anyone in the team can switch inactive to active, there is a very real chance that a commit made now goes straight to production. This has not been done anywhere in my company up until now and as far as I know, anywhere in Australia. The only other company I know that does that is Etsy (and they do it extremely well).

This aspect of development was certainly new to me. And this definitely produced a new way of working for me. The immediate affect it has is that I immediately began thinking of what was covered by good tests and how much. Seems simple doesn’t it! This can only be a good thing. No person should live behind the walled garden of tests written by someone else. A feature of our development is that every time work on a feature is done, and we do handover to a person doing QA, not only do we handover the completion of the feature, but also have to show the coverage of tests and have to justify the tests.

Another aspect that is developed in me was to be sure ensuring that I am always pairing on the most important business features of the application. When I am changing some technical aspects of the code, for example, when I am changing

some_array.map {|x| x.something}

to

some_array.map(&:something)

I really don’t need to have a pair as I am changing how something is calculated and what is calculated remains the same.

But when I am changing some core business feature, for example what determines a a listing is active, i.e.

def published?
  self.status == :active
end

becomes

def published?
  self.status == :active && self.expiration_time > DateTime.now
end

This is a very core change in business functionality and because I know that this change goes straight to production, I KNOW I want a pair to check and double check this. The temptation is to get over with it because it looks simple and hence when we work in the walled garden of other people writing tests, we change the code, and write a bunch of simple tests and here you go. And to be honest, there are times when I have been tempted with this behaviour. The idea of me typing

[~:] $ git commit && git push

And it going straight to production has definitely made me aware of how aware I have to be of the implications of my code changes, from trivial to the non trivial.

These may seem small things initially, but over the course of the project, this mentality helped us deliver a complete revamp and re-platform in only 7 weeks. I encourage whoever is reading this, to definitely try this, but if you are not used to this, then start small. Maybe from a local project and then show the benefits to your work colleagues. I assure you that this approach is definitely worth a try.

Today was fun. A lot of fun! I paired all day today with Jared Hunter from operations and we spent all day deploying about four applications written in three different languages to two different environments using three different deployment tools. And today’s attempts have made me realise a lot of things

….

OK fine, I am lying. It wasn’t a lot of things!!!

But this experience has made me realise a few things!

Operations are over worked. I knew this fact of course. I knew this in the same sense as I know getting shot or being stabbed hurts. There is a lot of difference between knowing and realising. We spent nearly two hours in total debugging failed deployment because certain assumptions were made in the applications which did not coincide across all environments. Sometimes the environments were at fault, sometimes the applications.

I have learnt that one cannot leave deployment towards the end of the iteration. It must be planned at the same time as when we kick our iteration off. When we discuss the work planned in the iteration, it is at that point, we must write our deployment notes, and keep adding to them as work keeps getting done. The deployment notes should be nothing less than a technical card and must be approached by the business in the exact same way as developer asking to remove some technical debt.

The problem with leaving deployment planning towards the end of the iteration is that people have to now remember work they did at the start. Do it at the same time as you hand over the card to QA and you are done with it; no more chasing up after the fact.

I have learnt that having consistent deployment procedures when you deploy applications written in three different languages is not possible. Layers and layers of decoration buffers must be applied between the deployment procedure and the applications for that to happen and God help you if you have to then debug something. Java is very different from Ruby and Perl. And even though the latter are both interpreted, the way they manage dependencies and interact with other system layers is different. At least the way we do it. Choose one platform, if you can, and stick with it, if you can. It makes deployment, debugging, rollbacks and dependencies much much easier.

I learnt Puppet is very cool. I managed to give access to a bunch of users across different environments, documented the change and kept the history of the change for posterity using puppet in a few simple steps. I will however have to shelve my idea of using Babushka and Puppet together. (I think I had had too much coffee at that time).

Last thing I learnt is that canadians have a very good sense of humour, no matter how much you needle them about maple syrup, or ice hockey or you know, basically being Canadian. They are a lot of fun to work with! Mad props to Jared and Kerri for giving me that belief. Come to think of it, even the tree we associate with Canada, the Maple tree and the tree associated with Kashmir, the Chinar tree are similar.

I was writing something in python some days ago and I realised that python actually provides a neat way of handling function pointers. Or at least their version of it. Its called lambda.


>>> a = lambda x : x**x
>>> a(4)
256
>>>

So that got me thinking! In how many languages could I redefine pointers to functions/methods dynamically. So I decided to try Ruby; and lo and behold, Ruby has a another unique way. It does so, by passing around blocks.


>> def name
>> puts "Start"
>> yield 1, 2 if block_given?
>> puts "End"
>> end
=> nil
>> name { |x,y| puts "#{x} - #{y}" }
Start
1 - 2
End
=> nil

Nifty, isn’t it!

Then there is the ever beautiful C. Simple and elegant


float add(float a , float b){return a+b;}
float minus(float a, float b){return a-b;}
float diva(float a, float b){return a/b;}
float mult(float a,float b){return a*b;}

float answer(float a, float b, float (*func)(float,float)){
return func(a,b);
}

Call => answer(a, b, &add);

As you can see, only in see do we actually explicitly pass around the address of the function itself, which is why I like C so much, because simply by looking at the code, I know exactly what is happening.

Now looking at Cocoa, I can hardly see any major difference between that and C. Sure, you have a whole heap of square brackets strewn about and the letters “NS” pop up irritatingly everywhere but, it is still C, which makes it cool. Although, I still have to figure out why interactions between Obj-C code and Interface Builder seem to resemble static method calls.

Perl too fairly easily resembles C when it comes to passing around subroutine pointers.

Of course the grand daddy of all these is Lisp and Scheme and you can not do much in them unless you deal in functions. Other languages I want to mess about in with regards to function pointers is Scala and Dylan.

This was heaps of fun 😀

As a software developer, we have to be both Michaelangelo and Sherlock Holmes every now and then. Michaelangelo in the sense that every now and then we get the opportunity to create something stunning, something beautiful virtually out of nothingness. Sherlock Holmes because more often than not, things go wrong and we have to go hunting for the clues that will lead us to the perpetrator. Guess which one we get to play more ?

So that raises the question that how can we make debugging easier for ourselves? Since we write the code that breaks down, surely there is a way to ensure that we get something in the log output every now and then to ensure things went smoothly, or otherwise. Hence we developers starting putting random statements into log outputs and helped ensure that system administrators who have to monitor this output day in and day out hate us with the passion of a supernova. Well done lads!!!

So here is my beef. Lets not put every thing in logs. As Holmes points out to Inspector Lestrade in the case of A Study in Scarlet, it would have been much better for the former if the latter had not muddied up the scene with too much superficial information. A case in point is the starting up log output of JBoss. It is horrendously loud, verbose and useless. Most of it is INFO and very little is needed to be debugged. It takes 53 seconds to start, yes 53 seconds and in that 53 seconds Apple’s native terminal might run out of buffer space. And if something goes wrong, then well best of luck with your scrolling abilities.

There is a reason why discretion is the better part of valour, and debugging. Log only that which is critical. Remember, you are not the only one who has to see these logs and perhaps sift through them. System administrators and QA also have to look and sift through them and it does no one good when it is full of rubbish. We do not need to log everything. If someone tells you to err on the side of caution, tell them to bugger off.

Remember, a log output is like the program trying to talk to you. The more appropriate its trumption, the easier it is to understand.