Some days ago Matt and I had to deploy a rails application. This was an application that both Matt and I had no idea existed about 2 days before we ended up deploying it so we were quite looking forward to it on a very rainy wednesday! (Why does it always rain when I am deploying???)

A day before Matt had tried to deploy the application to staging to see whether we could test it out before production got her hands on it, but the staging environment had not been puppetised for this particular app. When we realised that deployment to staging might not possible, we spoke to our resident friendly release manager Aaron and laid out several plans. The first action was for me to test the shi^H^H^H hell out of the app locally (with production data) as much as I could while Matt tried his best to puppetise staging.

Early wednesday morning, I managed to test the app completely and ensured that the changes some awesome developer *cough* had made worked. Matt had also finished his spike with staging but no goodness. So we both looked at each other and decided to deploy it, to production herself.
Four hours, a plate of sushi and a coffee later, the thing had been deployed and we had ensured that it worked in production and informed the relevant parties!

From this all, I have learnt a few things:

1. Even though we both knew nothing about the app, I ensured that Matt was involved in it from the very beginning, and it helped a great deal. Springing deployment surprises on OPS is not cool and helps no one, least of all you as a developer. JIRA and Zendesk are cool, but nothing, nothing will actually substitute going over and talking to your fellow OPS people.

2. Puppet. Puppet is cool. The fact that we could quickly spike things in staging and decide whether to deploy there or not was simply because puppet allowed us that level of automation. Invest of in such tools, whether they be puppet, chef or babushka

3. We did not panic and delay the release when we found out that we could not test it in staging. We sat with Aaron and figured out the risks of each scenario and associated rollbacks. We did as much testing as we could do before hand and then went ahead with the deployment. To quote my colleague Jon Eaves, “Fear is the mind Killer”.

Above all, it was great fun to work with Matt, and deploy something along the way :). This has made me realise that software development, and deployment, is amazingly rewarding when done with awesome people.

Today was fun. A lot of fun! I paired all day today with Jared Hunter from operations and we spent all day deploying about four applications written in three different languages to two different environments using three different deployment tools. And today’s attempts have made me realise a lot of things

….

OK fine, I am lying. It wasn’t a lot of things!!!

But this experience has made me realise a few things!

Operations are over worked. I knew this fact of course. I knew this in the same sense as I know getting shot or being stabbed hurts. There is a lot of difference between knowing and realising. We spent nearly two hours in total debugging failed deployment because certain assumptions were made in the applications which did not coincide across all environments. Sometimes the environments were at fault, sometimes the applications.

I have learnt that one cannot leave deployment towards the end of the iteration. It must be planned at the same time as when we kick our iteration off. When we discuss the work planned in the iteration, it is at that point, we must write our deployment notes, and keep adding to them as work keeps getting done. The deployment notes should be nothing less than a technical card and must be approached by the business in the exact same way as developer asking to remove some technical debt.

The problem with leaving deployment planning towards the end of the iteration is that people have to now remember work they did at the start. Do it at the same time as you hand over the card to QA and you are done with it; no more chasing up after the fact.

I have learnt that having consistent deployment procedures when you deploy applications written in three different languages is not possible. Layers and layers of decoration buffers must be applied between the deployment procedure and the applications for that to happen and God help you if you have to then debug something. Java is very different from Ruby and Perl. And even though the latter are both interpreted, the way they manage dependencies and interact with other system layers is different. At least the way we do it. Choose one platform, if you can, and stick with it, if you can. It makes deployment, debugging, rollbacks and dependencies much much easier.

I learnt Puppet is very cool. I managed to give access to a bunch of users across different environments, documented the change and kept the history of the change for posterity using puppet in a few simple steps. I will however have to shelve my idea of using Babushka and Puppet together. (I think I had had too much coffee at that time).

Last thing I learnt is that canadians have a very good sense of humour, no matter how much you needle them about maple syrup, or ice hockey or you know, basically being Canadian. They are a lot of fun to work with! Mad props to Jared and Kerri for giving me that belief. Come to think of it, even the tree we associate with Canada, the Maple tree and the tree associated with Kashmir, the Chinar tree are similar.