Mujtaba Hussain

My thoughts on DevOPS on the REA Engineering Blog.

Excerpt:

It should not be marketed as position to slot one person in, or even rotate people through. A software developer who writes the code should be as concerned about his/her code working in production as he/she is in development. Definitely operations are more experienced with deployment and maintenance of products in production like environments, but that does not mean a developer should not base his software decisions before being aware of implications of deploying it in production, or be unaware of the status of his code after it passes quality assurance. The thinking behind devops is the critical bit of software development.

Wrote my first bit of iOS code yesterday. Was great fun. Mostly it was adding a custom image to an app but it was fun nonetheless.

Also, I do not think I like XCode as an IDE but more on that later.

The process of software development is long and hard. It goes through a lot of stages. Development is usually done on your laptop/desktop and once you are done, it is hopefully moved to a production like machine where your changes are tested. If passed, it then is passed onto production. All these different environments are configured differently and possibly access and enable different areas of the application being deployed. All this is usually managed by configurations.

The scenario mentioned here gives rise to the possibility of two different types of configurations and that is going to be my rant for today.

There is application configuration and there is environment configuration. If you have to turn features on or off in your application in production, or release something that you want a subset of your users to see for some testing purposes or for whatever reason, you want flexibility, then that setting is an application configuration.

Any change in environment, whether machines, URL’s, databases etc when you move software from one system to another in the development lifecycle requires environment configurations.

Usually, the way to go about them is to put the configuration in a YAML file. For example

ThisReallyIsSomethingIHate: 1
ThisIsSomethingMeLikey: &AliasForIt
    Name: Gandalf
    Default: #98AFC7
    Update: #FFFFFF

This works fine for environmental configurations which rarely change and which do not need to be altered every so often. But for application changes, where toggling of features on and off, before and after deployment is a common thing, we need something that works, in my opinion similar to a JMX console. An admin console, which tells us the health of the feature/application, and allows us to toggle the features on and off.

This involves moving the application configuration from a flat file into a persistent storage, perhaps a database. This ensures that Operations do not have to worry editing files but actually have a console to do it in, something that is easier to start and manage. This would also help out QA during testing, because now they do not have to write automation tests to edit configuration files but a simple selenium step will accomplish that for them.

I do not know whether this is the right way of separating application and environment configuration, but I do know that we need to separate them.


gcc -framework Foundation message_file -o executable_name


and you can stuff in other tags in it as well.

I do not recommend it for actual iPhone development, but its neat and quick.

The main reason why I like tests is because they helps me be right! That is why I write them! If I am right, the code I write will be good and will not only serve me, but also the person paying me for the code. But as I write more and more code for more and more people, I need faster indicators. I need better indicators and I need consistent indicators. Hence we have build status indicators.

There are a number of variations on these indicators. Build lights, build status shown up on monitors etc. The primary purpose of these indicators is not to tell you that the build is broken; their primary purpose is to tell you that quickly! There is no point having a build indicator that takes its own sweet time! You might as well run tests locally if that be the case. But as you keep adding the tests to the suite, the time between builds will get larger and larger! What you can do is:

• Split builds across machines
• Split builds internally based on application being tests
• Throw hardware at the builds.

Also, a build indicator is useless if the environment it runs on does not mimic your version of production. We cannot ship your laptop, and hence their is no point writing code for it. You must write code keeping your production environment in mind and the build machines should mimic the environment as well. Other wise, your tests are useless. Let me repeat that. If the build machines you run your test suite on do not mimic production, your tests are useless.

A common problem that arises with build indicators is what to do once you have the feedback! Should you drop what you are doing, and try and fix the problem or finish the current work to a point where it is committable and then try and fix the problem? I don’t know! Which ever works best for you! But the build should not be left stagnant for more than an hour after it is broken! Whether the build gets fixed in an hour is not relevant, but someone working on it within the hour is! if you were connected to a ECG and the graph showed nothing, would you be so laid back about it?

Show your business interests that it is in their interest for you to fix the build. You cannot tell them and expect them to agree. Your continuous integration server (for example Hudson , *cough* I meant, Jenkins) should be producing graphs based on several important metrics. Use them to convince management. You are doing a disservice to your code base if you expect to waltz up to your management, soliloquies about importance of green build indicators and get them to agree! Talk to your business, card it up, and make sure you are working on it.

Lastly, be the first to fix a build. Its fun and its cathartic. Fixing things, no matter how small or how trivial, gives you a good feeling. Do it!

A long time ago in a university not so far away, I was about to commence a subject that every body said was one of the most fundamental subjects available. Software Engineering; Processes and Tools. Before that, I had completed(barely) the prerequisite for that which introduced us to the basics of Software Development. So I was excited and very motivated to get this one down pat. I really wanted to do this right and learn as much as I could about the intricacies of designing and developing software in a team environment.

Apart from the usual load of assignments, tests, tutorials and final exam, the one thing that marked this subject out was a large project designed to be completed in a team environment. Whereas previously, in the prerequisite, I had worked with just one other person, this subject would involve teams of five and/or six. I was very excited and immediately approached a few of my friends in the same year and we all agreed to do this project together. I distinctly remember thinking at that point, this is going to be fun!

Alas, I was wrong. Very very wrong. And all the misery that ensued from that point on had one and one simple reason. I was not objective! Don’t get me wrong, my friends were very talented and committed, but they had different scales of working. They all worked in a different manner, some at different times of the day! Some worked at home while some preferred to pull night shifts in the university! The only thing that bound us, was that we were friends! And therein lies the problem. Had I approached my friends on the basis that they were very talented people with whom I could objectively work and finish this project and my friends, in specifically that order, we might have developed very good software. As it turned out, my priorities were skewed. I wanted friends to work with, not to work with friends.

Hence the point of this post. One must always, always, be objective. If you are working with friends, in a professional environment, you must be able to tell the truth to them, in front of others and in private. If you are leading a team, then you must be able to tell people to pull their weight, first by example and then by actions. Being friendly with your friends only works when all of you are clearly working towards a goal and objectives are clear. Trust me, friendships are strengthened when you are brutally objective towards each other professionally.

Postscript: We managed to pass the subject, but ended up not learning much. And thanks to ashridah for the objective talk after my results.

Today, at Rea we had our fourth Innovation Day. We have learnt a lot over the past year about how to engage the business, the QA’s and Operations. We have made every effort to make Innovation Days not just about developers. So today, Steve and I decided to have a play around with Street View and properties.

Over the past few months, I have become a fair bit familiar with the Google Maps API, specifically there third version. It is a very neat little API and allows me a lot of handles on little things that make playing with Google Maps and Javascript such fun. But until now, I had not discovered so many things missing between the V2 and V3 versions.

And it is not just functionality. The very way Street View motions down a street is so much more grainier and jerky in V3. Was it solely because they switched from flash to JS? If so, why? I think it may have been to accomodate iPhone’s lack of flash, but why a simple UserAgent check did not suffice, I do not know!

Things that I found missing from V3 that were in V2

• panTo() method is missing from StreetView
• Cannot seem to give a pancake to the cursor
• Cannot put custom controls on StreetView

Out of those three, the most annoying is the first one, since that ensures that when you walk down the street in Street View, it zooms in rather than pans. And not only that, the movement between the two positions on Street View is a pulse, rather than a flow, negating what used to be a very cool warp effect. Also, these effects are still visible on their main site.

Do not get me wrong, V3 is a great; I am just wondering whether V3 should be called V1.9?

I had great fun reading through this post in the morning. Most of the ones regarding C, I knew, but the others were fun to peruse through. And I must say, Javascript is one weird child.

On the second and final day of the YOW 2010 conference, there were a few management oriented sessions in the evening, a few technically descriptive ones in the morning and two very nice focussed ones in the middle. These were the DevOPS session presented by Michael .T. Nygard and one about taming large builds by Chris Mountford

Chris started talking about how they worked at Atlassian, particularly on JIRA and how they have problems testing and managing the tests. He showed us a few screen shots from many of the builds and frankly they were monstrously large, especially the one about JIRA, which between the time he had made his presentation and presented it, grew larger. He mentioned some very nice techniques that could be used to prevent and/or maintain monstrously large builds as they are very bad. Why ? Because builds are there to give you better feedback and quick. A build that takes 10 hours is of no use in an Agile world. Another thing, which might interest the business owners more, is that large builds have large turn around time and consequently more time needed by the developer to fix the build, which is less time she or he is spending on the work that the business wants done. The one thing I figured that Atlassian does that we don’t do are Canary Builds. But soon, I think we will. A very good session that was very informative, and slightly reassuring that we are not the only ones wrestling with huge monstrous builds.

It was refreshing to have another session which based on actual events and the conclusions and/or lessons that could be and were drawn from them. I have seen many speakers over the last two days conjure up scenarios to fit the example on the upcoming slide, but this was very nicely structured around real world events, with pictures as proof. The whole session was basically an operational and development postmortem of a few events and why those events demonstrate the invaluable aspect of no barriers between developers and operations.

Then Michael went on about how they diagnosed incidents along with developers after chaotic incidents that led to downtime. How they went step by step through any causes and how they actually constructed a KanBan wall without realising it. How process went out of the window and how that had both good and bad consequences. He also reflected on how that defined, to an extent, the intersection of the roles a system administrator and a software developer.

All in all, of all the talks in these two days, these two are the ones I have enjoyed the most. Practical, simple, and extremely well presented.

On the first day of the YOW conference, I was pretty excited about the post lunch session (even after eating a delicious lamb kebab). The two topics that I was very interested in were NoSQL by Justin Sheehy and Strategic Design by Eric Evans.

So off I traipsed into the NoSQL talk, eager about some new news on things like CouchDB, MongoDB and maybe a bit about Mongoid. It certainly was not what I expected. The first half was passionate analysis of what exactly NoSQL is, and more importantly, how we should choose it! The second half of the talk focussed on Riak and I confess uptil today, I had no idea what it was. From there it was a bite more interesting to see actually how Riak worked and how scalability and extensibility were inbuilt into it. Not sure I will be using it anytime soon, but good to know me thinks.

Out I came from this one and went to hear the Strategic Design talk. It was a very well constructed talk on how the best of motives could lead the best of developers astray by setting wrong goals and the drive to never make a compromise. And also metaphors are slightly evil . But the talk was worth listening to, as it demonstrated how refactoring needs to be targeted and how to get business on your side to sponsor the things you think they won’t agree to. Eric demonstrated how models can be extracted in different forms by using a poem, a map and a quote as example domains. It was pretty instructive. Looking back at some of the work I have been doing at personally and some work we have been doing at REA, I would definitely say that we have implemented a fair number of said strategic ideals, especially for the latter body of work.

And that pleases me

Tomorrow, is the last day of YOW 2010 and I am excited about a devops session amongst others.