Continuous Deployment

Posted on February 23, 2010

I’ve been working lately trying to implement some of the things Eric Ries talks about. Primarily the idea of continuous deployments for lean startups and getting away from the fear of release.

On the surface and looking back on some of the disastrous releases I’ve seen (deployed) it’s a frightening concept. The problem with most of those releases though hasn’t so much been that there was a bug or a problem, it was there was no fall back position – the release was made and there was no way to unmake it. Also working in Java releasing WARs has been a pretty big issue – you have to upload the whole release to fix even a small bug like a typo. One bug in these cases can mean you’re 15-20 minutes from uploading a fix.

It’s a very bad place to be and I want to avoid it at all costs in the future, both the problems and as a result the fear of a release.

With that in mind I’ve been fashioning a system for Sproozi where a release is made as an exploded war using Hudson. On commit it’s tested, deployed to a single server in the cluster and tested some more. The server joins the main cluster and it’s monitored as the release is pushed to the rest of the servers to make sure it’s stable.

The only problem I’m having conceptually is how do I treat a commit that comes in before a release is finished – I guess I have to sit on it, but what if the release fails and I have to lock the repository? What about if the release is a success but in the meantime I have 2, 3 or even more commits that I’m sitting on? Do I deploy them one at a time until they’re either all successful or one fails?

How has anyone else practising dealt with these issues?

EC2 Spot instances

Posted on December 14, 2009

The thing I really like about Amazon’s cloud stuff is they’re constantly undermining themselves with new innovations – spot instances are another great example. Taking the utility metaphor a step further you can now rent their services when nobody else is for cheaper, like buying electricity at night.

A lot of the tasks I’m envisioning for Sproozi aren’t really time dependent. While it’s important to show you a page in a timely manner, crawl and index a brand new website you add quickly and basically be interactive there is also a lot I have to do in the background. The huge and growing list of URLs people add all need to be re-crawled and re-indexed regularly is just one of many examples of processing vast amounts of data. These tasks are always running, always in the background.

Spot instances are a perfect fit – I can bid the price I want on extra capacity spin up some extra instances to join the cluster when they’re cheap. Over the next few weeks I’ll probably try to add some spot instances to my Hadoop dev cluster and see what happens.

Useful commit messages

Posted on November 20, 2009

I realised that I’m not very good at this when I looked at my last commit for this theme. The message was “added a few things” a message I find myself using when I have nothing else to say, or if I’ve done too much work and either can’t remember all of what I did or be bothered writing it all out.

It’s bad form though, and I’m trying to learn my way into using git properly to make it happen less. Basically I’m still trying to find the right place between a commit for a unit of work and my previous more traditional view that all commits must compile and pass tests. At the same time I’m trying to reconcile that with a Kanban board way of organising what to do.

How does everyone else commit? When, how often and what ‘rules’ do you apply?

Stop asking questions.

Posted on July 19, 2009

I’ve had an ongoing conversation with a few people over the course of a few projects about the questions to ask when people sign up on the web or what hoops you make them jump through.

Ask most people what they need to know about user’s and you’ll end up with a list of information from the useful like name to the truly useless like job title or title. Give me one good reason for either. Are you going to be sending Mr John Smith or Dr. Sarah Jones a letter using their formal titles?

When it comes to things like Job Title, the reason I usually hear is that it’ll be useful for marketing and demographics later. Maybe, but I’ve never actually seen that later come to anything. So I don’t buy it.

If the signup form is the depth of your engagement with your user you might as well drop the pretense of a web application and go back to sending newsletters. Just don’t do it. Ask your user the bare minimum number of questions you need to get them started using what your product does now and stop throwing up barriers to entry.

Aside from just the questions that are asked there are other things, the hoops almost every site makes you jump through before you can start using it. Probably the best example is the ubiquitous confirm your email step. Do we need to be doing this? In some circumstances I’ll admit it makes sense, if your delivering content via email for example or signing up for a mailing list. Of course you want to make sure the user wants what you’re sending them.

But for every other site, I’m not sold. It’s just another step they have to take to start using your service and all the little steps all add up to users hating signing up for new things.

Also it begs the question, why even bother? For 99% of websites the only reason to make sure a user has entered their email address correctly is so that you can let them recover a password. You’re adding the step for people that have lost their password and haven’t managed to get their email right. That’s going to be a fairly small proportion of your users; even less if you let login with their email address, then they’ll know it’s wrong anyway as soon as they try it.

Do we pander to the idiots or make things as quick and easy for everyone else? I’ll take quick an easy any day.

I’m finding myself not just asking what any given field in a form is for and making it justify it’s self, I’m trying to figure out how to get by without it. I’m leaning towards going beyond just the bare minimum and into the realm of how can I modify this service to work with even less.

Open Source for Business

Posted on June 8, 2009

I didn’t get this posted yesterday because the Internet crapped out in our area. Nothing but excuses, I know.

I’ve been working beyond the bleeding edge, using a version of the Nutch code that’s not even made it into the Apache SVN for the project yet. To celebrate the fact that my contributions will make it in I figure its a good time to get into open source and business.

To put it briefly, and as you can probably guess, I’m pro open source. I use it extensively and I push back as much as I can. When it comes to the most of the code I write there really isn’t much commercial benefit in keeping it hidden so it just makes sense to give back.

There are two types of business on the web, one where you provide a software service and that is the product and others where you provide access to data. It’s pretty easy to tell which camp you’re in.

37 Signals for example, they’re in the first and their software probably isn’t something they just want to let people download – unless they’re incredibly brave. Doing that would mean that they’d be competing on margins for the cheapest hosting, users would flock to the cheaper services, have a poor experience and blame the software.

Sproozi on the other hand is the second type, our data is what users mostly care about and we’re not planning to be precious about our code. I’ve already been pushing some of the changes I’ve made to Nutch back into the project and we’re planning open source projects of our own in the coming months.

One of our plans we have is to build iPhone, Andriod and other phone based applications for our service and release them as open source projects. We’re planning to write them (or have them written for us) and release ‘official’ versions. Then release that code as open source project to provide a framework for developers so that they can build great things from it and on our API.

If there are any experienced iPhone, Android, Blackberry, Symbian or Pre developers out there that want to get involved, drop me a line were a ways off yet but would love to chat about it and get some very early feedback.

Related articles by Zemanta

• Android Application Development (oreilly.com)
• Palm Pre will debut with only a few apps available (computerworld.com)
• Yahoo! breeds Pig that talks elephant (theregister.co.uk)

Spartan Programming

Posted on July 10, 2008

Jeff Atwood was pleased to find that his coding conventions, the ones we all settle into, have been tending towards those of a style of programming called Spartan Programming. Which is minimalist programming, as in “a spartan lifestyle”. Check his post out for a more in depth explination but suffice to say the aim is to reduce complexity in any way possible.

I’ve got to agree that more and more I’ve tended in that direction too. I don’t remember the last time I had more than a handful of conditionals or loops in a method. My methods are terse and accomplish one thing, sub tasks are split out into other methods. It’s come very naturally, but it has taken time. I don’t think it’s something you can be taught, or learn. It’s definitely a practice thing.

One aspect of Spartan Programming where I disagree is with the terse variable names. I think they could end up introducing more accidental and avoidable errors. They lessen the utility of some key IDE features. And generally would just make my life harder, so I’ve gone the other way. Once upon a time I would happily create the following code:

public void sendMessage() {
    MimeMessage m = new MimeMessage();
    m.setBody("This is a test");
    m.send();
}

These days I try to avoid calling variables things like m and prefer to call them what they are:

public void sendMessage() {
    MimeMessage message = new MimeMessage();
    message.setBody("This is a test");
    message.send();
}

I know the arguments and logic:

• The method should be simple enough that the variable name m is clear enough.
• The shorter name reduces complexity and therefore errors.
• Less characters means lines don’t get too long

Of these three the third is the only one I really buy though. Long variable names and too many characters on a line can push some stuff too far to the right potentially hiding errors or overly complex code; but it’s an edge case and there are other ways to solve the problem.

While true that in a short minimalist method a variable m would be clear enough, a variable called message is equally clear. The shortening doesn’t gain anything in clarity. The argument for less characters reducing errors in the era of modern IDEs doesn’t wash with me either – My IDE tells me of I mistype a variable and auto completes them furthermore by typing actual words I actually improve the accuracy even more.

Shorter names on the other hand can tend more towards a crowded namespace and can result in properly typing the wrong variable. This is especially a problem in a weakly typed language.

Consider the following:

function sendInviteMessages(n)
    for i = 0 to n
        Set m = Server.CreateObject("MimeMessage)
        m.Body = "Welcome to our fancy new site!"
        m.Send
        Set n = Nothing
    next
end function

What you see here is something that wouldn’t happen, or not as often with a variable name like number instead of n and message instead of m. Worse still, depending on the language and how it treats null variables it might even pass a quick test with a single address and appear to be working!

The one place I totally disagree with the terse variable naming conventions though, is the method parameters. That actually makes life harder when working with code completion. Which is easier to follow?

There are plenty of other code completion and helpful shortcuts in a modern IDE where the parameter names are used to make our lives easier. All of which lost, with no real gain.

I’m not arguing that variable names shouldn’t be terse and sensible. Going back to the first example the following is just stupid and is far worse than using just m, it’s practically illegible and very difficult to just eyeball the code to see what is going on.

public void sendMessage() {
    MimeMessage theMessageToSend = new MimeMessage();
    theMessageToSend.setBody("This is a test");
    theMessageToSend.send();
}

Other times I go the opposite way and have adopted conventions for making some variables extremely terse. InputStream and OutputStream variables for example I always call is and os. So I do try and be as terse as possible, but never terseness at the expense of obviousness of intent.

While I agree with most of the practices and like Jeff was nodding along as I read about Spartan Programming, there are a few I don’t fully follow and some I’m against. In the end though I tend to agree with his assertion that Minimalism isn’t always the right choice, but it’s rarely the wrong choice as long as it’s not at the expense of clarity of intent.

Release early – Release often

Posted on July 9, 2008

Agile methodologies have recently been getting something of a bad name, it’s not unwarranted but it is unfortunate. The problem isn’t actually the process, an agile process will work very well – if the infrastructure supporting it is capable of of being agile as well. By supporting infrastructure for a process here I don’t mean servers and software I mean the people and the organisational foundations. The organisation trying to use an agile process must it’s self be agile, quick to react to change. Agile processes that work have one thing in common, they encourage features to be released early and released often.

A pretty basic overview of our process goes something like this:

Big Bag of Features

Our projects and releases go through a number of stages, the first we call The Big Bag of Features. When a project is really just an idea, it’s something we might want to build and we basically rough out some ideas and start to grow them into all sorts of crazy future versions, we talk about all the features we’ll need to achieve them and how we’ll go about it.

Minimum Stable Feature-set

From the BBoF we usually end up with at least one, more often than not a number of possible outcomes and development paths. By talking through all of the options, all the features and all the future versions we’d like to be able to build leads us naturally to something we can build; a vision for a product.

Citrus is a good example of this practice, we’re at the stage where we’ve decided the vision for the project, have a big bag of features we’d like to implement and are now in the process of creating our MSF

From the vision we decide what we need to make a product work; less is definitely more at this stage. We normally take each feature we’ve decided would be nice to have and try to think about who would use it and why. If we can’t come up with a compelling argument for most of our users using it, we normally chuck it back into the bag to worry about later. We refer to the result as theMinimum Stable Feature-set (MSF) or Minimum Launch Feature Set.The bare minimum we need to launch a working product.

The MSF becomes the next iteration, and once it’s set we don’t alter it. Nothing is added or taken away. On the surface this might seem either silly or a bit high and mighty, but we can do it because we put the work in ahead of time. We’ve discussed how we envision the product being used and we make sure we’ve covered all the bases. New great ideas just aren’t important enough to include, no matter how good they are. We leave them for another iteration and another day.

These ideas can be politically disastrous in some organisations, it’s downright heresy to tell the boss, or the sales director or a colleague what amounts to: their newest great idea just isn’t good enough. It’s important for an organisation to have focus though, this is where the MSF and short iterations come into their own. Releasing early and often means that you’re quickly on to the next round of updates and those great ideas can be looked at objectively alongside anything else in the Big Bag of Features.

Getting started

Finally we start coding the interfaces first, then backend to meet our MSF. In the case of a new project partial interim releases are made for testing. We try to work out an entire area of the project first then release that to our testing audience for feedback. As the iterations contiune towards release we add all the features from our MSF.

Updates

When it comes to updating projects we throw away the Big Bag of Features and start all over again.You heard it right, throw them away, gone. A long time ago I learned that feature lists just cause trouble. You can’t help but schedule them, the schedules are very loose and they’re never met but still promises are made based on them. In the end they start to read more like a list of failures, things your application doesn’t do. More importantly, things your application will more than likely never do. So we throw them away and start again with a new Big Bag of Features, the most important features will bubble to the top of your memory anyway and will end up right back in this new BBoF.

For an update our MSF is usually a single feature, or small feature set if they work best that way. The aim is to keep the releases small and concise, targeting one area of functionality. This doesn’t mean that we don’t set milestones or visions for the future development of the product, we normally make a milestone accomplish some greater goal or vision for the product. But we try not to roll a list of disparate fixes to various parts of the application into a single release, we break them up into many smaller releases.

Rinse and Repeat

We continue in this way for the life of the project releasing features to users early and often so bugs are spotted quicker and we never go too far down a dead end, creating something users won’t use. It’s during this stage that we really work hard to measure how an application is being used, or not used so we can focus our energy only on the most important parts. We never try to be everything to everyone, we’re not afraid to disappoint some people or to just say no. We want our application to be great at what it does, not mediocre at everything.

Get your development under control – all about version control

Posted on July 7, 2008

The key to any successful development team, far more than any other tool, is version control. No matter how many team members you have, if you’re dozens of people spread across the globe or one lone developer in a basement you need version control.

A number of free and paid for systems exist, largely they accomplish the same ends: the ability for multiple developers to collaborate on the same code, the ability to track changes and the ability to manage changes across concurrent versions of code.

change management

This is potentially the biggest benefit of any source control system, the ability to manage changes in code. Most version control systems manage changes through a series of checkouts and commits by developers.

You check the code you want to work on our of the repository creating a local copy, make changes to the code and commit your changes to the repository again. Any developers that are working on the code can update their local copies an the changes you’ve made will be merged in.

This can be a massive boon, even if you’re working alone. Version controls systems will also let you roll changes back, so if you’ve made a mistake, deleted a file or need to see how something worked before you can go back to any previous version.

concurrent versioning

Concurrent versioning is a great benefit when you’ve released software or are developing more than one major feature at the same time. It’s the process of having two working versions of the code which can be edited independently.

In the case of a released version of software this allows you to release a patch before new features still in development are ready to be released. You can patch the bug, release an update to users and then use the version control system to merge the change into the development code so that your next release with the new features gets the fix too.

If you’re working on a major feature you might also want to branch your code temporarily. This would allow you to perform some pretty significant refactoring on areas of the code which would not affect developers working on the rest of the code, when you finished the work you could then merge your code into the main development branch prior to release.

what else

For more information about version control see the excellent Visual guide to version control at betterexplained.com.

How to get started

There are a number of choices and a number of web based version control systems. Opensource subversion and git are used by a large number of people and compete on features with most commercial offerings. If you don’t want to set up your own server to host a repository Google code, beanstalk and lighthouse are just a few web based services that can host version control environments for you. Google code is free, but you’ll need to be developing an opensource project the other two are subscription services.

I personally use subversion and love it but with any version control system, commercial or free the most important thing is to pick and fit a system into your development environment. Don’t pick one that is going to force you to change the way you work. Try a couple if you can and pick the one that works best for you.

Minimum Stable Feature-set

Posted on July 3, 2008

From time to time this blog will reflect a decision we take as a company, this is one of those times. We’re currently working on a product called Citrus, a better way for agents to sell property online because frankly existing stuff sucks, scheduled for launch in the autumn.

One of the key decisions we’ve taken is not to implement everything on our feature list. We’d rather build half an app than a half assed one. As flippant as that sounds, it’s true- we could spend years analysing all the competition, ensure that we’ve not only bested them at the key features we know we need to implement, but at everything else as well and some day in the distant future launch a product. Or we can look at our feature list, choose the killer features we know our application needs to have to achieve everything 80% of our potential customers need from our app and forget the rest.

We’ve chosen the latter. It’s going to help us get to market faster, it’s going to help us get customers quicker and it will allow our customers actual use of the product drive our vision for it- not some paper list. Also if we’re totally wrong in assumption that a good product can gain traction in a poor market, we know and can move on sooner.

UPDATED: I renamed the title for consistency – we refer to the stage of development as either both the Minimum Launch Feature-set or the Minimum Stable Feature-set internally – the latter being the best really since it’s useful both for an initial launch and for discussing post launch updates.

Never rewrite your software

Posted on July 1, 2008

All software is a work in progress none is ever perfect. It will always have bugs, sometimes it might even seem that the software is nothing but bugs an it’s in these dark times when we normally say fuck it, let’s start over. Not always, some of the good excuses I hear for a rewrite are:

• New blood Someone new comes along and the first thing they suggest is a complete rewrite – It’s alarming the number of times I hear this suggested. Sometimes it’s contractors, which is somewhat exusable – it’s more billable time for them. Other times it’s by people newly in charge or new to the team.

• Team members have left – rightly or wrongly they’ve been blamed for the code, and coding them out seems like a good idea

• Too many bugs – An effort to hammer out the bugs feels like it would take longer than to rewrite from the ground up

• New tools! – A shiny new framework is out and you want to use it, it will solve all your problems and put all your spare cycles to work sorting out poverty and war.

Common to all the above is the fact that everything seems like it would be easier if only you could get out from under the crushing weight of legacy. And it’s a very attractive fallacy; Bugs could be solved by reworking the architecture; Features would be easier if only the code wasn’t written that way; That new framework would fix the bugs we’re seeing because of the one we’re using.

resist the temptation

They sound like good reasons, but that’s the problem with attractive things that are bad for us – they’re attractive. It’s so much easier to believe that the grass would be greener if we re-turfed the lawn than to work out how to fix those brown patches.

it almost always takes longer than you expect

A complete from the ground up rewrite will take longer than you think. It will probably take almost as long as it took you to get where you are now. It’s tempting to believe that you don’t need planning meetings to work out what has to be done. It’s right there a spec in the current implantation, just get started! So you’re initial estimate is just a stab in the dark anyway.

Add to this the further false belief that since you implemented the features once already, you’ll be able to do it faster this time around. You won’t, well you might – but you won’t implement them as optimistically quick as you think you will.

Throughout the process you’ll also be further from a releasable product than you think you are – much further. In a rewrite you have to rewrite every feature all those “all we have left in component x to do is…” bullet points add up.

All of this leads to one last problem with timescales for rewrites – in order to get anyone else to buy into them they have to be wildly optimistic about the schedule. So just when you need everyone most motivated and excited, as you’re gearing up for a release, you’ve set them up to be demoralised by missing an increasing number of deadlines.

you’re reinventing the [buggy] wheel

Programmers are people, people are creatures of habit. You’ll inevitably end up implementing part of your radical new system in a very similar way to your existing system. Then you’ll be forced to make the same compromises as you’ve made before and you’ll undoubtedly introduce some bug that’s already been patched in your existing software.

Some of your old code will also be good code, at best you might cut and paste it into the new code base but for the most part though trying to hammer it into place and wire it up is going to be as long and as buggy a process as rewriting it. So instead you follow the rewrite it all logic and rewrite it all.

you’re standing still

When you’re rewriting your software you’re stainding still in almost every way possible. If you’re rewriting everything, you’re spending an awful amount of time not moving your application or business forward. Personally and professionally you’re not learning much new (unless a new framework is your excuse).

Even if you’re squeezing some features into your rewrite, probably because it’s been so long since a release you’d be lynched by a mob if you refused, it’s so long before the users are actually going to see them that you might as well not be. The perception of your software isn’t going to be sometimes shaky, but getting better all the time – it’s going to that it’s buggy and the bugs are never fixed.

you’ll probably end up no better off

The sad and somewhat painful truth is if a team can’t fix something – getting them to make a new one is probably not going to fare much better. Your new version is probably going to be no better than the old, will certainly be worse than an itteratively developed and incrementally improved version of what you had could have accomplished in the same time.

Also problems you’re having with a framework or tool probably aren’t going to be magically solved by using a new one. Most of the these arguments are born out of a naive belief that because the framework isn’t doing what the developer is expecting, the framework has bugs, changing frameworks should then solve the bugs. The problem is of course developer expectation versus reality and more often than not due to some underlying misunderstanding of the platform the framework or tool is abstracting – a reality that a new framework cannot change.

how to sensibly go about a whole rewrite of your software

While it’s not sensible to start a full rewrite of your software, it’s also ludicrous to decide that nothing written should be changed. Software development is a process not an event. If it’s not working fix it; if it’s complicated and bug prone, refactor it; if it’s shit, code it out. Just don’t throw the good out with the bad.

Start where it hurts

In the case of bug riddled software start with an area that is causing the most trouble. Patch the big bugs first. It can be a painful process, I won’t lie. It’s demoralising to everyone to have buggy software and it’s not exciting or glamorous to fix it.

Bug fixes are one of the most thankless tasks developers are asked to perform, so make sure you break it up with some of more challenging and interesting tasks, throw a few features into each iteration.

if I could do it all over I’d change…

These are the generally the things that are hurting the developers rather than the user. Some of them are far reaching, architectural or framework choices. They can all be fixed though, and you don’t need to rewrite the whole application to get there.

Again, start with the ones that hurt the most, if it’s part of the architecture refactor it. If it’s a framework you’d like to integrate to solve problems, try it out on part of the code and work to roll it out incrementally.

I promise it will be quicker, a lot quicker to go down the incremental route than it will be to rewrite the whole thing in one go.