I've been working lately trying to implement some of the things Eric Ries talks about. Primarily the idea of continuous deployments for lean startups and getting away from the fear of release.
On the surface and looking back on some of the disastrous releases I've seen (deployed) it's a frightening concept. The problem with most of those releases though hasn't so much been that there was a bug or a problem, it was there was no fall back position - the release was made and there was no way to unmake it. Also working in Java releasing WARs has been a pretty big issue - you have to upload the whole release to fix even a small bug like a typo. One bug in these cases can mean you're 15-20 minutes from uploading a fix.
It's a very bad place to be and I want to avoid it at all costs in the future, both the problems and as a result the fear of a release.
With that in mind I've been fashioning a system for Sproozi where a release is made as an exploded war using Hudson. On commit it's tested, deployed to a single server in the cluster and tested some more. The server joins the main cluster and it's monitored as the release is pushed to the rest of the servers to make sure it's stable.
The only problem I'm having conceptually is how do I treat a commit that comes in before a release is finished - I guess I have to sit on it, but what if the release fails and I have to lock the repository? What about if the release is a success but in the meantime I have 2, 3 or even more commits that I'm sitting on? Do I deploy them one at a time until they're either all successful or one fails?
How has anyone else practising dealt with these issues?