So long IDEA

Posted on May 22, 2010

I’ve been using Intellij IDEA for years, I pretty much swear by it for development and I have no real complaints. But now I’ve got a new job and they use Eclipse. Now, so do I. I figure there is no way to properly learn to use Eclipse if my heart is still in IDEA, so there’s nothing for it except to say goodbye IDEA and move over to Eclipse for everything.

Looking around the Eclipse ecosystem and there are an awful lot of what look to be excellent plugins. I know IDEA had a plugin system but the ecosystem never seemed as good and I never really got into them.

Now it’s off to import my projects and see what’s what. I did try this a few weeks ago after my interview, but gave up with the excuse that I wanted to get some work done.

OAuth 2.0

Posted on May 20, 2010

I’ve been updating my OAuth library to support OAuth 2.0 mostly so I can add Facebook to Announce.ly and Sproozi, but more on that later. OAuth 2.0 is similar to 1.0 but changes a few key things fundamentally and isn’t backwards compatible.

What’s wrong with 1.0, doesn’t it work?

It does, but probably the biggest issue is the fact that you have to sign the message knowing all it’s content beforehand. This works well if the data is on the querystring in a GET request or for simple operations but isn’t optimal if your data is part of the POST body. It also means you have to construct your requests in a certain way, which is a bad thing.

Take photo, audio or video data – to post that you’ll need to sign the whole request and it’s not clear how it should work with multipart data. There are several extensions to the spec that deal with some of these issues, but the fact that there are non standard extensions to do something pretty standard kinda says it all.

Even if you’re not dealing with these issues you still have to work with your requests as units where you know the whole content beforehand.

What’s new in OAuth 2?

OAuth 2.0 in it’s simplest form works over HTTPS connections and simply asks for a token – the security and trust are built in to the protocol. It’s that easy.

OAuth 2.0 sill lets users sign messages to transmit them over insecure channels, plain HTTP, but the signing methods are much easier to implement. Gone is the complicated parameter normalisation algorithm and in it’s place is a much simpler version that doesn’t require POST data in the signature. So even with multipart submissions it should just work.

At the moment I’m cleaning things up and preparing the oauth library to work with oauth 2.0 and changing the way it works to reflect the simpler way oauth 2.0 does. You can check it out on GitHub [http://github.com/andrewmccall/oauth]

Some progress on continuous deployment

Posted on March 2, 2010

As I posted a couple of weeks ago I’m working on working a continuous deployment setup so that I can push changes live quicker and easier for the projects I’m working on.

I’ve installed and configured Hudson, which was a good first step. It’s shown me that my builds are a bit fragile, they’re heavily based on packages that I’ve compiled and installed into my local Maven repository. To solve the problem and to make my build a little more sensible I’m getting Artifactory working.

At this stage I’m just trying to get a very basic system in place so that I have an idea of how long a deployment will take. It’s a little more complicated because I’m using Java and the WARs contain all the libraries. I could just use the defaults and push a deployment directly to the target servers, but that doesn’t work. First wars are big, they can easily weigh in at 20Mb.

There are a couple of things I could do here. Ideally I could unroll all the jar libs and only include the classes I need. That would drastically reduce the size, as a bonus I could probably also have it give me a warning that I’m including an unused jar, even better if I could work out the path to the maven dependency. But there’s the whole problem of resources, the fact that using Springframework means most of the classes aren’t being explicitly called and the fact that since I don’t know anything that does all of this this, I’d have to write it all.

So I’ve written that whole idea off, and added a shorter to-do item to make sure I’m only including jars I need. Instead I’ve gone for a far easier solution: deploy exploded wars, and use rsync so I can push only changes to the server.

The next step is to get trigger a Hudson build with a commit-hook and use a successful Hudson build to trigger a script that does the rsync. Then I’ll have very basic continuous deployments up and running albeit without any automatic roll backs of a failed deployment which ultimately is a must, but it’s a start.

Continuous Deployment

Posted on February 23, 2010

I’ve been working lately trying to implement some of the things Eric Ries talks about. Primarily the idea of continuous deployments for lean startups and getting away from the fear of release.

On the surface and looking back on some of the disastrous releases I’ve seen (deployed) it’s a frightening concept. The problem with most of those releases though hasn’t so much been that there was a bug or a problem, it was there was no fall back position – the release was made and there was no way to unmake it. Also working in Java releasing WARs has been a pretty big issue – you have to upload the whole release to fix even a small bug like a typo. One bug in these cases can mean you’re 15-20 minutes from uploading a fix.

It’s a very bad place to be and I want to avoid it at all costs in the future, both the problems and as a result the fear of a release.

With that in mind I’ve been fashioning a system for Sproozi where a release is made as an exploded war using Hudson. On commit it’s tested, deployed to a single server in the cluster and tested some more. The server joins the main cluster and it’s monitored as the release is pushed to the rest of the servers to make sure it’s stable.

The only problem I’m having conceptually is how do I treat a commit that comes in before a release is finished – I guess I have to sit on it, but what if the release fails and I have to lock the repository? What about if the release is a success but in the meantime I have 2, 3 or even more commits that I’m sitting on? Do I deploy them one at a time until they’re either all successful or one fails?

How has anyone else practising dealt with these issues?

EC2 Spot instances

Posted on December 14, 2009

The thing I really like about Amazon’s cloud stuff is they’re constantly undermining themselves with new innovations – spot instances are another great example. Taking the utility metaphor a step further you can now rent their services when nobody else is for cheaper, like buying electricity at night.

A lot of the tasks I’m envisioning for Sproozi aren’t really time dependent. While it’s important to show you a page in a timely manner, crawl and index a brand new website you add quickly and basically be interactive there is also a lot I have to do in the background. The huge and growing list of URLs people add all need to be re-crawled and re-indexed regularly is just one of many examples of processing vast amounts of data. These tasks are always running, always in the background.

Spot instances are a perfect fit – I can bid the price I want on extra capacity spin up some extra instances to join the cluster when they’re cheap. Over the next few weeks I’ll probably try to add some spot instances to my Hadoop dev cluster and see what happens.

Downloading maven dependency source jars

Posted on November 3, 2009

I’ve been working on a new project that I’m planning to open source real soon – stay tuned.

When I’m implementing interfaces in a dependent package using Idea/Maven I want to tick the “copy javadoc” button to at least have the documentation from the intereface. The issue of course is that I don’t have the sources.

Run the following command:

mvn dependency:sources

Maven will download any sources it can find for in remote repositories for your dependencies and Idea finds them like magic. So now not only can you copy javadoc, you can also click the line number in the stack trace and get something meaningful – not “compiled code”.

Integration testing in maven – With Maven, Cargo, httpunit and Selenium

Posted on September 22, 2008

I’ve been trying to figure this out for months, and thought it should have been simple. All I wanted to do was write a set of unit tests to test my java code, have them run whenever I hit test. Next I wanted to have a set of HTTP Unit, Selenium or similar tests run to test that the actual application is working when it’s built and deployed to a container using the cargo plugin.

I didn’t really want to have a separate project to do this because it seemed like a massive pain in the ass to maintain the whole thing. I didn’t see any reason why it shouldn’t be easy with Maven either – it does have a phase for integration-tests already.

My first try looked good to me:

...
<build>
    ...
    <plugins>
        ...
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <excludes>
                    <exclude>**/integration/**</exclude>
                </excludes>
            </configuration>
            <executions>
                <execution>
                    <id>integrationtest</id>
                    <phase>integration-test</phase>
                    <goals>
                        <goal>test</goal>
                    </goals>
                    <configuration>
                        <excludes>
                            <exclude>none</exclude>
                        </excludes>
                        <includes>
                            <include>**/integration/**/*Test*.java</include>
                            <include>**/integration/**/*Test.java</include>
                            <include>**/integration/**/*TestCase.java</include>
                        </includes>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        ...
    </plugins>
    ...
</build>
...

I tried to run the tests, but unfortunately when it gets to the second round of tests – which should run my integration tests – it runs the same sets of as it ran the first time. I tried adding the combine.children="append" attribute to the includes, and excludes but that didn’t work either. Finally I came across a source file XppDom, part of plexus which maven uses. XppDom allowed the combine.children attribute as well as another I haven’t seen mentioned anywhere else childMergeOverride. I added that instead and it worked!

...
<build>
    ...
    <plugins>
        ...
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <excludes>
                    <exclude>**/integration/**</exclude>
                </excludes>
            </configuration>
            <executions>
                <execution>
                    <id>integrationtest</id>
                    <phase>integration-test</phase>
                    <goals>
                        <goal>test</goal>
                    </goals>
                    <configuration>
                        <excludes childMergeOverride="true">
                            <exclude>none</exclude>
                        </excludes>
                        <includes childMergeOverride="true">
                            <include>**/integration/**/*Test*.java</include>
                            <include>**/integration/**/*Test.java</include>
                            <include>**/integration/**/*TestCase.java</include>
                        </includes>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        ...
    </plugins>
    ...
</build>
...

Setting it up for Cargo

I’ve done this on a few projects, the first was a project that built a series of taglibs which are used across a number of applications. The release for the project is a jar so I have a separate test web-app structure in ${baseDir}/src/test/web-app. The code for generating this war looks like this:

...
<build>
    ...
    <plugins>
        ...
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-war-plugin</artifactId>
            <executions>
                <execution>
                    <id>generate-test-war</id>
                    <phase>pre-integration-test</phase>
                    <goals>
                        <goal>war</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <warSourceDirectory>${basedir}/src/test/webapp</warSourceDirectory>
                <warName>${project.artifactId}-test</warName>
                <webappDirectory>${basedir}/target/${project.artifactId}-test</webappDirectory>
                <primaryArtifact>false</primaryArtifact>
            </configuration>
        </plugin>
        ...
    </plugins>
    ...
</build>
...

During the pre-integration-test phase the above simple generates a test war. If you’re project is a web-app and it already generates a war you can skip the above as one will be generated for you already.

The next step for was then to get Cargo to deploy the application:

...
<build>
    ...
    <plugins>
        ...
        <plugin>
            <groupId>org.codehaus.cargo</groupId>
            <artifactId>cargo-maven2-plugin</artifactId>
            <configuration>
                <wait>false</wait>
                <container>
                    <containerId>tomcat5x</containerId>
                    <zipUrlInstaller>
                        <url>${integrationtests.tomcatURL}</url>
                        <installDir>${installDir}</installDir>
                    </zipUrlInstaller>
                    <output>
                        ${project.build.directory}/tomcat5x.log
                    </output>
                    <log>${project.build.directory}/cargo.log</log>
                </container>
                <configuration>
                    <home>
                        ${project.build.directory}/tomcat5x/container
                    </home>
                    <properties>
                        <cargo.logging>high</cargo.logging>
                        <cargo.servlet.port>8080</cargo.servlet.port>
                    </properties>
                </configuration>
            </configuration>
            <executions>
                <execution>
                    <id>start-container</id>
                    <phase>pre-integration-test</phase>
                    <goals>
                        <goal>start</goal>
                        <goal>deploy</goal>
                    </goals>
                    <configuration>
                        <wait>false</wait>
                        <deployer>
                            <deployables>
                                <deployable>
                                    <location>${basedir}/target/${project.artifactId}-test.war</location>
                                    <type>war</type>
                                    <pingURL>http://localhost:8080/${project.artifactId}-test/index.html</pingURL>
                                    <pingTimeout>300000</pingTimeout>
                                    <properties>
                                        <context>${project.artifactId}-test</context>
                                    </properties>
                                </deployable>
                            </deployables>
                        </deployer>
                    </configuration>
                </execution>
                <execution>
                    <id>stop-container</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>stop</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        ...
    </plugins>
    ...
</build>
...

Most of this is pretty self explanatory, I try to use parameters to configure as much as I can and try to maintain it in higher level POMs wherever appropriate. This is especially true for the tomcat URL which changes on a regular basis. The seem to post new releases and remove the older ones, and it can be a chore to keep up in multiple projects. You may want to look at the cargo plugin documentation, you can ignore some of what I’m doing above if you’re project’s default artifact (the one package creates) is a web-app. 

So there you have it, eventually I got there after a lot of digging. I’ve been pretty brief in my explanations and have assumed a fairly good understanding of maven. if you have any questions by all means leave a comment and I’ll do my best to make it clearer.

Related articles by Zemanta

• Testing Is Not Technically Hard, It Is Hard Because It Requires Clear Thought and Understanding
• Why do we (not) write tests