I’ve been faced with a bit of a problem lately, how to test the quality of results being produced from a changing dataset? It’s easy to write unit tests for components of an application, and to make sure they’re producing the expected results, but how do you test end to end?

Andraz from Zemanta asking the same question, How do you test a complex system that is trying to mimic being smart, last year.

when you have new content in the system, you get completely new related stories and you have to go back and have a human judge them. There is expansion of the evaluation data - as you add new tests you generally can’t send them through previous versions of your algorithms, since that would be prohibitely expansive. And there is statistics that hardly gives you overview over what exactly your changes caused, just few final numbers. And then there is the problem of pipelining the processing. Even if you improve the first stage, end results might be worse, since you’ve already adapted the second stage to previous first one. So you need to actually evaluate each part of the system in isolation and then together.

At the end you actually find out that you spend disproportional amount of time evaluating even the smallest changes. So you are in danger to just skip that evaluation which naturally you shouldn’t.

The fundamental problem you run up against is that the index is constantly changing, and it’s meant to change. So it’s hard to automatically test the output without a clear idea of what is going in. It’s also difficult to get an accurate picture of how small changes in code affect the general results if you’re just using a testing index with a small dataset.

One way to go about it is to gauge result quality based on measuring user interaction. Basically there are things users do when they get results they’re expecting and things they do when they haven’t found what they were looking for. So if you can get measure how they’re reacting, you can get an idea of quality.

At the moment I’m a lone developer putting all the data in the index, and I have a good idea of what I should be seeing out if it’s actually working. In the next while though we’re going to be rolling out to a few more internal beta users as we get a prototype system developed and we’re not going to be in control of the inputs or output anymore. So soon enough we’re going to actually be faced with trying to measure the quality of the results we’re giving users in a dynamic system - expect to hear much more about this as we go on.

Related articles by Zemanta

• Quality assurance
• Why Developers Balk at Unit Testing
• 12 Unit Testing Tips for Software Engineers

It’s very, very hard for us to admit that we’re wrong, let alone that we have no clue what we’re doing. The first step of course is being able to admit that you’ve got a problem and before you can admit you have a problem you need to be able to recognise that you’ve got one.

I came to the shocking realisation the other day, that I was guilty of not really know what I was doing. I came across a subtle bug in some of my code; my unit tests were passing but in production things weren’t working as I expected. I dug deeper and got a vague idea of what was happening, then I engaged in shotgun programing.

I knew I was doing it too. I had no idea what I was doing, and I knew it. I just started hacking away at my code, solving one problem and introducing another. Basically I knew where the problem was and as if it were some small animal rustling in the bushes nearby I just blasted away its general direction and just hoped I hit it, whatever it was.

I eventually solved the problem, and eventually understood what was going on; but it was touch and go there for a while. The problem wasn’t so much my trial and error approach, that did eventually solve the problem. It was more that I had no idea why I was trying the solutions I was trying, more specifically why I thought they were plausible I was just pointing in the general direction of the problem and letting loose.

Related articles by Zemanta

• 12 Unit Testing Tips for Software Engineers
• A unit testing success story
• How Maatkit benefits from test-driven development
• Quality assurance

I’m starting to get lots of messages about people following me, they’ve got one post and it’s a link to “check them out” lots of new pics added.

I’ve been blocking them as they show up, because I only have a few followers. I wonder how the popular people, who must be getting thousands of these cope?

Fucking spam, it’s got to try to ruin every good thing.

I added Zemanta to the blog today at lunch time and so far I think it’s great. I’ve gone through and quickly added a links to a few posts, I’m not going to go back and do many but I’m going to use it going forward.

One of the most common pieces of advice you hear about blogging and getting noticed is to link. With Zemanta linking far and wide is a few simple clicks away.

I have no vested interest in Zemanta - I know very little about them. I really just like what it does.

Related articles by Zemanta

• Zemanta: Unleash your blogging skills
• Zemanta is a small bloggers best friend
• Zemanta: A blessing in disguise for bloggers?

First we started to explore, then quickly we needed to figure out where we were and we’ve been building increasing numbers ever more sophisticated maps, guide books and other sources of location based information ever since. The extension of this location based information to mobile Internet enabled devices is the obvious next step and potentially huge market. Just think of all the location based data people pay for:

• Mapping probably the biggest market segment, and the first which we’ll see device convergence. Look to see the traditional GPS hardware/software become applications for the current and next generation devices.
• Reviews A common feature of newspapers, local and region
• Business Directories again, think of the big yellow book of business for every area. It’s one of the most common things to look for in an area.
• Travel Guides More than just reviews, travel guides also provide us with areas of interest and all of that is very useful in an up to date online service

All of this is pretty obvious and most of it exists now. The problem up to now has been the walled garden approach taken by the mobile phone companies. They had the ability to determine position, but wanted to charge for it. Unfortunately for us consumers, charging for that data has meant that for the most part services would need to turn a profit for every single request, from the start or they’d be very expensive to develop and very quickly.

Mobile data has recently become very cheap and new devices with integrated ability to determine their location is having a profound impact on the mobile landscape. They’re allowing us to build services which instead of answering the question: where is my nearest X, will let us explore the area around us. Mapping the wealth of knowledge on the internet to physical locations and allowing us to actually input data when we’re most likely to want to - when we’re actually there.

From experience, when I’m out there the data I want most is about out there.

Related articles by Zemanta

• More on the state of the mobile web
• Travel sites wake up: Nokia to offer €8 mobile Lonely Planet guides

I’ve been a bit lax with the posting again, but I’m trying. Continuing with the theme where I left off last time, starting companies in the downturn (likely to be a recession), the last time we talked the worlds markets were in turmoil, the banks had stopped lending money to us and to each other. Things have continued and are even worse two weeks on.

Now some would say is the time to tighten belts and budgets, for prudence sake. At least that’s what everyone else is doing, and what we’re doing at home to be fair. It’s cheaper shopping, soups and casseroles for the winter. Many businesses are reacting the same, I’ve heard it from them and so have you. Even businesses that have nothing to do with the finacial or property markets are tightening their budgets, and feeling or causing the pinch to be felt.

Is that good news for the nimble small business? Like every other news, yes. Small businesses face unique challenges their larger competitors don’t but they have one major advantage - they can move and react faster.

Don’t get me wrong, I’m not suggesting anyone should go on a spending spree. But neither am I suggesting that now is a time to be completely miserly either. What I am suggesting is that while your larger competitors, or even competitors of similar sizes are worrying about the future and cutting back their development efforts; or cutting back their staff, salaries and benefits. It’s a great time for you to make the leap to the next level, and come out ahead when this is all over- because it will eventually end, one way or another.

Related articles by Zemanta

• How To Profit From The Recession - Small Business Marketing Tips
• Now for the bad news: less investment money for startups
• 5 People and Things That Will Recession-Proof Your Small Business

Location based services, another area where there is certainly a case to be made for someone to make it suck less and probably a number of successful application to build on it.

Why it sucks

Location services suck because it’s inherently hard to map the collection of information on the Internet to physical locations.

Most of it just wont map because it’s not about something physical.
It’s obvious, but true - most of the Internet isn’t about a place, or a thing that is likely to stay in one place. This post for example, or this blog’s homepage - neither are about anything specific. My house isn’t

It’s hard to map because language is imprecise, physical location isn’t.
When you’re looking for a nearby restaurant a blog post with a review of a restaurant in London probably won’t give you an address and probably won’t even link to a site that will. It’s become the job of search to try and link the post, a very difficult job - was the post about London, England; London in Ontario or one of the dozens of other Londons around the world. At best the search engine might be able to link the name with another URL that contains an address - but there are no guarantees the name is unique or the address is right.

Sending letters for physical addresses with an activation code is one approach Google has tried, I’m not sure how successful it’s been though - I haven’t noticed locations search has stopped sucking, so I assume it’s not the silver bullet.

Sucking less

How to go about fixing it? If I knew the answer I’d be coding the solution right now and wouldn’t be sitting her writing and thinking about it.

The rise of location aware internet devices
It’s inevitable, the rise of the iPhone, Andriod phones and other location away devices with Internet connections mean people are going to want to consume location based resources. Though they’re not the first phones on the market to do any of this, they’re game changing because they’re the first platforms that make it easy to consume and more importantly produce location based content.

The other key ingredient these platforms offer is a browser, not just a cut down browser - a real browser, with tabs and javascript and all the trimmings!

Like any good problem, this one can be solved with metadata!
We’ve got the consumers with their million plus iPhones, we’ve got the producers who all want to start geo tagging. The problem is how do we link the two and how do we make it easy? We’ve got a few standards for location tagging document, but no clear forerunner.

Suddenly it doesn’t suck anymore

Services for producers and users to announce and tie information to physical locations is going to be the catalyst that really makes local search and local information portals of all kinds take off. It won’t be long until someone comes along with a killer solution.

I have a real problem with bookmarks. Actually I have a few problems with bookmarks: There are too many, they’re a mess, they don’t make any sense and I can’t search them. Basically they’re a good idea, really badly implemented.

I have a list of bookmarks, I try to keep them organised but there are hundreds every few months I have to go through and throw some away. The clutter in my bookmarks also leads to bad habits. If it’s something I want to read later, I drag a link to my desktop or I put it in a read later folder. These quickly grow out of hand. Then I start getting protective and avoid saving links, dooming myself to searching for them again.

Lately, using Google Reader I’ve come up against another problem - I’m building a whole new set of bookmarks in the form of starred items, except these are completely separate from the rest of my browsing but I keep doing it because it’s easy and I can search them.

When it comes down to it my problem is basically simple, I can’t ever find anything again! It’s even worse if it’s something I’ve found via searches because it might be weeks or months later I try to find the page again and it’s likely that the Google’s index will have changed in that time. So my searching for something I’ve seen gets harder.

Bookmarking the page doesn’t usually make it easier, there is no frame of reference. Unless I’ve set the title to something useful a lot of the time I find it hard to find things again because lot of the time with hundreds to go through, or I’ve cleaned it out and it doesn’t exist anymore anyway.

I’ve tried delicious and other similar tools, and I like them, tagging my links makes them infinitely more searchable. But it doesn’t go quite far enough - it still has a feeling of permanence and exclusivity to it when I tag something and I don’t always want to share everything I tag.

The other problem with social bookmarking and tagging sites is they provide a tag for me, but no context. So days, weeks or months later unless I’ve tagged the site well I may or may not be able to find it. I thought I saw a site that indexed the links from delicious, but I seem to have lost it and can’t find it again(You see what I mean!).

What I really want to see is a site that lets me star or bookmark a page or RSS feed and I want it to index my little corner of the web.

Again, I’m here apologising for letting things slip a little. I’ve been busy and posting has unfortunately suffered for it. I’ve been working hard trying to get things moving for goroam, and it’s been time consuming. I basically spend the last half of last year bogged down with a project and had no time at all to spend on it, so for the last half of this year the focus is really to turn it around, launch a few things and start bringing some revenue in.

Our biggest project at the moment is running an online marketing campaign for Mediqol a medical device consultancy. Our lofty goal is to show up in the first position for the search keywords medical device, short term we’re just interested in optimising the adwords campaigns, and increasing organic traffic for the keywords we know work.

Then there is Citrus - Work is progressing, but we might be looking at an end of the year launch for it. My gut feeling is that we won’t be anything approaching a launch before the end of November.

In the meantime we’re looking into a couple of smaller launches. I have a couple of ideas for tools and products that tie into the Citrus roadmap that we can launch.

I’m going to spend some time this afternoon and weekend writing some posts to get on top of it again.