I’ve been faced with a bit of a problem lately, how to test the quality of results being produced from a changing dataset? It’s easy to write unit tests for components of an application, and to make sure they’re producing the expected results, but how do you test end to end?

Andraz from Zemanta asking the same question, How do you test a complex system that is trying to mimic being smart, last year.

when you have new content in the system, you get completely new related stories and you have to go back and have a human judge them. There is expansion of the evaluation data - as you add new tests you generally can’t send them through previous versions of your algorithms, since that would be prohibitely expansive. And there is statistics that hardly gives you overview over what exactly your changes caused, just few final numbers. And then there is the problem of pipelining the processing. Even if you improve the first stage, end results might be worse, since you’ve already adapted the second stage to previous first one. So you need to actually evaluate each part of the system in isolation and then together.

At the end you actually find out that you spend disproportional amount of time evaluating even the smallest changes. So you are in danger to just skip that evaluation which naturally you shouldn’t.

The fundamental problem you run up against is that the index is constantly changing, and it’s meant to change. So it’s hard to automatically test the output without a clear idea of what is going in. It’s also difficult to get an accurate picture of how small changes in code affect the general results if you’re just using a testing index with a small dataset.

One way to go about it is to gauge result quality based on measuring user interaction. Basically there are things users do when they get results they’re expecting and things they do when they haven’t found what they were looking for. So if you can get measure how they’re reacting, you can get an idea of quality.

At the moment I’m a lone developer putting all the data in the index, and I have a good idea of what I should be seeing out if it’s actually working. In the next while though we’re going to be rolling out to a few more internal beta users as we get a prototype system developed and we’re not going to be in control of the inputs or output anymore. So soon enough we’re going to actually be faced with trying to measure the quality of the results we’re giving users in a dynamic system - expect to hear much more about this as we go on.

Related articles by Zemanta

• Quality assurance
• Why Developers Balk at Unit Testing
• 12 Unit Testing Tips for Software Engineers

It’s very, very hard for us to admit that we’re wrong, let alone that we have no clue what we’re doing. The first step of course is being able to admit that you’ve got a problem and before you can admit you have a problem you need to be able to recognise that you’ve got one.

I came to the shocking realisation the other day, that I was guilty of not really know what I was doing. I came across a subtle bug in some of my code; my unit tests were passing but in production things weren’t working as I expected. I dug deeper and got a vague idea of what was happening, then I engaged in shotgun programing.

I knew I was doing it too. I had no idea what I was doing, and I knew it. I just started hacking away at my code, solving one problem and introducing another. Basically I knew where the problem was and as if it were some small animal rustling in the bushes nearby I just blasted away its general direction and just hoped I hit it, whatever it was.

I eventually solved the problem, and eventually understood what was going on; but it was touch and go there for a while. The problem wasn’t so much my trial and error approach, that did eventually solve the problem. It was more that I had no idea why I was trying the solutions I was trying, more specifically why I thought they were plausible I was just pointing in the general direction of the problem and letting loose.

Related articles by Zemanta

• 12 Unit Testing Tips for Software Engineers
• A unit testing success story
• How Maatkit benefits from test-driven development
• Quality assurance

I’ve been working form home for quite some time now, in the beginning I lived alone and I worked from home a few days a week. Working the balance in an office with real people. I found I was far more productive when I was in an home than in an office, mostly let’s face it, because people had to pick up the phone to interrupt me.

Recently things have changed, my partner had a little baby girl and we’ve moved. The house is bigger and I now have a room which I use almost exclusively as my office, which is a good thing because previously I’ve been working from a series of one bedroom flats and that just wouldn’t work now.

Headphones and a door that closes

I have both and think they’re infinitely important. My daughter is very young but between myself and Becca there is the policy that if the door is closed I’m busy. Unless someone is bleeding to death I don’t want to know- we treat the closed door much like we’d treat it if I was working in an office half an hours commute away, if it wouldn’t be important enough to call me away from work in the car, then it’s not important enough to call me away from work here.

Sometimes, even with the door closed there can be a fair bit of noise and things to distract me, I have a good pair of headphones which acoustically seal me off from the rest of the world.

Working Hours

I try to keep my working hours 9-5, just as I would with any job. Mostly because that’s when everyone else is working and if I need to talk to someone, or if they need to talk to me they’re most likely to try. I’m pretty flexible about it though and unless I need to deliver something I’ll sometimes take a few hours off during the day to do the dishes, go do the recycling (everyone else goes on the weekend, but it’s almost always quiet weekday mornings) or just spend some time with my daughter to give my partner a break. I make the time up evenings and weekends, but for the most part you’ll find me adhering to the same schedule as the rest of the world - I just get to stay in bed while the rest of you commute.

Works for me

Working from home works well for me, it takes discipline but to be honest it’s not as hard as some people would like to make you think. Also for those bosses out there that don’t trust staff to work from home - if you can’t trust them not to bunk off at home, you probably can’t trust them not to bunk off in the office either.

Related articles by Zemanta

• Working from home - the good, the bad and the ugly
• Walking the Line When You Work from Home
• If You’re a Slacker and You Know It, Clap Your Hands (Office vs. Telecommuting)…

I’ve seen a lot of posts recently about how to go about hiring hackers. Well not just recently, it seems to be common thing people ask, because it’s one of the hardest things to do. You’re using an interview, or even a series of interviews to try your best to determine if the person is going to be up to the task.

The problem is of course, you’re really just determining if the person interviews well, or if they’re the graduates that they’ve got good marks. The real problem is of course neither of the two matter because neither of them have much if any bearing on how good a coder a candidate is going to be.

The problem is exacerbated in a startup because unlike some of the more successful bigger players you really can’t afford to take people on a trial basis for 6 months. What can a startup do?

There are a lot of universities that run placement programs for their students, programs allowing them to get real industry work experience. I’ve also worked in companies that made it a policy to hire students for internships then offer the best jobs once they’ve graduated. It was a great way to get talent.

What if you need to hire to fill a more senior role, or you’re not hiring at the right time of year? Student internships usually start once at the same time every year so unless you’re looking for fill a role then, you’re shit out of luck.

I’ve toyed with the idea of a ‘hacker camp’, getting all the candidates together, and paying them to work as consultants on a project - then hiring the best into permanent positions. All the candidates need to be free to work on the project, and a lot of people would be pretty scared to leave a job to take part and some jobs would prevent them from working on the project, so running it over a weekend might be a good idea, but it limits the scope of your project. You’d also still need to have interviews, to make sure you were getting candidates who were really interested in the job, and also to weed out any candidates you knew were totally unsuitable.

I think it’s important that a ‘hacker camp’ should produce something at the end of the weekend and you should get as much of your existing staff involved as possible. To make the cost easier to swallow you could even go so far as making the deliverable a new feature or part of your existing product but that’s not as important as actually delivering something, because that will be the bottom line when it comes to integrating them into your team.

Related articles by Zemanta

• Learning from a summer intern program
• Fuel business growth employ an intern/apprentice
• Companies get more savvy with internships to find next-gen IT workers

I’m starting to get lots of messages about people following me, they’ve got one post and it’s a link to “check them out” lots of new pics added.

I’ve been blocking them as they show up, because I only have a few followers. I wonder how the popular people, who must be getting thousands of these cope?

Fucking spam, it’s got to try to ruin every good thing.

As I’ve announced here, we’ve recently taken the difficult decision to put development of Citrus in hold, indefinitely (let’s be realistic, probably forever). I’m now wondering what if anything we should announce about our new project.

Is openness a strength or a weakness? Is there value to the community in publicly documenting our design decisions, and how we’ve come to them? Is it something our competitors could and would use against us? Is there anyway to mitigate the effects?

I like what Jeff Atwood and Joel Spolsky have done with their series of podcasts on the Stack Overflow blog where they’ve released their design meetings as a series of podcasts and given us all an insight into how they’ve come to some of the decisions they’ve taken. But let’s face it I’m neither Jeff or Joel and I don’t get anywhere near their traffic. So the first obvious question I’m asking myself is if it’s even worth the effort of writing posts, let alone recording and releasing podcasts.

Next is the question about maintaining a competitive advantage, given the fact that our team is so small, just the two of us at the moment; I’m only working about half time on the project and Simon, who isn’t a coder is working even less. So giving someone a guide to competing with us could be downright foolish. Then again, nobody is really listening are they? So does it even matter?

The other question, which I think is equally important as the project grows - hopefully into a product is what if any level of openness is appropriate post launch? Will an open discussion of ongoing design decisions as they’re being taken before, or even as they’re released be of any benefit to the community? Communities can be some of the biggest supporters and can be some of your most passionate users - sometimes these passions can work both ways? What level of support ongoing will a public discussion of the decisions you’re taking, before users have all the prototyped interfaces to play with, be required? Is there an ongoing effort you’ll need to make to maintain the level of trust and confidence of the community?

I think I’ve talked myself out of a podcast of our internal discussions, for now - but I’m still keen to use community interaction to help us develop a product that meets the needs of the people who are interested in using it and to drum up support and interest in the project. I’ll no doubt return to this topic in the coming weeks as I develop a clearer idea of how I think we should accomplish that.

If any of the few of you out there listening have any questions or comments, please let me know your thoughts.

Related articles by Zemanta

• Does Good Tech Need PR?
• stackoverflow.com
• PR-less launch kicks off a stack overflow of praise

I added Zemanta to the blog today at lunch time and so far I think it’s great. I’ve gone through and quickly added a links to a few posts, I’m not going to go back and do many but I’m going to use it going forward.

One of the most common pieces of advice you hear about blogging and getting noticed is to link. With Zemanta linking far and wide is a few simple clicks away.

I have no vested interest in Zemanta - I know very little about them. I really just like what it does.

Related articles by Zemanta

• Zemanta: Unleash your blogging skills
• Zemanta is a small bloggers best friend
• Zemanta: A blessing in disguise for bloggers?

First we started to explore, then quickly we needed to figure out where we were and we’ve been building increasing numbers ever more sophisticated maps, guide books and other sources of location based information ever since. The extension of this location based information to mobile Internet enabled devices is the obvious next step and potentially huge market. Just think of all the location based data people pay for:

• Mapping probably the biggest market segment, and the first which we’ll see device convergence. Look to see the traditional GPS hardware/software become applications for the current and next generation devices.
• Reviews A common feature of newspapers, local and region
• Business Directories again, think of the big yellow book of business for every area. It’s one of the most common things to look for in an area.
• Travel Guides More than just reviews, travel guides also provide us with areas of interest and all of that is very useful in an up to date online service

All of this is pretty obvious and most of it exists now. The problem up to now has been the walled garden approach taken by the mobile phone companies. They had the ability to determine position, but wanted to charge for it. Unfortunately for us consumers, charging for that data has meant that for the most part services would need to turn a profit for every single request, from the start or they’d be very expensive to develop and very quickly.

Mobile data has recently become very cheap and new devices with integrated ability to determine their location is having a profound impact on the mobile landscape. They’re allowing us to build services which instead of answering the question: where is my nearest X, will let us explore the area around us. Mapping the wealth of knowledge on the internet to physical locations and allowing us to actually input data when we’re most likely to want to - when we’re actually there.

From experience, when I’m out there the data I want most is about out there.

Related articles by Zemanta

• More on the state of the mobile web
• Travel sites wake up: Nokia to offer €8 mobile Lonely Planet guides

I’ve been a bit lax with the posting again, but I’m trying. Continuing with the theme where I left off last time, starting companies in the downturn (likely to be a recession), the last time we talked the worlds markets were in turmoil, the banks had stopped lending money to us and to each other. Things have continued and are even worse two weeks on.

Now some would say is the time to tighten belts and budgets, for prudence sake. At least that’s what everyone else is doing, and what we’re doing at home to be fair. It’s cheaper shopping, soups and casseroles for the winter. Many businesses are reacting the same, I’ve heard it from them and so have you. Even businesses that have nothing to do with the finacial or property markets are tightening their budgets, and feeling or causing the pinch to be felt.

Is that good news for the nimble small business? Like every other news, yes. Small businesses face unique challenges their larger competitors don’t but they have one major advantage - they can move and react faster.

Don’t get me wrong, I’m not suggesting anyone should go on a spending spree. But neither am I suggesting that now is a time to be completely miserly either. What I am suggesting is that while your larger competitors, or even competitors of similar sizes are worrying about the future and cutting back their development efforts; or cutting back their staff, salaries and benefits. It’s a great time for you to make the leap to the next level, and come out ahead when this is all over- because it will eventually end, one way or another.

Related articles by Zemanta

• How To Profit From The Recession - Small Business Marketing Tips
• Now for the bad news: less investment money for startups
• 5 People and Things That Will Recession-Proof Your Small Business