The open source tool balance is an essential part of the service infrastructure here at Last.fm. Multiple instances of balance are running on each and every web server node, on the various production back end servers, and also on our development machines. So at any given time there are probably thousands of instances running simultaneously on our machines.

What does it do?

balance is a so-called load balancer. It is generally used as a proxy to distribute a large number of incoming requests to a group of servers. In other words it is responsible for balancing the load between all the servers in a group. Quite often, load balancers are dedicated hardware products. However, balance is a software load balancer, which means it can just run as an additional program on any server.

In addition to load balancing, balance also supports a scheme called failover. This means you can define a second group of servers and balance will route requests to the second group if all servers in the first group fail. This failover scheme is used by most of our backend services at Last.fm. We usually have a main server and a backup server that kicks in once the main server fails.

End of story?

Certainly not! There are some subtleties in the use of balance that have given us headaches in the past. By far the biggest problem is that there are cases when failover just doesn’t work right in our environment. So here’s a real example…

One day we had to take down the main server for one of our backend services to replace a hard drive. The backup server was running fine and we relied on balance to take care of routing all requests through to the backup box. Unfortunately, shortly after the main server went down, we noticed that most requests to the service failed.

What had happened? balance has a configurable connect timeout, i.e. it tries to connect to a service and then waits for a certain amount of time until it figures out that it can’t connect. If the server machine is running, the connect will fail almost instantly if the service itself is unavailable. However, if the server is down, it’ll wait until the connect timeout has elapsed. So in our case, balance was trying to connect to the main server (which was down) and then waiting for 5 seconds before attempting to connect to the backup server. In the meantime, the client had already given up (it was using a much smaller timeout). balance would only notice that the client had given up by the time it had established the connection to the backup server. The next time the client tried to connect, the same thing would happen all over again.

But someone else would certainly have had the same problem before?

I’m quite sure of that. And I guess that’s what caused the autodisable feature to be added to balance. When this feature is being used, balance will automatically disable servers that it fails to connect to. The downside, though, is that there’s no way to automatically enable servers again. And manually enabling them isn’t really an option given the number of instances of balance we’re running and given that it could cause all servers to be permanently disabled in case of, for example, temporary network failure.

So what now?

We had to face the fact that in theory we had a really nice redundancy scheme, but it could fail quite miserably in practice. So I began to look around for alternatives to balance and found a couple of other open source load balancers. Sadly, all of them had either been abandoned by their authors, failed to build out of the box or just didn’t fulfill our requirements.

balance was actually just what we needed. The only thing it was missing was support for monitoring all back end connections and dynamically disabling and enabling them as they fail or pass the monitoring checks.

So eventually I started looking into adding exactly that functionality to balance.

balance.fm

Implementing monitoring for balance was relatively straightforward, even though it made me aware of how much I had gotten used to developing software in C++. With balance being written in pure C, I was really missing exception handling and the C++ standard library.

The amount of code changes was massive considering the rather small code base of balance. As of now, more than a thousand lines of code have changed and another thousand lines have been added. So we decided to fork the original project and rebrand it as balance.fm.

It took about a week to refactor the existing code and finally add the monitoring feature. Along the way of adding monitoring, quite a few bugs have been fixed as well (for details, just have a look at the commit log if you’re interested) and I hope these fixes make up for all the bugs that I’ve undoubtedly introduced by adding loads of new code.

The balance.fm code has since been reviewed by the MIR team here at Last.fm and is available from github.com/lastfm/balance.fm.

If you have an application for balance.fm, please give it a try and let us know what you think and like or dislike about it!

Last weekend, Russ Hall and I travelled to Berlin for the latest in the series of Music Hack Day events. Having been to previous ones in Stockholm and London, I was looking forward to the usual mix of creativity and collaboration that these events are famous for.

For those that don’t know, Music Hack Day is a chance for programmers, designers, artists, etc, to get together and create new and exciting music hacks based on the latest APIs from top music tech companies (or they can just use a soldering iron and some knitting needles). It’s always amazing to see what a room full of talented people can come up with in just 24 hours and this one was no different.

APIs

First up at a Music Hack Day are the API presentations. This is where music tech companies pitch their APIs and it’s also an opportunity to announce new features for devs to get stuck into straight away. We were there to present our API and the addition of our beta realtime API.

Hacking

After this, everyone fuelled up on Club-Mate and set about hacking. I played with a few side projects while Russ joined up with Tim Bormans of Soundcloud and Jens Nikolaus (designer of the much sought after Music Hack Day Berlin tote bag with Kristina Schneider) to create Sleev.in, an album obsession sharing site.

Here’s Tim and Russ mid-hack.

The amount of sleep you get over the Saturday night depends on how well your hack goes and/or how ambitious you were and at around 2pm on the Sunday, around 24 hours after you’d have started, it’s time to put down your laptop and present what you’ve done.

Hack demos

The hack demos are always fun to attend and see what everyone else has been making.

Some notable Last.fm related hacks were Lastcred.fm which would, to make your Last.fm profile look trendier, scrobble several random tracks for you based on a tag or artist of your choice (not something we really approve of, but it amused us nonetheless), Tractor which pulled in Last.fm data to help with displaying artist info for mentions of an artist on a web page, and RealTimeSentiTweetGagasm which used our realtime API.

Other favourites of mine were Heavy Shoes which used an Xbox Kinect sensor to detect foot stomps and then play drum noises (something I want to be doing at home very soon), and Eigendrums which also triggered drum sounds, but this time by detecting the sound of you clicking, slapping your leg, and thumping your chest. Both crowd pleasing impressive demos.

You can find the complete list of hacks here.

If any of this interested you then why not think about attending a Music Hack Day! The next one is in Barcelona in a few weeks, but they crop up fairly regularly.

And finally a big thanks to Roel van der Ven and Johan Uhle of Soundcloud for organising such a fantastic event.

Hey, look! It’s a pause button! I know right?!

Pause is a feature that users and staff alike have been requesting for quite some time, so one dreary Thursday evening I decided ‘Enough is enough! We NEED pause and we need it now!’

That’s almost true anyway. It’s been a big challenge to implement, and we’ve spent a little while testing this feature — writing some supporting infrastructure and making sure the feature works well across our different players.

That’s right, the Android and iPhone client also now come with pause too. We’re working on a new version of the desktop client now, and that will come with pause too.

I’ve been getting a ton of use out of it and I hope you guys do too! The specs are now available for partner players (like XBox and Windows Phone 7) to support the feature, and we’ll be updating the FAQ as and when they’ve implemented it.

Drum Roll – It’s time for the main feature!

But… but but but, we’re also releasing something new and exciting into the wild today.

Whenever you tune into a radio station on Last.fm we build a playlist of tracks based on various criteria: for Recommended Radio we’re looking at music that you might like based on what you’ve been listening to recently; for Friends Radio we’re looking at what your friends have listened to recently… and so on and so forth.

Up until now we haven’t surfaced why a particular song is being played to you, but that’s about to change with a little feature that puts some info text in the top left of the player.

When you’re listening to Similar Artist Radio or your Library Radio we’ll show you some information about the track being played (the song selection is kinda obvious — it’s in your library, or it’s similar to the artist you typed in).

Things start to get a little more interesting when you’re tuned to Friends, Neighbours, Recommended or Mix radio. You’ll see information about which artists or users fed into the song selection. If you click the “more” link you’ll scroll down to where there’s a little more detailed information; maybe it’s a few of your friends or a few artists that inspired the selection.

(By the way, if you’re using the Festive cheer or Bah! Humbug! radio settings then you’ll get a reduced amount of information. If you want to experience the magic you’ll have to turn them off for now, sorry!)

Hope you enjoy them! Remember, you can always offer feedback about features like these on the forums, and if you want to join the team who made them just head to the Jobs page.

Hello people. I’m cms, and my job here at Last.fm is looking after the databases. Much of the time I’m involved with operational running of database servers, designing and optimising SQL queries, and scaling work on our relational database clusters. Every now and then though, I do get an opportunity to poke around in the Last.fm dataset and explore some of the interesting relations.

I recently re-discovered the seminal album ‘Spirit Of Eden’ by ‘Talk Talk’ (haven’t tried it? You really should, it’s magical), and I’d been giving it quite heavy rotation. This prompted a comment on my profile by one of our lovely users, who suggested making a playlist from artists whose names consisted of repeating word patterns. This idea appealed to me, but off the top of my head I could only come up with a paltry half-dozen candidates. Surely there were many, many more. If only there was some kind of database nearby I could query…

We keep our main catalogue data in a PostgreSQL database. PostgreSQL has a nice set of extended string operators, including quite comprehensive regular expressions support, which would be useful for an ad-hoc query like this.

Here’s what I came up with initially off the top of my head

select name from artist where name ~* E'^(\\w+\\M)\\s+\\y\\1$' ;

Using the case insensitive regular expression match operator ~* and matching against a string that begins with a sequence of word characters leading up to a word boundary, which I’m capturing as a group, then a sequence of whitespace, then the start of a word boundary followed by the original captured match.

This query worked really well at defining the pattern for repeating names. I was matching well over 10,000 distinct strings. The problem was that we store all the submitted data for artists, and this includes data from a broad range of unverifiable sources. I was getting lots of great artist names in my set, but many of them were bogus; typos, mis-taggings, spelling corrections, and that was just the obvious mistakes.

I needed to come up with a way of filtering the set further. My first iteration was to use track information. Incorrect artist attributions seemed unlikely to have relations over tracks in the catalogue, and I could extend my query relatively easily to take account of prolificness like so.

select count(1), a.name from artist a, track t where a.name ~* E'^(\\w+\\M)\\s+\\y\\1$' and t.artist = a.id group by 2 order by 1 desc;

This got me a shorter set of artists (8000 odd), with some ordering. I could see that recognisable artist names (hello Duran Duran !) were sorting towards the top. However, ordering by catalogue volume still wasn’t quite right. Ideally I needed some kind of popularity weighting. Unfortunately we don’t store any scrobble data in the PostgreSQL catalogue schemas.

However we do store scrobbles, alongside exported catalogue information in our Hadoop cluster. Although I have been known to write Java code in the past, I’m mildly allergic to it. Luckily for me we have a Hive interface to Hadoop. Hive offers an interactive query language over Hadoop that is closely modelled on SQL. The only stumbling block remaining was porting my regular expression over to use Java syntax.

Here’s what I ended up with as a hive query:

select meta_artist.name, overallplayreach_artist.reach from meta_artist join overallplayreach_artist on meta_artist.id = overallplayreach_artist.id where meta_artist.name RLIKE '^(.+?\\b)\\s+\\b\\1$' and meta_artist.correctid IS NULL and overallplayreach_artist.reach > 50 order by overallplayreach_artist.reach desc ;

Joining against some “playreach” data to give a weighting according to rough popularity. My original SQL query took 17 minutes to run, on a fairly beefy database server. The hive query took less than 100 seconds to return, running across the entire Hadoop cluster. Awesome.

Without any further ado, here’s the top 10 results, roughly ordered by artist popularity.

I’ve created a tag artistartist, and tagged some of the entries already.

The full list is available here. There might well still be some rough data in there, I haven’t particularly sanity checked it by eye.

If you too would like the chance to play with Last.fm’s vast amounts of data and join our team, check out our job openings.

Tweet

It feels like just the other week that I posted this on the Last.fm developer forum to get feedback and ideas on a new version of our scrobbling API that we were mulling over.

For those less technically-inclined, the scrobbling API defines how data gets transmitted to Last.fm every time you listen to a song. Scrobbles are incredibly important to us. They’re the building blocks of your music profile, and put together, they power basically everything that Last.fm knows about music.

The current API does a decent enough job, but we’ve had many developers complain about it being inconsistent with the other ways of accessing Last.fm data (via our web services) and its rather shoddy feedback on errors.

We also wanted to make the API more extensible so that we could define certain information which must always be submitted (like track and artist name) while allowing us to provide extra functionality in future via optional fields that wouldn’t break existing scrobblers.

600 ways to scrobble

Our scrobbling servers get a lot of traffic – at certain times of the day we have nearly 800 people telling us what they are listening to every second, and we are nearing our 40 billionth scrobble! There are also many different ways to scrobble the music you’re hearing, some developed by us (such as our official Last.fm, Android, and iPhone apps) as well as applications developed by third parties and music-loving geeks from all over the world.

Scrobbles-per-second monitor in the Last.fm operations room, powered by CactiView.

All told we have more than 600 scrobblers created by people other than us, covering popular online services like Spotify and The Hype Machine, hardware devices like the Onkyo TX-NR807 and the Logitech Squeezebox, as well as online storage services like Bitspace, extensions for browsers like Chrome (via Chrome Scrobbler), Opera (via Seesu) and Firefox (via FoxyTunes), and finally, for the real geeks, plugins for Gnome’s totem player and a promising-looking fork of Amarok called Clementine, to name just a few. A fair share of all existing scrobblers is listed on build.last.fm – browse around if you’re curious!

Preparing a new version of the scrobbling API

Given the heavy use of the current scrobbling API, releasing a new version of it is not something we take lightly – which is why it’s taken more than a year to get to where we are today. My post back in January 2009 generated pages of suggestions, plenty of e-mail conversations with developers and led to many hours of internal discussions and arguments involving nearly everyone in the company in some way.

We are finally able to unveil our first draft of what the new API might look like. Please bear in mind that this is not complete or final; we’re releasing it as a “request for comment” from the developer and user community. All the technical details can be read on our forum here and we’d like to keep detailed discussion there. We’ll be monitoring the post and taking feedback onboard.

Here’s a summary of just some of the highlights planned for the new API:

• The scrobbling API will become a fully-fledged member of the Last.fm Web Services under a new “Scrobble” package joining its friends Track.love and Track.ban instead of being all sad and lonely on the sidelines. This should simplify things for developers by having one unified authentication, request and response mechanism. We also hope that this will lead to applications which currently just scrobble to use the rest of our API and vice-versa, with the end result being cooler apps with more features for everyone.
• Migrating to the web services will improve our ability to track the use of scrobble applications, so we can do groovy things like charts of the most popular scrobblers, and analyses of musical tastes across different scrobblers. Yes, we will finally be able to answer the burning question – “Do Amarok users have better taste than XBox Live users?” We hope that our scrobbling partners and their users will be able to do cool things with this data.
• Corrections information will be returned where relevant so users can be prompted to fix any incorrect metadata they may have.
• Changes to Last.fm radio scrobbling will allow us to improve our recommendations. We’ll get more specific listener feedback because loves, bans and skips can be tied to a specific radio stream, not just to a particular track.
• We’ll return more detailed error messages which should simplify the process of developing a scrobbler.
• Third party developers will be able to upload their own icons which will show up on a Last.fm user’s profile when they are listening with a particular scrobbler. We currently provide this as a service for our most popular scrobblers but will extend this to all third party apps (this was our most requested feature after improved error logging!).

There were a lot of great ideas which didn’t make the cut, but the new API should allow us to add new features more easily and we plan to expand on this release in the future. After a round of feedback from the community we hope to put a beta version of the API out for testing and will then work towards finalising it a month or two after that. Third party developers will then be able to start updating their existing applications (or writing new ones) and passing the benefits of the new features on to you, our faithful users.

We’re hoping that by making scrobbling development easier we will be taking more steps towards getting every musical device on the planet scrobbling. Let us know what you think.

Last weekend I was lucky enough to voyage to Stockholm with Jonty and Michael to represent Team Last.fm at Music Hack Day.

Music Hack Day’s premise is simple – find the best and brightest tech and music geeks, get them all together for a weekend, mix in APIs and workshops from every online music service worth its salt, and then spend 24 hours making… well, anything!

Started by Soundcloud’s Dave Haynes here in London last July, subsequent Music Hack Days in Berlin and Boston have cemented their reputation as the best tech events going, music or no. (Anthony from the Hype Machine did a nice write-up on some of the ingredients that make them great.)

So it was with some excitement that we boarded our plane on Friday and headed north. In addition to our standard hack day paraphernalia — laptops, check; giant headphones, check; world’s tiniest Guitar Hero, check — we also carted along some limited edition stickers, newspapers, and a short presentation on the venerable Last.fm API. (You can grab those slides here as a PDF download.)

Photo by Brian Whitman.

Stockholm certainly didn’t disappoint — the weekend was awesome! We came, we hacked, we even conquered.

We also learnt a lot. Some highlights included…

Swedish hospitality

“Hospitality” isn’t generally up there on the list of familiar Swedish traits (unlike, say, tasteful flat-pack furniture, or expensive booze). But our hosts — Henrik and Mattias — made everyone feel welcome and created an environment that let everyone just get on with creating cool stuff.

There were some uniquely Swedish touches too, like the delicious bread and cheese breakfasts and the snow-based beer fridge. Oh, and the Batmobile showed up. No, really.

APIs in the mirror

Though we’ve offered public APIs to developers since 2003, nothing makes you see them in a new light like face-to-face interaction with people trying to make clever and unusual things with them. We’ve come back to London with a long list of suggested improvements, things that could be clearer in the docs, and even a couple of bug fixes that were reported by intrepid Stockholm hackers. Thanks to everyone who spoke to us!

We also handed out free subscriptions to everyone who demo’d a hack that used the Last.fm API.

A few of our favourites:

• My City vs. Your City Uses our new geo.getMetro* city charts API to compare top artists across hundreds of cities worldwide. Neat!
• SimilarArtists A simple way to generate Spotify playlists of recommended music based on Last.fm similar artists.
• Holodeck An attractive way to create an artist website that based on content from SoundCloud, Last.fm, Songkick, and Tumblr.
• Mashboard A dashboard for your Soundcloud tracks that pulls in rich audio metadata from Echo Nest. And it scrobbles!

We also managed to sneak in a few hacks of our own:

• HacKey Ever wondered what your favourite key is? Thanks to the Last.fm and Echo Nest APIs, now you can find out.
• ProximRadio and Blobble Jonty and Michael came up with a deadly trio of new tech that enables a long-standing dream: proximity-based multi-profile radio stations, complete with group scrobbling. Whoa.

A complete list of hacks is available here.

The online music ecosystem = crazy delicious

It was humbling to be in the presence of so many talented companies and developers, from the music mad scientists of The Echo Nest to the streaming wizards of Spotify, not to mention entire teams who travelled to Stockholm from Songkick, Soundcloud, and many others.

It’s pretty clear that 2010 is going to be an exciting year in music and tech. (And not just because people are building Playdar-enabled beatmatched collaborative Spotify playlist generators that scrobble via robot arm attachments…although that helps.) Team Last.fm will be in attendance the next Music Hack Day and also at some events of our own, so stay tuned.

Until then, happy hacking!

One night at the pub we discussed whether one could replace Hadoop (a massive and comprehensive implementation of Mapreduce) with a single bash script, an awk command, sort, and a sprinkling of netcat. This turned into a weekend project dubbed bashreduce.

To be fair, Hadoop probably does a few more things than bashreduce. But we’ve managed to cover a few key concepts in our script:

• Task coordination (kind of! sort of!)
• Mapping/Partitioning
• Reducing
• Merging
• Distributed file system (sort of! if you squint just right)

More than just a toy project, bashreduce lets us address a common scenario around these parts: we have a few analysis machines lying around, and we have data from various systems that are not in Hadoop. Rather than go through the rigmarole of sending it to our Hadoop cluster and writing yet another one-off Java or Dumbo program, we instead fire off a one-liner bashreduce using tools we already know in our reducer: sort, awk, grep, join, and so on.

I think it’s a neat idea! If you think it’s a neat idea, and you look at this gnarly bash code and think of ways to improve it, to make it more useful or more elegant, you would enjoy working for us. We’re looking for a clever C++ developer to help us tackle data mining and scale problems. My favorite line in the job posting is Interested in – we do all those things save one, which you can probably guess.

We’ve collected a few of our developer’s blogs here as well – more fodder for those of you interested in what we do.

Sometimes the endless rows of music in my media player leave me at a loss. I have a music collection that I’ve spent years lovingly crafting; all my favourite bands. Yet as I spin my mouse wheel its full length, nothing springs out. I scroll up, scouring the rows for something fresh. I scroll down, searching for some long forgotten treasure. After a few minutes I select “shuffle” and go make a cup of tea.

But maybe it doesn’t have to be like that. Wouldn’t it be great if you could tune into your local music like you do with Last.fm radio?

That last sentence sounded very much like a product announcement did it not? Well, it was!

Pick a tag, maybe another related tag, and click play. Last.fm Boffin. Strictly a tech demo. Let us know how you like it, and we’ll roll it into the next major release of our desktop software :)

Forum announcement and download links

Flickr group
Wordle gallery

In August last year we organized the first Hadoop User Group in the UK. We liked it so much we’re doing another one on the 14th of April.

Quite a few of you probably haven’t heard about Hadoop, in short it’s an awesome piece of software that is used to process large datasets on multiple machines. If that’s your kind of thing, read more about it here.

So far the event schedule looks like this:
10.00 – 10.15: Arriving and chatting

10.15 – 11.15: Practical MapReduce (Tom White, Cloudera)

11.15 – 12.15: Introducing Apache Mahout (Isabel Drost, ASF)

12.15 – 13.15: Lunch (three kinds of pizza, sponsored by Sun)

13.15 – 14.15: Terrier (Iadh Ounis and Craig Macdonald, University of Glasgow)

14.15 – 15.15: Having Fun with PageRank and MapReduce (Paolo Castagna, HP)

15.15 – 16.15: Apache HBase (Michael Stack, Powerset)

16.15 – 17.00: General chat, perhaps lightning talks (powered by Sun beer)

17.00 – 00.00: Discussions continues at a nearby pub

The meetup is held at Sun’s office near Monument station in London. It’s free, but we ask that you register if you want to come. For more up to date news keep an eye on the blog.

A big thanks to Sun for sponsoring the event with a venue, food and beer!

A week or so ago, on Sunday 14th December, we held our first open Hack Day, giving developers a chance to show off what they could build in a day with nothing but their wits and the Last.fm API.

At around 10:30, the hungry and cold developers started pouring into Corbet Place, behind Brick Lane in the heart of East London. With free food and drink behind the bar, plenty of comfy sofas to drape themselves over, and a Surface table with which to amuse themselves, the hackers dug in. (Sorry you guys had to wait in the cold for longer than we’d hoped, that sucked.)

By 6:30, and in spite of wifi woes throughout the day, we had 30 quick fire demos lined up to wow the assembled crowd of geeks. As anyone who’s run an event like this will attest, getting 30 odd laptops hooked up to 2 projector adaptors on rotation with a 2 minute turnaround is no mean feat, but no one was trampled underfoot and only one person outright gave up (apologies Steve!).

Amongst our favourite hacks were Bret Ehlert’s Nostalgia.fm, an app that creates playlists based on your historical charts so you can relive your headier musical days; Your Next Favourite Band by Utku Can and Phil Nash, which finds the band everyone’s listening to but you; and Neil Crosby’s Last Genius, a bookmarklet that builds a playlist using any track on Last.fm as the starting point.

We were extremely impressed with everyone’s work but after careful deliberation, we had to select three winners for the awesome prizes provided by Codeplex:

Rob Mckinnon walked away with a shiny XBox 360 for his work building Gig notifications with Growl. In his own words:
“Need help remembering to get gig tickets? This hack gives you local event notifications via Last.fm for the band you’re now playing. Implemented as Ruby script that uses last.fm api and growl notifications. Could be wrapped up as a Mac dashboard widget with a bit of work.”

Cameron Ross also snagged an XBox for the awesome Universal Scrobbler:
“This suite of tools allow you to scrobble songs from previously unscrobblable sources. There is a FireFox extension to allow scrobbling of songs listened on MySpace, a tool to browse MusicBrainz for albums and tracks to scrobble (for example for if you listen to an album in the car or CD player), a tool to scrobble songs retrospectively that you listened to on BBC Radio, and a tool to scrobble a custom song.”

David Padbury and Jamie Hollingworth stormed to win the grand prize of £1000 with Staff Wars.fm:
“StaffWars works by playing a user’s personal last.fm station communally to the office. When someone becomes offended by their colleague’s poor taste in music they initiate a challenge to take control of the office stereo from the current user. At this point StaffWars analyses the profiles of the competing users and looks for similar tastes in music. It will generate a small music quiz based on these similar tastes in music and ask it too both users. If the challenging user wins they will take control of the stereo otherwise the existing station will carry on playing.”

The day was rounded off with an excellent set from Hexstatic while the last free drinks were squeezed out of the bar.

Thanks again to those who came and made this event a success. I can’t wait for the next one!

We’ll try to keep this list updated with all the other hacks as info comes in, let us know if we’re missing you.

• Stitch: Léon Brocard
• Spotify Scrobbler: George Brocklehurst
• Where should I live? and the perceptron: Mary Rose Cook
• Distance Select: Brian Degger with help from Nigel Crawley
• Playby: Alf Eaton
• Huffduffer: Jeremy Keith
• Songcolours: Matthew Ogle
• Lastbot: Gareth Rushgrove
• With It: Mark Stickley
• Stalker: Dave Sutherland and Lars Bortfeldt
• Flighty: Nicky Thompson
• Gigs Feed Builder: Oli Thuong

More photos from the day, courtesy Russ and Dimi.