Last.fm – the Blog

Here at Last.fm we love Open Source. Most of the time we’re just using a lot of Open Source Software, sometimes we’re contributing changes or fixes back to existing projects, and sometimes we release our own software to the public. This week, we’ll be releasing some exciting projects to the C++ community. The first of these projects is a build system we’ve conceived for our C++ codebase and which has helped us a lot — and it might be useful for you, too!

Last.fm’s MIR team is responsible for maintaining more than a hundred libraries, tools and backend services, most of which are written in C++, although some projects are in Python, Perl or Java. Back in 2011, all these projects had to be built from one giant Subversion repository, they contained hard-coded relative paths to other projects they depended on, yet as developers we would still have to know all the dependencies and build them in the correct order to actually build the project we were interested in. Also, every project contained a lot of boilerplate code and over time, this code changed, so it could be substantially different between any two projects. All of this made it quite painful to build projects or set them up for continuous integration, let alone distribute them to our production servers.

As we were thinking about migrating our codebase to Git, we wondered whether there was an easier way to build our projects. Our ideal solution at that point would have been a tool that allowed us build, test, install and package every project, regardless of the language it’s written in, with exactly the same command. We couldn’t find anything like that and so we decided to write our own tool, which we called mirbuild (for hopefully obvious reasons).

mirbuild is a meta-build-system, which means it’s basically delegating the actual build process to other build systems, but hides this behind a common interface. It is just a set of Python libraries, so the actual build scripts are written in Python. For a simple project, such a build script (usually called build.py) looks like this:

  #!/usr/bin/env python
  from mirbuild import *
  project = CMakeProject('libcyclone')
  project.depends('libmoost')
  project.find('boost')
  project.find('log4cxx')
  project.version('include/cyclone/version.h',
             namespace = ['cyclone', 'version'])
  project.run()

As you can guess from the class name, this project uses CMake under the hood. But if you just want to build the project, you don’t have to care. You just run

  ./build.py build

and it “just works”. But mirbuild does a bit more than just forwarding commands to CMake. For example, it will create a file that controls compile flags and include and link paths of project dependencies. It will also create a version header for your project if you ask it to do so.

Here’s are some of mirbuild’s features:

• supports CMake (C/C++), Python, Thrift (C++/Python) and “simple” projects
• can build, test, install and clean up projects
• can resolve dependencies between projects
• can create Debian packages
• can build different configurations (release, debug, coverage) of a project
• can run code coverage analysis tools

Over the last one and a half years, mirbuild has saved us from a lot of grief and it has made building projects a lot of fun. Thanks to mirbuild, we’ve also simplified our continuous integration framework and have now got all our production packages built on disposable virtual machines (but that’s a different story). If you’re maintaining lots of C++ code and aren’t happy with how you’re building it, check it out, it’s on Github.

To be continued…