Build me, please!

Monday, 18 February 2013
by marcus
filed under Announcements and Code
Comments: 8

Here at Last.fm we love Open Source. Most of the time we’re just using a lot of Open Source Software, sometimes we’re contributing changes or fixes back to existing projects, and sometimes we release our own software to the public. This week, we’ll be releasing some exciting projects to the C++ community. The first of these projects is a build system we’ve conceived for our C++ codebase and which has helped us a lot — and it might be useful for you, too!

Last.fm’s MIR team is responsible for maintaining more than a hundred libraries, tools and backend services, most of which are written in C++, although some projects are in Python, Perl or Java. Back in 2011, all these projects had to be built from one giant Subversion repository, they contained hard-coded relative paths to other projects they depended on, yet as developers we would still have to know all the dependencies and build them in the correct order to actually build the project we were interested in. Also, every project contained a lot of boilerplate code and over time, this code changed, so it could be substantially different between any two projects. All of this made it quite painful to build projects or set them up for continuous integration, let alone distribute them to our production servers.

As we were thinking about migrating our codebase to Git, we wondered whether there was an easier way to build our projects. Our ideal solution at that point would have been a tool that allowed us build, test, install and package every project, regardless of the language it’s written in, with exactly the same command. We couldn’t find anything like that and so we decided to write our own tool, which we called mirbuild (for hopefully obvious reasons).

mirbuild is a meta-build-system, which means it’s basically delegating the actual build process to other build systems, but hides this behind a common interface. It is just a set of Python libraries, so the actual build scripts are written in Python. For a simple project, such a build script (usually called build.py) looks like this:

  #!/usr/bin/env python
  from mirbuild import *
  project = CMakeProject('libcyclone')
  project.depends('libmoost')
  project.find('boost')
  project.find('log4cxx')
  project.version('include/cyclone/version.h',
             namespace = ['cyclone', 'version'])
  project.run()

As you can guess from the class name, this project uses CMake under the hood. But if you just want to build the project, you don’t have to care. You just run

  ./build.py build

and it “just works”. But mirbuild does a bit more than just forwarding commands to CMake. For example, it will create a file that controls compile flags and include and link paths of project dependencies. It will also create a version header for your project if you ask it to do so.

Here’s are some of mirbuild’s features:

  • supports CMake (C/C++), Python, Thrift (C++/Python) and “simple” projects
  • can build, test, install and clean up projects
  • can resolve dependencies between projects
  • can create Debian packages
  • can build different configurations (release, debug, coverage) of a project
  • can run code coverage analysis tools

Over the last one and a half years, mirbuild has saved us from a lot of grief and it has made building projects a lot of fun. Thanks to mirbuild, we’ve also simplified our continuous integration framework and have now got all our production packages built on disposable virtual machines (but that’s a different story). If you’re maintaining lots of C++ code and aren’t happy with how you’re building it, check it out, it’s on Github.

To be continued…

Comments

  1. Phillip Wirth
    18 February, 18:02

    Well fine, I use maven, easy simple open-source and thousands of plugins.

    Okay maven is mostly for Java, but it can be used with other languages as well.

    http://maven.apache.org/

    Phillip Wirth – 18 February, 18:02
  2. MusX
    18 February, 18:09

    R
    http://cran.r-project.org/web/packages/RLastFM/index.html

    MusX – 18 February, 18:09
  3. Graham
    19 February, 16:11

    womankharkov – I believe the design should be your own, to make it clear to Last.fm users who has created the site. We have a “Powered by Last.fm” logo that we encourage you to use.

    Please see:
    http://www.last.fm/resources
    and http://www.last.fm/api/tos

    Thanks,
    Graham

    Graham – 19 February, 16:11
  4. Samuel
    20 February, 12:18

    If you find this interesting, you might find teapot interesting too. It is a similar build system, but written in Ruby and designed to simplify cross-platform builds.

    Samuel – 20 February, 12:18
  5. Paradoks
    22 February, 13:53

    Your lvoe of open source does not line up with your love of closed borders

    Paradoks – 22 February, 13:53
  6. DJ Micha
    14 March, 14:43

    great news, didn’t even know I already delete so many artists :P question: will be available in future possibility to restore artists and tracks from global reset in account settings? really would like to do that

    DJ Micha – 14 March, 14:43
  7. Cao Hung
    17 March, 06:48

    great news, didn’t even know I already delete so many artists :P question: will be available in future possibility to restore artists and tracks from global reset in account settings? really would like to do that

    Cao Hung – 17 March, 06:48
  8. james morgan
    24 March, 19:48

    at the age of three i loved music its my drug it keeps me happy6 when i am sad glad when i am mad either way i am building on myself from top 2 the bottom of my soul to not perish before my time. i was married that a great life. but i start putting off thing i needed done for myself to be happy. i lost my beloved brother in-law and fell in a very bad depression stage be i still rosed and overcame that struggle just to be in another.

    james morgan – 24 March, 19:48

Comments are closed for this entry.