Index ¦ Archives ¦ RSS

Hermetic Build Systems

Over time, I've come to really like and expect a good hermetic build system. It allows you to really be sure of what's going to happen when you run your binary on any supported system. This makes keeping development and production systems in line very easy, just have the same architecture and the few system packages required for each language, irregardless of what will get run on the machine (or VM, Container, virtualenv, etc).

I think I'm currently spoiled, but I've looked at pants (Twitter) and buck (Facebook) and they both seem to be missing the point, which is that every build rule is really a genrule and that the build files are python syntax and therefore can use more of python than just the function-call syntax. Once you get really big build targets that take tons of time just for the analysis phase (building the graph of targets from the files), then you can consider removing the benefits of 'running' the files as python, but really I think it should never be removed as much as discouraged.

So here are the expectations of what a proper open-source build system should provide, lest it stay used by only the creating company/group/person:

  • Hermetic builds of the supported languages, including at least C++, Python and Java. I wouldn't mind if Java weren't supported, but with so many people writing Android apps it's sadly unavoidable.
  • Genrule support at its core. Supporting multiple inputs, multiple outputs, 'tools', and dependencies. Everything else can be written on top of genrules if necessary.
  • Easy extension without modifying the build system source code, preferably by writing in the build files' syntax (usually Python).

Here, pants has missed the second bullet point by not even having genrule support. Instead, twitter has managed to build every rule in a custom fashion, making extension (the third point) difficult and tedious. Every rule is mentioned in multiple places, just search for ThriftLibrary or PEXBuilder (which isn't even a rule).

Facebook has gotten closer, but missed the first and third rule. They only support java, with python support 'in alpha' and generally lacking despite being the simplest to add (zipimport makes it as simple as 'zip \$INS \$OUT.zip'). Extension is even documented as being unsupported and requires forking Buck, ridiculous. BUCK files are python-syntax (don't lie, I know it's from Google's system which was originally just python and still is python-syntax with the capability of being python preprocessed), so use that instead of acting like it was all an accident.

© Fahrzin Hemmati. Built using Pelican. Theme by Giulio Fidente on github.