Big news: tipi.build is joining EngFlow 🚀 !

Launching CMake RE Public Beta for local CMake Hermetic Containerized Builds

Today is a big milestone for us and the CMake Ecosystem :

1. tipi joins forces with EngFlow growing the CMake RE Development Team
2. We launch our Open Source CMake module Hermetic FetchContent
3. Last but not least, we launch CMake RE Local Containerized Builds in Public Beta !

You might ask yourself, but why would local execution of cmake-re be interesting, isn't the remote part absolutely required to reap the benefits ?

Why CMake RE Containerized builds ?

CMake is a remarkably versatile tool, that can bootstrap a build on any platform! It is extremely flexible and configurable and can detect a way to compile on platforms it was never thought it would be used for (for ex. linux cmake working with a microsoft compiler) and it can build massive C and C++ projects.

This versatility which is CMake core strength turns out to also be one of it's main weaknesses, which make reproducible build, incremental build caching and build distribution very tough problems.

Reproducibility

Trying to share a CMake project (that worked on one's machine) with others turns to be challenging. Reproducing the build is never trivial, even with well documented steps, the smallest differences like an additional package installed on the system can already start to be complex to debug.

Adding to this that cmake -DPARAMETERS=VALUE command-line options are persisted to the build tree in the CMakeCache.txt and that therefore two cmake invocations on an existing build tree are not always having the same effect (e.g. based on their orders) this yields the most inconceivable and unforeseeable combinations, build errors and frustrations.

CMake users are familiar with the following obtuse error message of find_package when on-boarding on a new project :

By not providing "Find<YourModule>.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "<YourModule>", but
CMake did not find one.

Because of subtle system or invocation differences, builds that seemed well tested on one developer machine refuse to work on another.

CMake RE Containerized Builds for reproducibility

cmake-re implements a concept named environment + toolchain generalization which allows tying a cmake toolchain to a given container image. The generalization process strips user specific information from a build environment and ensures build can be reproduced or cache entry be reused on different machines. In addition to the hash computed from the generalized environment CMake configuration flags like cmake -DPARAMETERS=VALUE are used to compute a final ABI-hash and build trees are stored based on this hash.

This mechanism ensures any build parameter combination (flags, CMake variables, toolchains) results in a deterministic build tree layout, which in turn enables reusing the build trees and makes builds reproducible between users, machines and cmake invocations.

Chasing Safety requires bigger and more build variants

The C++ community is heavily focused on bringing Safety to their codebases, this has been pushed by the assesment that 80% of vulnerabilities found at Microsoft's or Google's C++ codebase were due to Memory Safety issues that can be caught with tools like sanitizers and static analysis enforcing rules like the C++ Core Guidelines. (c.f. Software Memory Safety, National Security Agency, November 2022)

The issue with sanitizers enabled builds is that their instrumentation is mutually exclusive. This means that it is impossible to combine all of them in a single build, essentially requiring to rebuild and retest the same software in many different configurations (i.e. AddressSanitizer, LeakSanitizer, ThreadSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer).

Furthermore the full dependency chain needs to be instrumented for the results of such analysis to be precise and debug symbols need to be made available for the analysis' to be actionable and leverage-able for the developers. To help with both of these points we published our Open Source Hermetic FetchContent.

Making Disk Space for Sanitizer builds

The disk space requirements even for mid-size projects easily exceed multiple hundreds of gigabytes todays, especially with debug symbols enabled. This is exacerbated by the aformentioned build instrumentation requirements, meaning that even potent developer workstations face disk space issues.

CMake RE Local L1 Caching was designed, with this in mind and it's fast yet high ratio compression enable typical workflows.

Keeping engineers in the flow : Avoiding CI Build Queuing

Being able to build and run the tests of a codebase on each developer changes and provide the fastest feedback to developers, is another key element to be able to support many build variants.

Few dare to run a Continuous Integration workflow with CMake and run incremental builds on CI, as any configuration changes might not represent the result of a clean build (i.e. as it might reuse existing CMakeCache.txt values it produced in the first configuration) and this defeats the goal of CI : ensuring a project can be built.

This makes unsupervised incremental builds almost impossible with CMake, one can get some speedup by caching at the lowest level with tools like ccache but this will only cache parts of the build that are compiler invocations, no custom code generator tools and will be fragile to reconfigurations requiring the full graph build execution.

This results in most teams resorting to clean build on automated CI, loosing the chance to just do an incremental build and adds significantly to the time builds take and how long they interrupt the developers productivity.

Reliable incremental builds to the rescue

With CMake RE L1 Containerized builds and its integrated cache one can get CI nodes build faster and be confident of the reprodubility of the cached incremental build.

Faster clean builds

We built support for remote builds with our CMake RE L2 feature on the proven EngFlow's build distribution technology providing build speedup in the 24x range even for clean and non-cached builds.

Combining CMake RE L1 and CMake RE L2 yields even more speedups while saving on cloud costs. If you are interested in such a setup contact us to get access to CMake RE L2.

Get Reproducible & Reliable incremental CMake builds

Today we are excited to contribute to the CMake Ecosystem by providing a drop-in enhancement for CMake : cmake-re.

Get reproducible, cached local builds for CMake : join us to get your cmake-re download link.