Don't Rebuild Yourself - an Intro to Nix Package Caches

John Lago -

In previous blog posts, we’ve talked about how we speed up Devbox installs by fetching package closures directly from the official Nix Cache. This blog post will dig a bit more into how a Nix cache solution works, and how it can speed up your package installation time. In addition to covering the official Nix cache, we’ll talk about how you can integrate the Jetify Cache with Devbox to speed up your shells and share your custom packages across your team.

Building Reproducible Packages with Nixpkgs

The Nixpkgs repository provides over 100,000 packages, and supports multiple  architectures and operating systems. Nix packages define all their requirements and dependencies within their build definition, making them reproducible and isolating them from the dependencies on your host machine. This means you can install multiple versions of the same Nix package on the same machine without conflicts.

This isolation comes with a trade-off. Due to the unique way Nix defines packages, it can't simply download upstream binaries. Instead, it needs to ensure the package links its dependencies in the Nix store correctly. The only way to guarantee a package does this is to build the package and it’s dependencies from source, and link it’s dependencies within the Nix store.

All that building can add up. Building a package like MongoDB can take up to 30 minutes, and that's just for MongoDB. Building a full closure from source can take hours, and updating a dependency in the graph means you have to rebuild the whole universe. All that building leads to the obligatory XKCD comic:

Speeding up Installations with Nix Package Caches

Fortunately, Nix's reproducibility and isolation also makes it easy to share and reuse the build outputs of packages. Each Nix package has a unique store path determined by the hash of its inputs. This unique store path means Nix can identify and reuse build outputs easily.

This reuse happens in two layers:

  1. Nix can reuse existing packages and outputs in your Nix Store. When Nix starts building your package's dependencies, it first checks if a package with the correct store path exists in your local store, and then reuses that store path. The Nix Store enables sharing packages on a single machine, but you'll need a different layer if you want to share across machines.
  2. To share across machines, you need a Nix Cache. If someone is building and pushing the packages you need to a cache, you can skip building and download precisely what you need. Developers can configure multiple Nix caches as "substituters," which tell the Nix Daemon where to look for “substitute” store paths that aren't in their current Nix store.

The largest cache is the Public Cache maintained by the NixOS Foundation. The Foundation knows that users want to avoid constantly rebuilding, so they build most of Nixpkgs packages in Hydra (Nix’s CI build system) and push them to the public cache. For the most popular binaries and platforms, the Nix Public Cache has you covered.

The public cache does have some gaps, however, that your project can fall into

  1. Packages outside the Nixpkgs like custom packages on our machine or flakes hosted in public repositories.
  2. Packages such as MongoDB, Terraform, and Vault aren't hosted in the Nix cache because they lack an open-source license.
  3. Older packages, or packages on less popular platforms, can be garbage collected or excluded from the public cache.

For this reason, many developers set up their own Nix caches to use as substituters for their custom packages. Technically, any machine running Nix can serve as a cache or substituter for another machine. If you're trying to support a team, however, managing issues like trust, access control, and populating the cache with the latest packages can take a lot of extra effort and development.

Jetify Cache: A Package Cache designed for Devbox and Nix

For teams that want a cache without the extra overhead, we built the Jetify Cache. Jetify Cache is an enterprise grade Nix Cache that integrates nicely with Devbox, so you can speed up your shells with little extra effort.

Our Jetify Cache offers two solutions that can help your team develop faster:

  1. For packages in Nixpkgs that are not available in the Nix Cache, we offer the Jetify Prebuilt Cache. This provides prebuilt versions of packages not in the official Nix Cache. If you have a Jetify Cloud account, Devbox will automatically configure the Prebuilt cache as a substituter. When Devbox prepares to install a package, it will use the Prebuilt cache if there are any gaps in your local Nix Store or the official Nix cache.
  2. For your custom packages and flakes, we offer the Jetify Private Cache. Jetify Privat Cache lets you share binaries across all your devices and developers. After you build your Devbox project (either locally, or in CI), you can run devbox cache upload to push the entire closure of your project to the cache. Developers who share your Jetify Cloud Project can then easily download their packages from the cache without building from source.

Devbox: Optimized for Nix Package Caching

Devbox can automatically configure the Jetify Cache caches for you when you authenticate with Jetify Cloud With the Jetify Cache enabled, you now have 4 layers where Devbox and Nix can check for binaries before building from source:

  1. The Nix Store (`/nix/store`)
  2. Your Jetify Private Cache
  3. The Official Nix Cache (cache.nixos.org)
  4. The Jetify Prebuilt Cache.

As mentioned above, Devbox can install packages by copying them directly from the cache, without needing to re-run evaluation steps. We do this by retrieving the store paths directly from Nixhub (our Nix Package Search engine) at install time, and then save the store paths in our devbox.lock file. With this optimized cache experience, you can start reducing your installation times in seconds.

Keep Up To Date on Jetify and Devbox

If you want to speed up your package installs, the Jetify Prebuilt Cache is available to any developer who signs up for a free Jetify Cloud account. If you have custom packages or binaries you want to cache, you can check out our pricing plans for Jetify Cloud.

Try Jetify Cache

We’d love to hear your feedback on Devbox and Jetify Cache. You can follow us on Twitter, or chat with our developers live on our Discord Server. We also welcome issues and pull requests on our Github Repo.