Enterprise Linux is old on purpose

Your cluster runs an enterprise distribution because the vendor supports the interconnect, the parallel filesystem and the GPU driver against that kernel and no other. The cost is a frozen userland.

The first two rows are what you run. The third is what your users’ documentation assumes.

So everybody installs everything, everywhere

• The user’s move fails - sudo dnf install ... on a login node, and the sudoers lecture
• The administrator’s move does not scale - one modulefile, then another version, then a second compiler, then a third MPI. All written by hand.
• And the site next door is doing it too - same packages, same sources, different results

The effort is real, the people are good, and almost all of it is duplicated.

The axis is not build-or-mount

It is how many times. The tree gets built either way.

“Isn’t this what containers are for?”

Two things the word reproducible is doing at once:

• Repeatability - running the same bits again. Containers are excellent at this.
• Accountability - saying what those bits are. A container cannot touch it.

A container is a distribution format, not a derivation. Freezing an unknown does not make it known.

And, separately, the practical failure

• The image grows with the catalogue - the unit of change is the whole image, not the package
• You ship what might be used, not what is - importing one Python library touches ~3,500 files out of hundreds of thousands

Containers stay in the picture. They carry an OS and a mount point, not a stack.

CernVM-FS: software, streamed

A read-only directory under /cvmfs that behaves like an on-demand streaming service. Nothing installed, nothing downloaded up front - files arrive when a process opens them.

• A filesystem, not a package manager - FUSE, so every tool that opens a file can use it
• Content-addressed - identical bytes stored once, everywhere
• Compressed on the server, transparent on the client
• Ordinary HTTP - the cache is Squid, the mirror is Apache

Integrity: one signature, covering everything

Exactly one object is signed. Everything else is reachable from it by content hash.

The manifest is the entire entry point

C7d1e...                 # root catalog, by content hash
B1179648                 # its size in bytes
Xf3a9...                 # hash of the signing certificate
T1747526400              # when this revision was published
D240                     # how long a client may cache it
S5348                    # revision number
Nsoftware.eessi.io       # fully qualified repository name
--                       # everything below is the signature

The transport does not need to be trusted. A proxy, a mirror or a disk can be wrong without the client being wrong.

Publishing is a transaction

cvmfs_server transaction  software.example.org
#   ... install ...
cvmfs_server publish -a r-4.5.1 -m 'R 4.5.1, foss/2023a' software.example.org

cvmfs_server abort        software.example.org   # throws it all away
cvmfs_server rollback -t  r-4.5.1                # go back to a named revision

Every publish is an immutable revision. There is no half-published state for a client to catch.

Four tiers, and what each one is for

Sizing, in one slide

• Stratum 0 - the only writable copy. Small, controlled, not load-bearing for readers.
• Stratum 1 - full replica. Private one ≈ 360 GB for EESSI, +1 GB/day. Your disconnect insurance.
• Proxy - one per 100-500 nodes, at least two. This is what makes MPI start-up bearable.
• Client cache - on the SSD, big enough. Where almost every read is answered.

The entire client configuration

# /etc/cvmfs/default.local
CVMFS_HTTP_PROXY="http://proxy-a:3128|http://proxy-b:3128"
CVMFS_QUOTA_LIMIT=10000          # 10 GB of local cache
CVMFS_CACHE_BASE=/ssd/cvmfs      # the fastest disk the node has

sudo cvmfs_config reload
ls /cvmfs/software.eessi.io      # mounted on demand by autofs

What it scales to (CERN, May 2026)

• ~15 Stratum 1 mirror servers
• > 33 billion files in the /cvmfs tree
• ~2 PB accessible - proven to 100 PB
• ~290 repositories, including 8,000+ container images

It is faster than the filesystem you already own

Importing one Python package touches ~3,500 files / ~1.1 GB, most of them tiny.

Why: only the closure crosses the wire

The catalogue grows without bound because nothing pays for the parts it does not open.

Four layers

…and the middle one is a role, not a product

• Canada filled the compatibility slot with Nix
• Canada then replaced it with Gentoo Prefix
• EESSI chose Gentoo Prefix from the start

Three configurations, two programmes, one working architecture each time - and the layers above and below did not change when it was swapped.

This converts a technology bet into an architectural choice.

The compatibility layer

A full OS userland installed somewhere other than / - its own loader, glibc, coreutils, bash, autotools - built from source, without root.

Why Nix was replaced, in their words

• Store hashes leaked upward through CMake, qmake and Python virtualenvs
• Garbage collection was destructive - the feature, from the other side

Gentoo Prefix: no symlinks, no store leak, minimal solution.

— Digital Research Alliance of Canada, on the 2020 migration

EasyBuild records it

The easyconfig is copied into the installation. Provenance is a by-product, not a chore.

Lmod makes it choosable

Architecture belongs in the hierarchy, not the name

$ module load fftw
Lmod has detected the following error: These module(s) exist but cannot be
loaded as requested: "fftw"

$ module spider fftw/3.3.10
  You will need to load all module(s) on one of the lines below first:
     gcc/12.3  openmpi/4.1.5

$ module load gcc/12.3 openmpi/4.1.5 fftw/3.3.10

~1,000 applications × 4 CPU generations × 4 GPU generations = 10,000+ permutations. The user sees fftw/3.3.10.

The Canadian stack, in numbers

• 5 regional consortia, 38 institutions, ~18,000 accounts, 6 national systems
• ~1,000 scientific applications, 10,000+ version/CPU/toolchain permutations
• 17,000+ Python wheels
• One catalogue, one scheduler, one way of installing things - since 2017

Getting in

# the Alliance stack - public, mountable anywhere
source /cvmfs/soft.computecanada.ca/config/profile/bash.sh
module load StdEnv/2023

# EESSI - in the default CernVM-FS config since November 2023
source /cvmfs/software.eessi.io/versions/2023.06/init/bash
module load R/4.3.2-gfbf-2023a

Old environments stay mountable

Nothing is promoted by being edited

Already a qualification pipeline, whether or not anyone calls it one.

EESSI: the same architecture, new hardware

Validated with a ReFrame test suite a site runs against its own mount.

The tree carries all of them; the machine picks

“We moved to Arm” stops being a recompile and a revalidation.

The seam: host injections

CUDA’s licence permits redistributing runtime libraries only - so nvcc is a symlink into a local SDK.

…and the influence now runs both ways

• The Alliance’s Gentoo Prefix bootstrap is based on EESSI’s Ansible implementation
• The Alliance has adopted EESSI’s variant-symlink approach for CUDA driver libraries

The method has transferred between programmes that share no staff, funding or hardware - and is now transferring back.

The four error messages, decoded

Five commands worth knowing first

cvmfs_config probe software.eessi.io       # is it reachable at all
cvmfs_config stat -v software.eessi.io     # revision, cache use, hit rate
cvmfs_config showconfig software.eessi.io  # what it thinks it was told
cvmfs_talk -i software.eessi.io ncleanup24 # cache evictions in 24h
sudo cvmfs_config wipecache                # last resort, not first

The boundary - what this does not solve

• Updates are visible within minutes, not instantly
• The signing whitelist expires - an availability dependency on the runbook
• Unprivileged mounting works, but each process gets its own cache
• EasyBuild does not sandbox - bit-for-bit is a target, not a guarantee
• Licensed software does not fit the model
• Derivation proves construction, never meaning

The one that catches people

If your nodes have no local disk, do not run one client and export it over NFS.

It can be made to work. It is slow. It generates more operational problems than any other configuration in the deployment guidance.

Use a loopback cache file on the shared filesystem instead.

Four questions that stop being open

Consequences, not benefits

• “Which version is this?” has one answer - same source, same compiler, same BLAS, same bytes
• Approval attaches once, on the trunk - a second approval path is two things to keep in agreement forever
• Reproducing an old result is a mount, not a rebuild
• Changing architecture stops being a project - Arm is the same line resolving elsewhere

Where this leaves us

None of this is novel and all of it is in production, at national scale, for around eighteen thousand researchers.

The useful conversation is not whether the model works - two national programmes have answered that - but which parts of it fit the estate you already have.

https://what.ndexr.io