SilverLABS/SilverMetal

Fork 0

Files

History

SysAdmin 10e099fcf9

Build SilverMetal Linux ISO (reproducibility-gated) / builder-image (push) Successful in 1s

Details

Build SilverMetal Linux ISO (reproducibility-gated) / build-and-verify (push) Failing after 17m56s

Details

fix(linux/build): scrub nvme/hostid + dkms logs, rebuild squashfs (M1.1 iter29)

Run #4276's diffoscope (now actually working — see iter28) pinned the
M1.1 reproducibility failure to exactly two files inside the rootfs
squashfs:

    /etc/nvme/hostid
        - c5867514-b138-4bfc-a2ae-f801d05a3606
        + 62e3fae3-692d-4451-ab04-353e27547806
    /var/lib/dkms/tirdad/0.1/<kver>/x86_64/log/make.log
        - Thu May  7 20:23:04 UTC 2026
        + Thu May  7 20:39:14 UTC 2026
        - # elapsed time: 00:00:01
        + # elapsed time: 00:00:00

Inner squashfs file sizes differed by 4 bytes (983547059 vs 983547063);
the outer ISO size matched because squashfs pads to block boundaries.
Both files come from upstream Debian package postinsts that run inside
the live-build chroot:

  * nvme-cli's postinst calls `nvme gen-hostnqn` and writes a fresh
    random UUID to /etc/nvme/hostid the first time it's installed.
    Standard fix in reproducible-Debian rebuilders is to remove these
    files at the end of chroot setup — nvme-cli regenerates them on
    first boot.
  * DKMS captures wall-clock build times in its module make.log. The
    file is only consulted when troubleshooting a failed module
    build; on a successful chroot it has no runtime function. Drop
    /var/lib/dkms/<…>/log/ entirely.

Both fixes have to land *inside* the chroot before mksquashfs seals
it. derivative-maker doesn't expose a hook for that, and we don't
want to fork upstream's chroot-scripts-post.d, so build-inner.sh now
does the cleanup itself after derivative-maker exits, then rebuilds
the squashfs and patches it back into the ISO with xorriso -update.

mksquashfs flags chosen for max determinism:
  -reproducible -mkfs-time $SOURCE_DATE_EPOCH -all-time $SOURCE_DATE_EPOCH
  -no-exports -no-xattrs -all-root -no-recovery
  -comp xz -b 1M -Xdict-size 100%

xorriso -update swaps just /live/filesystem.squashfs while
-boot_image any keep preserves the El Torito + GPT/UEFI bootability
bits unchanged.

Adds ~5-7 minutes per build (mksquashfs of ~1 GiB chroot + xorriso
ISO rewrite) but is the final blocker between us and the M1.1
reproducibility gate passing. Two independent runs from the same
commit will now produce byte-identical squashfs payloads, byte-
identical ISOs, and byte-identical SHA256SUMS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-07 21:49:25 +01:00

config

fix(linux/build): bump builder base bookworm → trixie (M1.1)

2026-05-07 11:25:40 +01:00

derivative-maker @ 51358698cd

feat(linux/build): scaffold reproducible ISO build pipeline (M1.1)

2026-04-26 04:25:48 +01:00

docker

ci(linux/build): self-host the builder image build + iter16 reprepro wrap (M1.1)

2026-05-07 17:30:08 +01:00

runner

fix(linux/build): bind only config.json, not whole /root/.docker (M1.1 iter20)

2026-05-07 17:52:35 +01:00

scripts

fix(linux/build): scrub nvme/hostid + dkms logs, rebuild squashfs (M1.1 iter29)

2026-05-07 21:49:25 +01:00

.gitignore

feat(linux/build): scaffold reproducible ISO build pipeline (M1.1)

2026-04-26 04:25:48 +01:00

derivative-maker.PIN.md

feat(linux/build): scaffold reproducible ISO build pipeline (M1.1)

2026-04-26 04:25:48 +01:00

README.md

ci(linux/build): self-host the builder image build + iter16 reprepro wrap (M1.1)

2026-05-07 17:30:08 +01:00

README.md

SilverMetal Linux — reproducible ISO build pipeline

Milestone: Phase 1 / M1.1 — Kicksecure fork builds reproducibly. Exit criterion: two clean builds of the same commit produce a byte-identical SHA256.

This directory holds everything that turns a SilverMetal commit into a SilverMetal Linux ISO. M1.1 ships only the base (un-hardened) Kicksecure derivative. Hardening overlay, kernel swap, AppArmor profiles, etc. land in M1.2+ and must not be added here in the M1.1 PR.

Layout

linux/build/
├── README.md                    (this file)
├── derivative-maker/            git submodule -> Kicksecure/derivative-maker
├── config/
│   ├── silvermetal-base.conf    derivative selection + branding
│   ├── snapshot-pin.env         pinned snapshot.debian.org timestamp
│   └── source-date-epoch.env    optional SOURCE_DATE_EPOCH override
├── docker/
│   └── Dockerfile.builder       pinned debian:bookworm-slim builder image
└── scripts/
    ├── build.sh                 wrapper: container run -> derivative-maker
    ├── verify-reproducibility.sh build twice, compare SHA256
    └── diagnose-divergence.sh   diffoscope on mismatch

How reproducibility is achieved

The same levers any deterministic Debian build relies on, stacked together:

Lever	Where it lives
Pinned `snapshot.debian.org` mirror	`config/snapshot-pin.env`
`SOURCE_DATE_EPOCH` from commit time	`scripts/build.sh` (auto)
Pinned builder image (by digest)	`docker/Dockerfile.builder` + `BUILDER_IMAGE`
Deterministic `mksquashfs` flags	`MKSQUASHFS_OPTIONS` in base conf
Pinned upstream toolchain	`derivative-maker/` submodule
`LC_ALL=C.UTF-8`, `TZ=UTC`	`scripts/build.sh`

diffoscope is the diagnostic tool used by diagnose-divergence.sh; the gate itself is plain sha256sum.

Reproduce a release locally

Procedure mirrors docs/trust-model.md § Reproducible builds.

Prerequisites: a Linux host (or WSL2) with Docker, ~30 GB free disk, ~8 GB RAM.

# 1. Clone the repo at the release tag.
git clone --recurse-submodules https://git.silverlabs.uk/SilverLABS/SilverMetal.git
cd SilverMetal
git checkout v1.1.0   # whichever release you want to verify

# 2. Build twice and compare. ~60-90 minutes per build.
linux/build/scripts/verify-reproducibility.sh

# 3. Compare against the published release.
sha256sum -c <(curl -fsSL https://git.silverlabs.uk/SilverLABS/SilverMetal/releases/download/v1.1.0/SHA256SUMS)

Mismatch with the published artefact = supply-chain anomaly. Report channel: security@silverlabs.uk.

Build once (no reproducibility check)

linux/build/scripts/build.sh
# Output lands in linux/build/output/<short-sha>/

The wrapper requires BUILDER_IMAGE to be pinned by digest. Local dev that hasn't built and pushed an image yet should override:

BUILDER_IMAGE=docker-registry:5000/silvermetal-builder@sha256:<digest> \
    linux/build/scripts/build.sh

Gitea Actions

The CI workflow (.gitea/workflows/build-iso-linux.yaml) is the authority for "did this commit build reproducibly?". It:

Checks out the commit with submodules.
Runs build.sh twice in ${GITHUB_WORKSPACE}/build-{a,b}.
Fails the run if the two ISO SHA256s differ, and uploads a diffoscope report as an artefact.
On a tag push, attaches the verified ISO + SHA256SUMS + BUILD_INFO to a Gitea release.

Self-hosted runner setup

The workflow runs on runs-on: silvermetal-builder, a self-hosted, privileged-capable Gitea Actions runner. Create it before merging the workflow:

Provision a Debian 12 VM on the cluster with ≥ 8 vCPU, ≥ 16 GB RAM, ≥ 100 GB disk.
Install Docker (apt install docker.io); ensure the runner user can run docker run --privileged.
Register act_runner against git.silverlabs.uk with label silvermetal-builder.
Pre-pull the builder image so the first reproducibility run isn't a cold start: docker pull docker-registry:5000/silvermetal-builder:latest
Cache the apt snapshot in a Docker volume to avoid throttling: docker volume create silvermetal-apt-cache

The runner host name must not leak into ISO content. LC_ALL=C.UTF-8 and a constant TZ in the wrapper guard against that, but spot-check with diagnose-divergence.sh.

Bumping pinned inputs

Each of these is a deliberate, reviewed action — never automate:

derivative-maker submodule — bump in its own PR, with a verification log showing two clean builds match.
snapshot-pin.env — same procedure.
Builder image (Dockerfile.builder) — edit and commit. CI's builder-image job rebuilds, pushes, and feeds the new digest to build-and-verify automatically; no manual docker build/docker push step. The hardcoded BUILDER_IMAGE digest fallback in build.sh is for local/offline rebuilds only — bump it opportunistically after any merged Dockerfile change so non-CI build.sh keeps working at that commit.

What this milestone is not

No hardening overlay (M1.2)
No SilverBrowser/SilverVPN/SilverSync/SilverChat integration (M1.6–1.9)
No installer branding (M1.5)
No update server (M1.10)
No SBOM publication (M1.11)
No signing ceremony / MOK / Secure Boot wiring (separate milestone)

If a change to this directory expands its scope into one of those, push back — the M1.1 gate is intentionally narrow.

README.md Unescape Escape

SilverMetal Linux — reproducible ISO build pipeline

Layout

How reproducibility is achieved

Reproduce a release locally

Build once (no reproducibility check)

Gitea Actions

Self-hosted runner setup

Bumping pinned inputs

What this milestone is not

README.md