SilverLABS/SilverMetal

Fork 0

Files

SysAdmin e260fe1c81

Build SilverMetal Linux ISO (reproducibility-gated) / builder-image (push) Failing after 2s

Details

Build SilverMetal Linux ISO (reproducibility-gated) / build-and-verify (push) Has been skipped

Details

ci(linux/build): self-host the builder image build + iter16 reprepro wrap (M1.1)

Two coupled changes that unblock the M1.1 iter loop. Both belong in CI;
iter1-15 was wrong to require human-in-the-loop steps to make progress.

1. **CI now builds Dockerfile.builder.**

   `.gitea/workflows/build-iso-linux.yaml` grows a `builder-image` job
   that runs ahead of `build-and-verify`. It rebuilds the silvermetal-
   builder image from `linux/build/docker/Dockerfile.builder`, pushes it
   to `docker-registry.silverlabs.uk/silvermetal-builder:m1.1-<sha>` (and
   `:latest`), reads the resulting digest off `docker inspect`, and
   feeds it forward as a job output. `build-and-verify` consumes that
   digest as the `BUILDER_IMAGE` env override that `build.sh` already
   honours (and validates is digest-form on line ~37).

   That kills the old workflow where every Dockerfile.builder change
   required a human to `docker build` + `docker push` on 10.0.0.51 by
   hand and then bump the digest in `build.sh` in lockstep. The crash
   that triggered this (exit 126 mid-iter16 build run) was a symptom of
   that off-CI step still existing.

   Both jobs run on the existing `silvermetal-builder` runner; the host
   docker daemon is shared via DooD and is already authenticated to
   `docker-registry.silverlabs.uk` (linux/build/runner/docker-compose.yml
   mounts `/root/.docker:/root/.docker:ro`), so no extra login step.

   The hardcoded `BUILDER_IMAGE` digest in `build.sh` stays as the
   local-developer / offline-rebuild fallback. Comments updated in
   `build.sh`, `Dockerfile.builder`, and `linux/build/README.md` to
   match the new flow.

2. **reprepro wrapper for the benign "No priority for X" case.**

   Pinned derivative-maker's `2100_create-debian-packages` (with
   --target iso) re-imports source packages from snapshot.debian.org
   into a local apt repo via `reprepro --basedir … includedsc local
   <foo>.dsc`. The local repo's `conf/distributions` ships no
   `DscOverride` entries, so any source package whose `.dsc` lacks an
   explicit Priority field trips:

       No priority for 'X', skipping.
       There have been errors!

   …and reprepro exits 255. dm-reprepro-wrapper bubbles that up,
   2100_create-debian-packages aborts. The current offender is
   `virtualbox_*.dsc` (key import is now fine — debian-keyring landed in
   commit 4aa59ba — but the priority field gap remains). VirtualBox is
   not in SilverMetal's `--target iso` set, so the sane behaviour is
   "log it, continue".

   New `linux/build/docker/silvermetal-reprepro-wrap.sh` shadows
   `/usr/bin/reprepro` at `/usr/local/bin/reprepro` (PATH precedence).
   It runs the real reprepro, captures merged stdout+stderr, and:
   - if rc != 0 AND every non-blank output line matches one of the
     known-benign patterns ("No priority for 'X', skipping." plus the
     trailing "There have been errors!"), emits the output, logs one
     line of explanation to stderr, and exits 0;
   - otherwise emits the output and propagates rc unchanged.

   Any *other* reprepro error path stays fatal — only the specific
   "No priority for X" pattern is neutralised. `dm-reprepro-wrapper`
   resolves `reprepro` via `\$PATH` so it picks up the wrapper
   transparently.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-07 17:30:08 +01:00

5.6 KiB

Raw Permalink Blame History

SilverMetal Linux — reproducible ISO build pipeline

Milestone: Phase 1 / M1.1 — Kicksecure fork builds reproducibly. Exit criterion: two clean builds of the same commit produce a byte-identical SHA256.

This directory holds everything that turns a SilverMetal commit into a SilverMetal Linux ISO. M1.1 ships only the base (un-hardened) Kicksecure derivative. Hardening overlay, kernel swap, AppArmor profiles, etc. land in M1.2+ and must not be added here in the M1.1 PR.

Layout

linux/build/
├── README.md                    (this file)
├── derivative-maker/            git submodule -> Kicksecure/derivative-maker
├── config/
│   ├── silvermetal-base.conf    derivative selection + branding
│   ├── snapshot-pin.env         pinned snapshot.debian.org timestamp
│   └── source-date-epoch.env    optional SOURCE_DATE_EPOCH override
├── docker/
│   └── Dockerfile.builder       pinned debian:bookworm-slim builder image
└── scripts/
    ├── build.sh                 wrapper: container run -> derivative-maker
    ├── verify-reproducibility.sh build twice, compare SHA256
    └── diagnose-divergence.sh   diffoscope on mismatch

How reproducibility is achieved

The same levers any deterministic Debian build relies on, stacked together:

Lever	Where it lives
Pinned `snapshot.debian.org` mirror	`config/snapshot-pin.env`
`SOURCE_DATE_EPOCH` from commit time	`scripts/build.sh` (auto)
Pinned builder image (by digest)	`docker/Dockerfile.builder` + `BUILDER_IMAGE`
Deterministic `mksquashfs` flags	`MKSQUASHFS_OPTIONS` in base conf
Pinned upstream toolchain	`derivative-maker/` submodule
`LC_ALL=C.UTF-8`, `TZ=UTC`	`scripts/build.sh`

diffoscope is the diagnostic tool used by diagnose-divergence.sh; the gate itself is plain sha256sum.

Reproduce a release locally

Procedure mirrors docs/trust-model.md § Reproducible builds.

Prerequisites: a Linux host (or WSL2) with Docker, ~30 GB free disk, ~8 GB RAM.

# 1. Clone the repo at the release tag.
git clone --recurse-submodules https://git.silverlabs.uk/SilverLABS/SilverMetal.git
cd SilverMetal
git checkout v1.1.0   # whichever release you want to verify

# 2. Build twice and compare. ~60-90 minutes per build.
linux/build/scripts/verify-reproducibility.sh

# 3. Compare against the published release.
sha256sum -c <(curl -fsSL https://git.silverlabs.uk/SilverLABS/SilverMetal/releases/download/v1.1.0/SHA256SUMS)

Mismatch with the published artefact = supply-chain anomaly. Report channel: security@silverlabs.uk.

Build once (no reproducibility check)

linux/build/scripts/build.sh
# Output lands in linux/build/output/<short-sha>/

The wrapper requires BUILDER_IMAGE to be pinned by digest. Local dev that hasn't built and pushed an image yet should override:

BUILDER_IMAGE=docker-registry:5000/silvermetal-builder@sha256:<digest> \
    linux/build/scripts/build.sh

Gitea Actions

The CI workflow (.gitea/workflows/build-iso-linux.yaml) is the authority for "did this commit build reproducibly?". It:

Checks out the commit with submodules.
Runs build.sh twice in ${GITHUB_WORKSPACE}/build-{a,b}.
Fails the run if the two ISO SHA256s differ, and uploads a diffoscope report as an artefact.
On a tag push, attaches the verified ISO + SHA256SUMS + BUILD_INFO to a Gitea release.

Self-hosted runner setup

The workflow runs on runs-on: silvermetal-builder, a self-hosted, privileged-capable Gitea Actions runner. Create it before merging the workflow:

Provision a Debian 12 VM on the cluster with ≥ 8 vCPU, ≥ 16 GB RAM, ≥ 100 GB disk.
Install Docker (apt install docker.io); ensure the runner user can run docker run --privileged.
Register act_runner against git.silverlabs.uk with label silvermetal-builder.
Pre-pull the builder image so the first reproducibility run isn't a cold start: docker pull docker-registry:5000/silvermetal-builder:latest
Cache the apt snapshot in a Docker volume to avoid throttling: docker volume create silvermetal-apt-cache

The runner host name must not leak into ISO content. LC_ALL=C.UTF-8 and a constant TZ in the wrapper guard against that, but spot-check with diagnose-divergence.sh.

Bumping pinned inputs

Each of these is a deliberate, reviewed action — never automate:

derivative-maker submodule — bump in its own PR, with a verification log showing two clean builds match.
snapshot-pin.env — same procedure.
Builder image (Dockerfile.builder) — edit and commit. CI's builder-image job rebuilds, pushes, and feeds the new digest to build-and-verify automatically; no manual docker build/docker push step. The hardcoded BUILDER_IMAGE digest fallback in build.sh is for local/offline rebuilds only — bump it opportunistically after any merged Dockerfile change so non-CI build.sh keeps working at that commit.

What this milestone is not

No hardening overlay (M1.2)
No SilverBrowser/SilverVPN/SilverSync/SilverChat integration (M1.6–1.9)
No installer branding (M1.5)
No update server (M1.10)
No SBOM publication (M1.11)
No signing ceremony / MOK / Secure Boot wiring (separate milestone)

If a change to this directory expands its scope into one of those, push back — the M1.1 gate is intentionally narrow.

5.6 KiB Raw Permalink Blame History Unescape Escape