SysAdmin
4aa59ba633
Some checks failed
Build SilverMetal Linux ISO (reproducibility-gated) / build-and-verify (push) Failing after 11m33s
Run #4260 cleared every harness layer and ran for 18 minutes — past sanity-tests, prepare-build-machine, cowbuilder-setup, local-deps — into 2100_create-debian-packages, where it died on: Could not check validity of signature with '92978A6E195E4921825F7FF0F34F09744E9F5DD9' in '/home/user/derivative-binary/temp_packages_debian_sid/virtualbox_7.2.8-dfsg-1.dsc' as public key missing! …and then *also* hung the runner indefinitely because, on any error, derivative-maker's exception_handler_general detected a TTY (we passed `docker run -t`) and dropped into an interactive `read -p 'Answer? '` prompt that nothing was ever going to answer. The orphan docker run in turn orphaned the act_runner job container, blocking the runner until manual cleanup. Three coordinated fixes, validated end-to-end with docker-side smoke tests on 10.0.0.51: 1. **Non-interactive mode without losing output visibility.** The original architectural goal: keep derivative-maker out of interactive mode (`[ -t 0 ]` must be false) AND keep the build log visible to docker run / Gitea Actions (PTY needed somewhere). Resolution: - `docker run -t` is kept (required for /dev/console to be a real PTY back to docker), but no `-i`, so fd 0 stays /dev/null. - docker-entrypoint.service: `StandardInput=tty-force` → `StandardInput=null` so the service's fd 0 is /dev/null too. Verified inside the container: `[ -t 0 ]` returns false. - entrypoint.sh now wraps the user command with an explicit `> /dev/console 2>&1` redirect before writing it to /etc/docker-entrypoint-cmd. systemd's `StandardOutput=inherit` does NOT propagate PID-1's stdout to services in this PID-1- systemd-in-container topology — the service log was going nowhere visible. /dev/console under `docker run -t` IS the allocated PTY back to docker, so the redirect surfaces the log to the act_runner / Gitea Actions log. - entrypoint.sh's `[ ! -t 0 ] && exit 1` guard removed (it would now always trigger). 2. **debian-keyring for reprepro source-package signature checks.** 2100_create-debian-packages calls dm-reprepro-wrapper includedsc on every .dsc in temp_packages_debian_sid (including virtualbox_*.dsc, even for `--target iso` — see line 114 of that build step). reprepro verifies the dsc signature against the user's GPG keyring; without the maintainer keys it fails. Adds `debian-keyring` to Dockerfile.builder. build-inner.sh now imports debian-keyring.gpg / debian-maintainers.gpg / debian-nonupload.gpg into the user's keyring before running derivative-maker. 3. **BUILDER_IMAGE digest re-pinned.** Built natively on 10.0.0.51 (per memory: never on WSL/aarch64). New digest: sha256:2f680c96…f0db. Smoke-test results (against this exact image): ==> START ← user output reaches docker stdout (keyring present) ← debian-keyring imported successfully STDIN_NOT_TTY ← derivative-maker WILL stay non-interactive ==> END ← clean shutdown docker run exit: 42 ← exit code propagates correctly on failure Files: Dockerfile.builder, systemd-entrypoint/entrypoint.sh, systemd-entrypoint/docker-entrypoint.service, scripts/build.sh, scripts/build-inner.sh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SilverMetal Linux — reproducible ISO build pipeline
Milestone: Phase 1 / M1.1 — Kicksecure fork builds reproducibly. Exit criterion: two clean builds of the same commit produce a byte-identical SHA256.
This directory holds everything that turns a SilverMetal commit into a SilverMetal Linux ISO. M1.1 ships only the base (un-hardened) Kicksecure derivative. Hardening overlay, kernel swap, AppArmor profiles, etc. land in M1.2+ and must not be added here in the M1.1 PR.
Layout
linux/build/
├── README.md (this file)
├── derivative-maker/ git submodule -> Kicksecure/derivative-maker
├── config/
│ ├── silvermetal-base.conf derivative selection + branding
│ ├── snapshot-pin.env pinned snapshot.debian.org timestamp
│ └── source-date-epoch.env optional SOURCE_DATE_EPOCH override
├── docker/
│ └── Dockerfile.builder pinned debian:bookworm-slim builder image
└── scripts/
├── build.sh wrapper: container run -> derivative-maker
├── verify-reproducibility.sh build twice, compare SHA256
└── diagnose-divergence.sh diffoscope on mismatch
How reproducibility is achieved
The same levers any deterministic Debian build relies on, stacked together:
| Lever | Where it lives |
|---|---|
Pinned snapshot.debian.org mirror |
config/snapshot-pin.env |
SOURCE_DATE_EPOCH from commit time |
scripts/build.sh (auto) |
| Pinned builder image (by digest) | docker/Dockerfile.builder + BUILDER_IMAGE |
Deterministic mksquashfs flags |
MKSQUASHFS_OPTIONS in base conf |
| Pinned upstream toolchain | derivative-maker/ submodule |
LC_ALL=C.UTF-8, TZ=UTC |
scripts/build.sh |
diffoscope is the diagnostic tool used by diagnose-divergence.sh; the gate itself is plain sha256sum.
Reproduce a release locally
Procedure mirrors
docs/trust-model.md§ Reproducible builds.
Prerequisites: a Linux host (or WSL2) with Docker, ~30 GB free disk, ~8 GB RAM.
# 1. Clone the repo at the release tag.
git clone --recurse-submodules https://git.silverlabs.uk/SilverLABS/SilverMetal.git
cd SilverMetal
git checkout v1.1.0 # whichever release you want to verify
# 2. Build twice and compare. ~60-90 minutes per build.
linux/build/scripts/verify-reproducibility.sh
# 3. Compare against the published release.
sha256sum -c <(curl -fsSL https://git.silverlabs.uk/SilverLABS/SilverMetal/releases/download/v1.1.0/SHA256SUMS)
Mismatch with the published artefact = supply-chain anomaly. Report channel: security@silverlabs.uk.
Build once (no reproducibility check)
linux/build/scripts/build.sh
# Output lands in linux/build/output/<short-sha>/
The wrapper requires BUILDER_IMAGE to be pinned by digest. Local dev that hasn't built and pushed an image yet should override:
BUILDER_IMAGE=docker-registry:5000/silvermetal-builder@sha256:<digest> \
linux/build/scripts/build.sh
Gitea Actions
The CI workflow (.gitea/workflows/build-iso-linux.yaml) is the authority for "did this commit build reproducibly?". It:
- Checks out the commit with submodules.
- Runs
build.shtwice in${GITHUB_WORKSPACE}/build-{a,b}. - Fails the run if the two ISO SHA256s differ, and uploads a
diffoscopereport as an artefact. - On a tag push, attaches the verified ISO +
SHA256SUMS+BUILD_INFOto a Gitea release.
Self-hosted runner setup
The workflow runs on runs-on: silvermetal-builder, a self-hosted, privileged-capable Gitea Actions runner. Create it before merging the workflow:
- Provision a Debian 12 VM on the cluster with ≥ 8 vCPU, ≥ 16 GB RAM, ≥ 100 GB disk.
- Install Docker (
apt install docker.io); ensure the runner user can rundocker run --privileged. - Register
act_runneragainstgit.silverlabs.ukwith labelsilvermetal-builder. - Pre-pull the builder image so the first reproducibility run isn't a cold start:
docker pull docker-registry:5000/silvermetal-builder:latest - Cache the apt snapshot in a Docker volume to avoid throttling:
docker volume create silvermetal-apt-cache
The runner host name must not leak into ISO content. LC_ALL=C.UTF-8 and a constant TZ in the wrapper guard against that, but spot-check with diagnose-divergence.sh.
Bumping pinned inputs
Each of these is a deliberate, reviewed action — never automate:
derivative-makersubmodule — bump in its own PR, with a verification log showing two clean builds match.snapshot-pin.env— same procedure.- Builder image (
Dockerfile.builderdigest) — rebuild, push, updateBUILDER_IMAGEinbuild.sh, run reproducibility check, commit all four together.
What this milestone is not
- No hardening overlay (M1.2)
- No SilverBrowser/SilverVPN/SilverSync/SilverChat integration (M1.6–1.9)
- No installer branding (M1.5)
- No update server (M1.10)
- No SBOM publication (M1.11)
- No signing ceremony / MOK / Secure Boot wiring (separate milestone)
If a change to this directory expands its scope into one of those, push back — the M1.1 gate is intentionally narrow.