SysAdmin
e260fe1c81
Two coupled changes that unblock the M1.1 iter loop. Both belong in CI;
iter1-15 was wrong to require human-in-the-loop steps to make progress.
1. **CI now builds Dockerfile.builder.**
`.gitea/workflows/build-iso-linux.yaml` grows a `builder-image` job
that runs ahead of `build-and-verify`. It rebuilds the silvermetal-
builder image from `linux/build/docker/Dockerfile.builder`, pushes it
to `docker-registry.silverlabs.uk/silvermetal-builder:m1.1-<sha>` (and
`:latest`), reads the resulting digest off `docker inspect`, and
feeds it forward as a job output. `build-and-verify` consumes that
digest as the `BUILDER_IMAGE` env override that `build.sh` already
honours (and validates is digest-form on line ~37).
That kills the old workflow where every Dockerfile.builder change
required a human to `docker build` + `docker push` on 10.0.0.51 by
hand and then bump the digest in `build.sh` in lockstep. The crash
that triggered this (exit 126 mid-iter16 build run) was a symptom of
that off-CI step still existing.
Both jobs run on the existing `silvermetal-builder` runner; the host
docker daemon is shared via DooD and is already authenticated to
`docker-registry.silverlabs.uk` (linux/build/runner/docker-compose.yml
mounts `/root/.docker:/root/.docker:ro`), so no extra login step.
The hardcoded `BUILDER_IMAGE` digest in `build.sh` stays as the
local-developer / offline-rebuild fallback. Comments updated in
`build.sh`, `Dockerfile.builder`, and `linux/build/README.md` to
match the new flow.
2. **reprepro wrapper for the benign "No priority for X" case.**
Pinned derivative-maker's `2100_create-debian-packages` (with
--target iso) re-imports source packages from snapshot.debian.org
into a local apt repo via `reprepro --basedir … includedsc local
<foo>.dsc`. The local repo's `conf/distributions` ships no
`DscOverride` entries, so any source package whose `.dsc` lacks an
explicit Priority field trips:
No priority for 'X', skipping.
There have been errors!
…and reprepro exits 255. dm-reprepro-wrapper bubbles that up,
2100_create-debian-packages aborts. The current offender is
`virtualbox_*.dsc` (key import is now fine — debian-keyring landed in
commit 4aa59ba — but the priority field gap remains). VirtualBox is
not in SilverMetal's `--target iso` set, so the sane behaviour is
"log it, continue".
New `linux/build/docker/silvermetal-reprepro-wrap.sh` shadows
`/usr/bin/reprepro` at `/usr/local/bin/reprepro` (PATH precedence).
It runs the real reprepro, captures merged stdout+stderr, and:
- if rc != 0 AND every non-blank output line matches one of the
known-benign patterns ("No priority for 'X', skipping." plus the
trailing "There have been errors!"), emits the output, logs one
line of explanation to stderr, and exits 0;
- otherwise emits the output and propagates rc unchanged.
Any *other* reprepro error path stays fatal — only the specific
"No priority for X" pattern is neutralised. `dm-reprepro-wrapper`
resolves `reprepro` via `\$PATH` so it picks up the wrapper
transparently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
117 lines
5.6 KiB
Markdown
117 lines
5.6 KiB
Markdown
# SilverMetal Linux — reproducible ISO build pipeline
|
||
|
||
**Milestone**: Phase 1 / M1.1 — *Kicksecure fork builds reproducibly*.
|
||
**Exit criterion**: two clean builds of the same commit produce a byte-identical SHA256.
|
||
|
||
This directory holds everything that turns a SilverMetal commit into a SilverMetal Linux ISO. M1.1 ships only the *base* (un-hardened) Kicksecure derivative. Hardening overlay, kernel swap, AppArmor profiles, etc. land in M1.2+ and **must not** be added here in the M1.1 PR.
|
||
|
||
## Layout
|
||
|
||
```
|
||
linux/build/
|
||
├── README.md (this file)
|
||
├── derivative-maker/ git submodule -> Kicksecure/derivative-maker
|
||
├── config/
|
||
│ ├── silvermetal-base.conf derivative selection + branding
|
||
│ ├── snapshot-pin.env pinned snapshot.debian.org timestamp
|
||
│ └── source-date-epoch.env optional SOURCE_DATE_EPOCH override
|
||
├── docker/
|
||
│ └── Dockerfile.builder pinned debian:bookworm-slim builder image
|
||
└── scripts/
|
||
├── build.sh wrapper: container run -> derivative-maker
|
||
├── verify-reproducibility.sh build twice, compare SHA256
|
||
└── diagnose-divergence.sh diffoscope on mismatch
|
||
```
|
||
|
||
## How reproducibility is achieved
|
||
|
||
The same levers any deterministic Debian build relies on, stacked together:
|
||
|
||
| Lever | Where it lives |
|
||
|----------------------------------------|-----------------------------------------------|
|
||
| Pinned `snapshot.debian.org` mirror | `config/snapshot-pin.env` |
|
||
| `SOURCE_DATE_EPOCH` from commit time | `scripts/build.sh` (auto) |
|
||
| Pinned builder image (by digest) | `docker/Dockerfile.builder` + `BUILDER_IMAGE` |
|
||
| Deterministic `mksquashfs` flags | `MKSQUASHFS_OPTIONS` in base conf |
|
||
| Pinned upstream toolchain | `derivative-maker/` submodule |
|
||
| `LC_ALL=C.UTF-8`, `TZ=UTC` | `scripts/build.sh` |
|
||
|
||
`diffoscope` is the *diagnostic* tool used by `diagnose-divergence.sh`; the gate itself is plain `sha256sum`.
|
||
|
||
## Reproduce a release locally
|
||
|
||
> Procedure mirrors `docs/trust-model.md` § *Reproducible builds*.
|
||
|
||
Prerequisites: a Linux host (or WSL2) with Docker, ~30 GB free disk, ~8 GB RAM.
|
||
|
||
```bash
|
||
# 1. Clone the repo at the release tag.
|
||
git clone --recurse-submodules https://git.silverlabs.uk/SilverLABS/SilverMetal.git
|
||
cd SilverMetal
|
||
git checkout v1.1.0 # whichever release you want to verify
|
||
|
||
# 2. Build twice and compare. ~60-90 minutes per build.
|
||
linux/build/scripts/verify-reproducibility.sh
|
||
|
||
# 3. Compare against the published release.
|
||
sha256sum -c <(curl -fsSL https://git.silverlabs.uk/SilverLABS/SilverMetal/releases/download/v1.1.0/SHA256SUMS)
|
||
```
|
||
|
||
Mismatch with the published artefact = supply-chain anomaly. Report channel: `security@silverlabs.uk`.
|
||
|
||
## Build once (no reproducibility check)
|
||
|
||
```bash
|
||
linux/build/scripts/build.sh
|
||
# Output lands in linux/build/output/<short-sha>/
|
||
```
|
||
|
||
The wrapper requires `BUILDER_IMAGE` to be pinned by digest. Local dev that hasn't built and pushed an image yet should override:
|
||
|
||
```bash
|
||
BUILDER_IMAGE=docker-registry:5000/silvermetal-builder@sha256:<digest> \
|
||
linux/build/scripts/build.sh
|
||
```
|
||
|
||
## Gitea Actions
|
||
|
||
The CI workflow (`.gitea/workflows/build-iso-linux.yaml`) is the authority for "did this commit build reproducibly?". It:
|
||
|
||
1. Checks out the commit with submodules.
|
||
2. Runs `build.sh` twice in `${GITHUB_WORKSPACE}/build-{a,b}`.
|
||
3. Fails the run if the two ISO SHA256s differ, and uploads a `diffoscope` report as an artefact.
|
||
4. On a tag push, attaches the verified ISO + `SHA256SUMS` + `BUILD_INFO` to a Gitea release.
|
||
|
||
### Self-hosted runner setup
|
||
|
||
The workflow runs on `runs-on: silvermetal-builder`, a self-hosted, privileged-capable Gitea Actions runner. Create it before merging the workflow:
|
||
|
||
1. Provision a Debian 12 VM on the cluster with ≥ 8 vCPU, ≥ 16 GB RAM, ≥ 100 GB disk.
|
||
2. Install Docker (`apt install docker.io`); ensure the runner user can run `docker run --privileged`.
|
||
3. Register `act_runner` against `git.silverlabs.uk` with label `silvermetal-builder`.
|
||
4. Pre-pull the builder image so the first reproducibility run isn't a cold start:
|
||
`docker pull docker-registry:5000/silvermetal-builder:latest`
|
||
5. Cache the apt snapshot in a Docker volume to avoid throttling:
|
||
`docker volume create silvermetal-apt-cache`
|
||
|
||
The runner host name **must not** leak into ISO content. `LC_ALL=C.UTF-8` and a constant `TZ` in the wrapper guard against that, but spot-check with `diagnose-divergence.sh`.
|
||
|
||
## Bumping pinned inputs
|
||
|
||
Each of these is a deliberate, reviewed action — never automate:
|
||
|
||
- **`derivative-maker` submodule** — bump in its own PR, with a verification log showing two clean builds match.
|
||
- **`snapshot-pin.env`** — same procedure.
|
||
- **Builder image (`Dockerfile.builder`)** — edit and commit. CI's `builder-image` job rebuilds, pushes, and feeds the new digest to `build-and-verify` automatically; no manual `docker build`/`docker push` step. The hardcoded `BUILDER_IMAGE` digest fallback in `build.sh` is for local/offline rebuilds only — bump it opportunistically after any merged Dockerfile change so non-CI `build.sh` keeps working at that commit.
|
||
|
||
## What this milestone is *not*
|
||
|
||
- No hardening overlay (M1.2)
|
||
- No SilverBrowser/SilverVPN/SilverSync/SilverChat integration (M1.6–1.9)
|
||
- No installer branding (M1.5)
|
||
- No update server (M1.10)
|
||
- No SBOM publication (M1.11)
|
||
- No signing ceremony / MOK / Secure Boot wiring (separate milestone)
|
||
|
||
If a change to this directory expands its scope into one of those, push back — the M1.1 gate is intentionally narrow.
|