Some checks failed
Build SilverMetal Linux ISO (reproducibility-gated) / build-and-verify (push) Failing after 11m33s
Run #4260 cleared every harness layer and ran for 18 minutes — past sanity-tests, prepare-build-machine, cowbuilder-setup, local-deps — into 2100_create-debian-packages, where it died on: Could not check validity of signature with '92978A6E195E4921825F7FF0F34F09744E9F5DD9' in '/home/user/derivative-binary/temp_packages_debian_sid/virtualbox_7.2.8-dfsg-1.dsc' as public key missing! …and then *also* hung the runner indefinitely because, on any error, derivative-maker's exception_handler_general detected a TTY (we passed `docker run -t`) and dropped into an interactive `read -p 'Answer? '` prompt that nothing was ever going to answer. The orphan docker run in turn orphaned the act_runner job container, blocking the runner until manual cleanup. Three coordinated fixes, validated end-to-end with docker-side smoke tests on 10.0.0.51: 1. **Non-interactive mode without losing output visibility.** The original architectural goal: keep derivative-maker out of interactive mode (`[ -t 0 ]` must be false) AND keep the build log visible to docker run / Gitea Actions (PTY needed somewhere). Resolution: - `docker run -t` is kept (required for /dev/console to be a real PTY back to docker), but no `-i`, so fd 0 stays /dev/null. - docker-entrypoint.service: `StandardInput=tty-force` → `StandardInput=null` so the service's fd 0 is /dev/null too. Verified inside the container: `[ -t 0 ]` returns false. - entrypoint.sh now wraps the user command with an explicit `> /dev/console 2>&1` redirect before writing it to /etc/docker-entrypoint-cmd. systemd's `StandardOutput=inherit` does NOT propagate PID-1's stdout to services in this PID-1- systemd-in-container topology — the service log was going nowhere visible. /dev/console under `docker run -t` IS the allocated PTY back to docker, so the redirect surfaces the log to the act_runner / Gitea Actions log. - entrypoint.sh's `[ ! -t 0 ] && exit 1` guard removed (it would now always trigger). 2. **debian-keyring for reprepro source-package signature checks.** 2100_create-debian-packages calls dm-reprepro-wrapper includedsc on every .dsc in temp_packages_debian_sid (including virtualbox_*.dsc, even for `--target iso` — see line 114 of that build step). reprepro verifies the dsc signature against the user's GPG keyring; without the maintainer keys it fails. Adds `debian-keyring` to Dockerfile.builder. build-inner.sh now imports debian-keyring.gpg / debian-maintainers.gpg / debian-nonupload.gpg into the user's keyring before running derivative-maker. 3. **BUILDER_IMAGE digest re-pinned.** Built natively on 10.0.0.51 (per memory: never on WSL/aarch64). New digest: sha256:2f680c96…f0db. Smoke-test results (against this exact image): ==> START ← user output reaches docker stdout (keyring present) ← debian-keyring imported successfully STDIN_NOT_TTY ← derivative-maker WILL stay non-interactive ==> END ← clean shutdown docker run exit: 42 ← exit code propagates correctly on failure Files: Dockerfile.builder, systemd-entrypoint/entrypoint.sh, systemd-entrypoint/docker-entrypoint.service, scripts/build.sh, scripts/build-inner.sh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
197 lines
8.5 KiB
Bash
Executable File
197 lines
8.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# SilverMetal Linux — ISO build wrapper.
|
|
#
|
|
# Runs the Kicksecure derivative-maker inside the pinned builder container
|
|
# with the reproducibility levers locked down. This script is the single
|
|
# entry point for both local developer builds and CI — there is no separate
|
|
# CI-only path. If you need to debug, run *this*, not lb directly.
|
|
#
|
|
# Usage:
|
|
# linux/build/scripts/build.sh # writes to linux/build/output/<commit>
|
|
# BUILD_DIR=/tmp/build-a linux/build/scripts/build.sh # override output root
|
|
#
|
|
# Exit codes:
|
|
# 0 ISO produced and SHA256SUMS written
|
|
# 1 argument / environment error
|
|
# 2 derivative-maker submodule missing
|
|
# 3 build failed
|
|
# 4 post-build hash/manifest step failed
|
|
|
|
set -euo pipefail
|
|
|
|
# --- Locate repo root -------------------------------------------------------
|
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../../.." && pwd)"
|
|
cd "${REPO_ROOT}"
|
|
|
|
# --- Pinned builder image ---------------------------------------------------
|
|
# Bumped together with linux/build/docker/Dockerfile.builder. The digest form
|
|
# is required; refusing the tag-only form is what stops a silent host drift.
|
|
#
|
|
# docker-registry.silverlabs.uk is the canonical hostname both inside and
|
|
# outside the LAN — it's the entry that fleet-wide /etc/docker/daemon.json
|
|
# registers as an insecure-registry. The host-style "docker-registry:5000"
|
|
# is *not* DNS-resolvable; do not use it.
|
|
BUILDER_IMAGE="${BUILDER_IMAGE:-docker-registry.silverlabs.uk/silvermetal-builder@sha256:2f680c9629833400573d26a0e25b32b4f13a61a7e4d91543df8983a67801f0db}"
|
|
|
|
if [[ "${BUILDER_IMAGE}" != *"@sha256:"* ]]; then
|
|
echo "build.sh: BUILDER_IMAGE must be pinned by digest, got: ${BUILDER_IMAGE}" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# --- Sanity: submodule present ---------------------------------------------
|
|
if [[ ! -f "linux/build/derivative-maker/.git" && ! -d "linux/build/derivative-maker/.git" ]]; then
|
|
echo "build.sh: linux/build/derivative-maker submodule is not initialised." >&2
|
|
echo " Run: git submodule update --init --recursive" >&2
|
|
exit 2
|
|
fi
|
|
|
|
# --- Compute SOURCE_DATE_EPOCH ---------------------------------------------
|
|
# Order of preference:
|
|
# 1. Explicit env var passed in (CI may set it for cross-runner consistency)
|
|
# 2. config/source-date-epoch.env override (offline rebuilds)
|
|
# 3. git commit timestamp of HEAD (default)
|
|
# shellcheck disable=SC1091
|
|
source linux/build/config/source-date-epoch.env || true
|
|
if [[ -z "${SOURCE_DATE_EPOCH:-}" ]]; then
|
|
if [[ -n "${SOURCE_DATE_EPOCH_OVERRIDE:-}" ]]; then
|
|
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH_OVERRIDE}"
|
|
echo "build.sh: using SOURCE_DATE_EPOCH override = ${SOURCE_DATE_EPOCH}"
|
|
else
|
|
SOURCE_DATE_EPOCH="$(git log -1 --pretty=%ct)"
|
|
fi
|
|
fi
|
|
export SOURCE_DATE_EPOCH
|
|
|
|
# --- Pinned snapshot timestamp ---------------------------------------------
|
|
# shellcheck disable=SC1091
|
|
source linux/build/config/snapshot-pin.env
|
|
export SNAPSHOT_TIMESTAMP
|
|
|
|
# --- Resolve commit & output dir -------------------------------------------
|
|
COMMIT_SHA="$(git rev-parse --short=12 HEAD)"
|
|
BUILD_DIR="${BUILD_DIR:-${REPO_ROOT}/linux/build/output/${COMMIT_SHA}}"
|
|
mkdir -p "${BUILD_DIR}"
|
|
|
|
echo "build.sh: commit=${COMMIT_SHA} epoch=${SOURCE_DATE_EPOCH} snapshot=${SNAPSHOT_TIMESTAMP}"
|
|
echo "build.sh: output -> ${BUILD_DIR}"
|
|
|
|
# --- Mount strategy: local vs CI -------------------------------------------
|
|
# Locally we bind-mount the repo into the build container at the *same*
|
|
# path (self-referential), so internal references work transparently and
|
|
# the inner script doesn't need to care which host it's on.
|
|
#
|
|
# In CI we can't do that. build.sh runs inside a Gitea Actions job
|
|
# container which talks to the *host's* docker daemon via /var/run/docker.sock.
|
|
# Bind-mounting REPO_ROOT (= /workspace/<owner>/<repo>) would resolve
|
|
# against the host filesystem where that path doesn't exist; docker
|
|
# silently creates an empty dir on the host and mounts that, leaving the
|
|
# build container with an empty /work and a confusing "No such file or
|
|
# directory" error on the first config source.
|
|
#
|
|
# The standard fix for that DooD topology is --volumes-from of the parent
|
|
# job container, which inherits its /workspace mount intact. That keeps
|
|
# paths identical inside and outside, so the inner heredoc below is the
|
|
# same in both environments.
|
|
if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
|
|
BIND_ARGS=(--volumes-from "$(hostname)")
|
|
else
|
|
BIND_ARGS=(-v "${REPO_ROOT}:${REPO_ROOT}:rw")
|
|
# If BUILD_DIR lives outside REPO_ROOT (uncommon, but the env-var
|
|
# override allows it), mount it explicitly too.
|
|
if [[ "${BUILD_DIR}" != "${REPO_ROOT}/"* && "${BUILD_DIR}" != "${REPO_ROOT}" ]]; then
|
|
BIND_ARGS+=(-v "${BUILD_DIR}:${BUILD_DIR}:rw")
|
|
fi
|
|
fi
|
|
|
|
# --- Run the build inside the container ------------------------------------
|
|
# This is a systemd-in-container build host. Upstream Kicksecure's
|
|
# derivative-maker assumes a real systemd-managed Debian — its build steps
|
|
# call `systemctl restart approx-derivative-maker.socket`,
|
|
# `systemctl daemon-reload`, etc. and depend on those services *actually*
|
|
# running. Without systemd as PID 1 we'd be playing whack-a-mole with
|
|
# every service derivative-maker starts.
|
|
#
|
|
# Required runtime flags for systemd-in-container:
|
|
# --privileged live-build needs loop devices + chroot mounts
|
|
# --cgroupns=host systemd needs to manage cgroups; with its own
|
|
# namespace it can't see the host hierarchy
|
|
# --tmpfs /run, /run/lock systemd writes runtime state here
|
|
# -v /sys/fs/cgroup:rw the cgroup tree systemd manages
|
|
#
|
|
# `tail -f /dev/null` is NOT used — control flow goes through systemd:
|
|
# entrypoint.sh writes the user command to /etc/docker-entrypoint-cmd,
|
|
# execs systemd, systemd boots docker-entrypoint.service which runs the
|
|
# command, and docker-entrypoint-stop.sh propagates exit code via
|
|
# `systemctl exit <code>` so the container exits with the right status.
|
|
#
|
|
# `-t` (PTY for stdout/stderr) is required for build-log visibility:
|
|
# systemd-as-PID-1 inherits the PTY from the container, and the
|
|
# docker-entrypoint.service propagates it to the build via
|
|
# StandardOutput=inherit. Without -t, the service log goes to the
|
|
# journal only (invisible to docker run / Gitea Actions).
|
|
#
|
|
# Critically, we do NOT add `-i`. Without -i, fd 0 inside the container
|
|
# is /dev/null, not a TTY — and the vendored docker-entrypoint.service
|
|
# explicitly sets `StandardInput=null` so the service inherits no TTY
|
|
# on fd 0 either. That keeps derivative-maker's `[ -t 0 ]` check false
|
|
# so its exception handler stays in non-interactive mode (otherwise any
|
|
# error drops into a `read -p 'Answer? '` prompt and the whole
|
|
# container hangs forever, orphaning docker run and blocking the runner).
|
|
docker run --rm --privileged \
|
|
--cgroupns=host \
|
|
--tmpfs /run \
|
|
--tmpfs /run/lock \
|
|
-v /sys/fs/cgroup:/sys/fs/cgroup:rw \
|
|
--network=host \
|
|
-t \
|
|
"${BIND_ARGS[@]}" \
|
|
-e SOURCE_DATE_EPOCH \
|
|
-e SNAPSHOT_TIMESTAMP \
|
|
-e LC_ALL=C.UTF-8 \
|
|
-e LANG=C.UTF-8 \
|
|
-e TZ=UTC \
|
|
-e REPO_ROOT="${REPO_ROOT}" \
|
|
-e BUILD_DIR="${BUILD_DIR}" \
|
|
"${BUILDER_IMAGE}" \
|
|
bash -c '
|
|
# docker-entrypoint.service runs this as root via systemd, with
|
|
# the env vars captured by entrypoint.sh into
|
|
# /etc/docker-entrypoint-env. We hand workspace ownership to the
|
|
# unprivileged user (uid 1000), then sudo into it for the
|
|
# derivative-maker invocation. derivative-maker uses sudo
|
|
# internally for the bits that need root.
|
|
set -e
|
|
chown -R 1000:1000 "${REPO_ROOT}" "${BUILD_DIR}"
|
|
exec sudo --non-interactive --preserve-env -u user -- \
|
|
"${REPO_ROOT}/linux/build/scripts/build-inner.sh"
|
|
' \
|
|
|| { echo "build.sh: derivative-maker failed"; exit 3; }
|
|
|
|
# --- Hash artefacts ---------------------------------------------------------
|
|
# Run hashing on the host (not in the container) so a busted container image
|
|
# can't tamper with the digests we publish.
|
|
shopt -s nullglob
|
|
ISO_FILES=("${BUILD_DIR}"/*.iso)
|
|
shopt -u nullglob
|
|
if (( ${#ISO_FILES[@]} == 0 )); then
|
|
echo "build.sh: no ISO produced in ${BUILD_DIR}" >&2
|
|
exit 4
|
|
fi
|
|
|
|
(
|
|
cd "${BUILD_DIR}"
|
|
sha256sum -- *.iso > SHA256SUMS
|
|
cp -- "${REPO_ROOT}/linux/build/config/snapshot-pin.env" snapshot-pin.env
|
|
{
|
|
echo "commit=${COMMIT_SHA}"
|
|
echo "source_date_epoch=${SOURCE_DATE_EPOCH}"
|
|
echo "snapshot_timestamp=${SNAPSHOT_TIMESTAMP}"
|
|
echo "builder_image=${BUILDER_IMAGE}"
|
|
echo "host_uname=$(uname -srm)"
|
|
} > BUILD_INFO
|
|
)
|
|
|
|
echo "build.sh: SHA256SUMS:"
|
|
cat "${BUILD_DIR}/SHA256SUMS"
|