M1.1 ISO reproducible build — bring in fresh eyes after 37 failed iterations #1

New Issue

SilverLABS · 2026-05-14T18:00:08Z

SilverLABS commented

2026-05-14 18:00:08 +00:00

Context

The M1.1 ISO reproducible-build effort has gone through iterations 18 → 37 over the last month (2026-04-14 → 2026-05-14) with 37 consecutive failed CI runs. The iteration pattern suggests we've exhausted the obvious fixes (xorriso/squashfs dating, Rock Ridge TF date patching, diffoscope inside builder, cgroupns container self-detection) and are now chasing residual non-determinism.

Why this matters

Reproducible builds are a foundational trust property for the OS product line. Each iteration costs a CI run and engineering time, and the failure cluster has not narrowed.

Proposed actions

Pull in a second pair of eyes — ideally someone with prior reproducible-builds (rb.debian.net / r-b.org) experience
Run a full diffoscope diff between two consecutive failed builds and attach the canonical output here, so we can see what is still varying rather than guessing
Audit non-obvious sources of entropy: build host kernel uname, locale, timezone, /proc mount UUIDs leaking into initramfs, glibc nss host order
Decide a circuit-breaker: if iter40 fails, pause the effort and write up a "what we tried" doc before continuing

Reference

Identified in monthly executive report 2026-05-14. Highest-severity item this cycle.

Filed by Mr Tickles on behalf of the monthly project review.

## Context The M1.1 ISO reproducible-build effort has gone through iterations 18 → 37 over the last month (2026-04-14 → 2026-05-14) with **37 consecutive failed CI runs**. The iteration pattern suggests we've exhausted the obvious fixes (xorriso/squashfs dating, Rock Ridge TF date patching, diffoscope inside builder, cgroupns container self-detection) and are now chasing residual non-determinism. ## Why this matters Reproducible builds are a foundational trust property for the OS product line. Each iteration costs a CI run and engineering time, and the failure cluster has not narrowed. ## Proposed actions - [ ] Pull in a second pair of eyes — ideally someone with prior reproducible-builds (rb.debian.net / r-b.org) experience - [ ] Run a full diffoscope diff between two consecutive failed builds and attach the canonical output here, so we can see *what* is still varying rather than guessing - [ ] Audit non-obvious sources of entropy: build host kernel uname, locale, timezone, /proc mount UUIDs leaking into initramfs, glibc nss host order - [ ] Decide a circuit-breaker: if iter40 fails, pause the effort and write up a "what we tried" doc before continuing ## Reference Identified in monthly executive report 2026-05-14. Highest-severity item this cycle. --- *Filed by Mr Tickles on behalf of the monthly project review.*

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SilverLABS/SilverMetal#1