Run #4275's TOC parser worked perfectly — found
/live/filesystem.squashfs as the largest file (983,547,904 bytes,
right where it should be) — but extraction still bailed:
diagnose: largest file in ... is /live/filesystem.squashfs; extracting
diagnose: could not extract rootfs from A
xorriso's -extract action requires -osirrox to be turned on at the
start of the command line; without it, -extract is silently rejected
("OSIRROX is not enabled by default. -osirrox on permits it."). Our
script swallowed stderr and the only signal was the empty output
file.
Two changes:
* Add `-osirrox on` to every -extract invocation.
* On extraction failure, surface the captured stderr (last 30
lines) into the workflow log instead of dropping it. Saves us
one round-trip if the next thing breaks.
ISO layout from the iter27 dump for the record:
/live/filesystem.squashfs 983547904 bytes ← rootfs
/live/initrd.img-... 62929840 bytes
/live/vmlinuz-... 12113856 bytes
/boot/grub/efi.img 3342336 bytes
/EFI/boot/{boot,grub}x64.efi
+ grub modules under /boot/grub/{i386-pc,x86_64-efi}/
The named-path probe for /live/filesystem.squashfs was already first
in the list — it'll succeed cleanly now and we skip the largest-file
fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run #4274 made progress: identical ISO sizes, identical TOC, identical
first 8 KiB — divergence is fully in file payload bytes. But the
diagnostic stalled because extract_squashfs() couldn't find the rootfs:
diagnose: could not extract squashfs from A
diagnose: could not extract squashfs from B
Two reasons to address:
1. The named-path probes only checked /live/filesystem.squashfs,
/casper/filesystem.squashfs and /filesystem.squashfs. Some live-build
configs use /install/... or no canonical name at all.
2. The fallback that used `xorriso -find / -name '*.squashfs'` then
piped to `xorriso -extract` didn't work because xorriso's -find
output quotes paths, and -extract chokes on quotes.
This iteration:
* Adds /install/filesystem.squashfs and /boot/filesystem.squashfs
to the named-path probes.
* Replaces the -find/-name/tail fallback with a generic "biggest
file in the ISO" picker. In a live-build ISO the rootfs payload
is reliably the largest file regardless of what it's called.
Parses lsdl output (with awk, handling spaces in paths and
stripping single-quote framing).
* On extraction failure, dumps the top 20 files by size to stderr
so the workflow log shows what's actually in the ISO — answers
"what should the named-path probe match" for the next iter.
* Always echoes the first 30 lines of toc-a.txt (and the line
count) so we can sanity-check the ISO layout in every run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run #4273 confirmed two things:
1. The reproducibility gate works end-to-end. Both builds produced
ISOs (1077194752 vs 1077202944 bytes — 8 KiB delta, exactly one
squashfs block worth of compressed-payload drift) and the compare
step caught it.
2. diffoscope, run on the whole 1 GB ISO inside the silvermetal-builder
container, gets OOM-killed before producing any output:
diagnose-divergence.sh: line 44: 13 Killed
diffoscope --max-report-size 100000000 --html ... --text ... A.iso B.iso
The host has 19 GiB free, but diffoscope's full recursion through
ISO -> squashfs -> ~thousands of inner files needs more memory than
that for a 1 GB image. Setting --max-report-size only caps the
output, not the working-set.
Rewrite diagnose-divergence.sh to do staged, cheap-to-expensive
analysis:
1. sha256 + sizes (always)
2. xorriso TOC of both ISOs (every node: mode/size/mtime/path) -> diff
3. Pull just live/filesystem.squashfs out of each ISO,
sha256 it + `unsquashfs -ll` it, diff the listings — this is
where the per-file-size signal lives.
4. Targeted diffoscope on the squashfs payload only, with
--max-container-depth 2 + --max-text-report-size 5MB + --no-html
+ a 10-minute timeout. Bounded enough to finish without the OOM.
Drops `set -e` — every step `|| true`s itself so we get partial output
even when one stage fails.
Workflow tail-into-log step now prints the new staged outputs:
* toc-diff.txt — what changed at the ISO level
* sqfs-ls-diff.txt — which inner files have different sizes/mtimes
* sqfs-diff.txt — diffoscope on the squashfs only
* squashfs-sha256.txt
* iso-header-cmp.txt — first-8KB cmp -l for header-level drift
* sizes.txt / sha256.txt / checklist.md as before
Should land us a focused list of "these N files inside the squashfs
have different bytes" — that's what we need to find what's leaking
non-determinism into the build.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vendors Kicksecure derivative-maker as a pinned submodule (18.1.7.4),
adds the wrapper + verify + diagnose scripts, the pinned builder image,
and the reproducibility-gated Gitea Actions workflow. Base flavour only —
no hardening overlay (that's M1.2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
