From 373454ec002e93ab3ea65b6be450c4ee2517be20 Mon Sep 17 00:00:00 2001 From: Anachronaut Date: Tue, 1 Sep 2026 11:22:43 -0400 Subject: [PATCH] A fresh disk for every test, fixtures included A fixture built by makedisks.sh was handed to each test where it lay. Twenty four tests name disks/cosmos.img and several of them write to one, so a test could hand the next one a disk with its leavings on. romBoot is what found it. Its recorded output described a directory that selfBoot had made earlier in the same run, so it passed in a full run and failed on its own - which is the worst way round for a test to be wrong, because the form nobody runs is the one telling the truth. Its recording now says "made" like selfBoot's, which is what running the same input on the same disk should always have said. Fixed as a class rather than as an instance: run.sh copies a fixture before attaching it, the same way it already removed a scratch image. Then every one of the 138 run and rom tests was run on its own to see whether anything else was leaning on what ran before it. Nothing was, before or after. Also, cosmosEditKeys.in was written by Python's write_text, which encodes as UTF-8, so every key byte was 0xC2 and then the key. The test passed anyway, because the shell ignores a byte it has no use for - a fixture working for a reason it was not built on, which is exactly the thing that stops working without anybody touching it. Written as bytes now; the recording is unchanged, which is the proof the stray bytes were being ignored. docs.sh is what caught that, and it turns out to draw the line in the right place by construction: a deliberately binary fixture does not decode as UTF-8 and is skipped, while one that is accidentally UTF-8 decodes and is reported. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW --- SplitBit Test Manual.md | 11 ++++++++++- Tests/expected/romBoot.out | 2 +- Tests/input/cosmosEditKeys.in | 10 +++++----- Tests/run.sh | 28 ++++++++++++++++++++-------- 4 files changed, 36 insertions(+), 15 deletions(-) diff --git a/SplitBit Test Manual.md b/SplitBit Test Manual.md index dbbf339..0ee7f1a 100644 --- a/SplitBit Test Manual.md +++ b/SplitBit Test Manual.md @@ -350,7 +350,7 @@ them. Four forms: | Form | Meaning | | --- | --- | | `name.img` | Scratch. Removed before the run, so the emulator makes a blank one | -| `disks/name.img` | A fixture built by `makedisks.sh`, used as it stands | +| `disks/name.img` | A fixture built by `makedisks.sh`, copied fresh for the run | | `...:ro` | Attached write protected, so a test can check the *device* bars writes | | `...@N` | Given a latency of N cycles, so a test can check the filesystem waits | @@ -358,6 +358,15 @@ The `@N` form deserves a note. Every other test runs with the disk's answer avai before the next instruction, which is the one condition under which failing to wait looks exactly like working. +**Both kinds of disk are fresh for every test**, and the fixture was not always. It used to +be handed to each test where it lay, and twenty four tests name `disks/cosmos.img` while +several of them write to one. `romBoot` was the test that found it: its recorded output +described a directory that `selfBoot` had made earlier in the same run, so it passed in a +full run and failed on its own. **That is the worst way round for a test to be wrong** - the +form nobody runs is the one that tells the truth - and the shape of it is worth naming, +because it is the same shape as a program that works only because of what ran before it. +The whole suite was checked one test at a time afterwards, and that was the only one. + **keys** names a file in `Tests/input` to be fed to the console as a *keyboard* rather than as standard input, and the difference between those is the whole reason the field exists. diff --git a/Tests/expected/romBoot.out b/Tests/expected/romBoot.out index bc027b8..bf05c04 100644 --- a/Tests/expected/romBoot.out +++ b/Tests/expected/romBoot.out @@ -9,7 +9,7 @@ deleted it and it is gone finished > mkdir Notes -cannot make that: check the path, the name, and whether it is taken +made > cd Notes /Notes> dir 0 files diff --git a/Tests/input/cosmosEditKeys.in b/Tests/input/cosmosEditKeys.in index 8a8a00e..d223682 100644 --- a/Tests/input/cosmosEditKeys.in +++ b/Tests/input/cosmosEditKeys.in @@ -1,7 +1,7 @@ -eco ok„ƒƒh -Xecho ok„† -echo o„…k +eco ok„ƒƒh +Xecho ok„† +echo o„…k echo okX -echo k‚o -Xecho aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa„†…! +echo k‚o +Xecho aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa„†…! exit diff --git a/Tests/run.sh b/Tests/run.sh index 493ce74..1afd53f 100755 --- a/Tests/run.sh +++ b/Tests/run.sh @@ -248,9 +248,9 @@ while IFS='|' read -r name src mode stdin limit disk keys; do EMUARGS+=(--keyboard "$INPUT/$keys") fi [ "$limit" != "-" ] && EMUARGS+=(--cycles "$limit") - # A disk starts fresh for every run, so a test cannot pass because of what a - # previous one left lying on it. The emulator makes the image if it is - # missing, which is what removing it first arranges for. + # A disk starts fresh for every test, so a test cannot pass because of what a + # previous one left lying on it. How that is arranged differs between a scratch + # image and a built fixture, and both are below. # ---- More than one, separated by a plus ---- # # The machine has four drives, so a test may name up to four images and they @@ -277,12 +277,24 @@ while IFS='|' read -r name src mode stdin limit disk keys; do case "$DISKFILE" in *@*) DISKWAIT="${DISKFILE##*@}"; DISKFILE="${DISKFILE%@*}" ;; esac - # A name with a directory in it is one of the images makedisks.sh built, - # and is used as it stands. A bare name is scratch: it is removed first so - # that nothing a test writes can be seen by the next one, and the emulator - # makes a blank image in its place. + # ---- A fresh disk for every test, whichever kind it is ---- + # + # A bare name is scratch: it is removed, and the emulator makes a blank image + # in its place. + # + # A name with a directory in it is one of the images makedisks.sh built, and + # it is COPIED rather than used where it lies. Twenty four tests name + # disks/cosmos.img and several of them write to it, so a test used to be able + # to hand the next one a disk with its leavings on. That is not hypothetical: + # romBoot's recorded output described a directory that selfBoot had made, so + # it passed in a full run and failed on its own, which is the worst way round + # for a test to be wrong. case "$DISKFILE" in - */*) ;; + */*) + mkdir -p "$BUILD/.fixture" + cp "$BUILD/$DISKFILE" "$BUILD/.fixture/${DISKFILE##*/}" + DISKFILE=".fixture/${DISKFILE##*/}" + ;; *) rm -f "$BUILD/$DISKFILE" ;; esac EMUARGS+=(--disk "$BUILD/$DISKFILE")