A tool for breaking things, since doing it by hand went wrong twice

A check that passes proves nothing until it has been seen to fail. Doing that by hand failed twice in two days, and BOTH TIMES IT LOOKED LIKE A RESULT - the suite ran, went green, and read exactly like "this check does not catch that". Once the edit produced code that would not compile, make failed, the exit status was not looked at, and the previous binary ran the suite. Once the anchor was right and the filename was wrong, so nothing was edited at all. Neither had anything to do with header dependencies, which have always worked: DEPFLAGS is -MMD -MP and every .d is included. What was missing was a harness that refuses to report a result it did not earn. So Tests/break.sh checks every step of its own work and treats anything unexpected as a hard error rather than a green run. Not finding the break is the answer it exists to give, and it is worthless if it can also be the answer when the break never happened. It restores the file on the way out, including on an interrupt. It is not in the suite and docs.sh does not count it, for the reason makedisks.sh is not counted turned round - but being left out of the count is not being left out of the manual, and that gap is where a script goes undocumented for months. So docs.sh now requires both of them to be described, and caught this one being missing. Also: video.sh reads the fixture disks and does not build them, so after make sanitize clears the build directory it reported SEVEN product-looking failures for a missing file. It builds them now and says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW
2026-09-02 13:19:22 -04:00
parent 9eed23120f
commit f8c3db5d56
4 changed files with 162 additions and 2 deletions
@@ -70,6 +70,28 @@ Individual scripts can be run on their own, from anywhere:
 `Tests/makedisks.sh` is not in that list because it checks nothing. It builds the disk
 images the other scripts read, and `run.sh` calls it.

+`Tests/break.sh` is not in it either, for the reason turned round: it checks that a check
+works. Run it by hand when a check is written, not as part of `make test`.
+
+```
+./Tests/break.sh <file> <anchor> <replacement> <suite> [suite ...]
+```
+
+It replaces the anchor - which must appear exactly once - rebuilds, runs the suites, restores
+the file, and reports whether anything failed. **A suite failing is the good outcome.**
+
+A check that passes proves nothing until it has been seen to fail, and doing that by hand
+went wrong twice in two days. Both times it looked like a result: the suite ran, went green,
+and read exactly like "this check does not catch that". Once the edit did not compile, `make`
+failed, the exit status was not looked at, and the previous binary ran the suite. Once the
+anchor was right and the filename was wrong, so nothing was edited at all.
+
+Neither had anything to do with the build system; headers have always rebuilt what depends on
+them. What was missing was a harness that refuses to report a result it did not earn, so
+every step is checked and anything unexpected is a hard error rather than a green run. **Not
+finding the break is the answer this tool exists to give, and it is worthless if it can also
+be the answer when the break never happened.**
+
 Everything is built into `Tests/build`, which is removed and remade at the start of every
 run. **The suite never writes into `Programs/`.** A binary sitting next to its source came
 from `make`, not from here.