Write the Test Manual, and make the suite check it

The test system had grown to seven scripts making five genuinely different
kinds of claim, and nothing said which was which. A recorded transcript and a
byte-for-byte comparison against a second implementation both print [ok  ] and
are worth wildly different amounts, so the fourth manual exists to say so: what
each script can and cannot answer, why every determinism rule is there, how to
add a test, and - the part written nowhere else - where the suite is blind.

That last section is the reason for the document. Three buffer overruns into
adjacent variables were all found by a person using the machine and none by the
suite, the sanitizers cannot see them because emulated Data Memory is one
legitimate host array, and there is no second opinion about the CPU at all. A
document listing only strengths teaches the wrong lesson.

The bullets describing each script move out of the README, so docs.sh now reads
the manual for them, and five more numbers in it are settled from the source
rather than trusted: the shape of the manifest, the xfail count, how many
fixture disks makedisks.sh builds, how large the lint baseline is, and the tool
count in either document. Each of the new checks was broken on purpose and
watched to report before being kept, which is the discipline the manual itself
argues for.

Also drops the stale "70 instructions" from instructiontable.py's docstring.
There are 72, and a number that carries no meaning is better removed than
corrected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW

README.md

@@ -504,45 +504,43 @@ $(BUILD)/%.bin: %.asm
 make test
 ```
 
-The suite assembles and runs every program in `Programs/` and compares the results against recorded output. Tests are defined in `Tests/manifest`, one line per program. To record the current output as the expected result, after you have checked that it is correct:
+Builds the four tools, checks they build clean under strict ISO C, and runs seven scripts.
+`Tests/run.sh` assembles and runs every program in `Programs/` and compares the results
+against recorded output; six more ask the questions a recorded file cannot answer. Between
+them they check the two assemblers against each other byte for byte, the two SBFS
+implementations against each other on the same disk, the disk tool against the format, the
+linter against a fixture written to trip it, the terminal behaviour that a pipe makes
+invisible, and the manuals against the code.
 
 ```
 make bless
 ```
 
-Programs are built inside `Tests/build`, so running the suite never overwrites anything in `Programs/`. To run only some of the tests, call the runner directly with their names:
-
-```
-./Tests/run.sh hello 8bitFibonacci
-```
-
-The disk images tests read from are built first by `Tests/makedisks.sh`, using SplitDisk. A
-test that reads one is therefore checked against a filesystem written by different code from
-the same written specification, rather than against itself.
-
-`Tests/run.sh` drives that comparison. Six more scripts run alongside it, and each exists
-because a recorded file cannot answer its question:
-
-- **`Tests/disk.sh`** checks the disk tool on its own: files of every awkward size onto an image and off again, and the things the format says cannot happen refused rather than half done.
-- **`Tests/terminal.sh`** checks what a recorded file cannot see. Piped output is buffered and flushed at exit, so a prompt shown before its answer is asked for and one shown an hour late produce identical files; and key mode only touches a terminal when there is one. Both have gone wrong here, and both were found by a person whose terminal stopped working rather than by anything in this suite. So it runs the emulator under a pseudo-terminal and asks directly: that a prompt arrives before input is read, that a keystroke arrives without Return, that the terminal is handed back however the machine dies, and that suspending and resuming leave it as they found it. It also asks the one question about cycles that a recorded file cannot, since the count is stripped from every one: whether a program on a slow disk slept through the wait or spun on it. Both print the same characters and take the same elapsed time, and only the split between idle and bus cycles tells them apart.
-- **`Tests/native.sh`** checks the assembler that runs on SplitBit against the one that runs on the host, byte for byte, on a boot image and four loadable programs, and then on CosmOS and on itself, and then on the CosmOS that CosmOS built.
-- **`Tests/agree.sh`** checks the two implementations of SBFS against each other rather than each against itself, by building the same disk with SplitDisk and with CosmOS and comparing the images byte for byte. Every field one of them writes and the other only reads is checked there and nowhere else.
-- **`Tests/lint.sh`** checks SplitLint against a fixture written so that every line of it trips exactly one rule. It compares which warning came out and at which line rather than how many came out in total: a count stays right while the thing behind it goes wrong, and breaking one rule's message left the total untouched at twenty three.
-- **`Tests/docs.sh`** checks the manuals against the code: that every instruction has a row and every row is an instruction, that the counts in the headings are right, that every directive is written down, that every service the system implements is described and every service described is implemented, that every routine the manuals promise exists, that CosmOS still fits in the half of the machine its memory map gives it, and that the worked examples still assemble to the bytes printed beside them.
-
-A cycle count is deliberately **not** part of a recorded result. The last line of the emulator's output has the number taken out before anything is compared, keeping only whether the program stopped on its own or ran into its limit, which is behaviour. Two instructions added to CosmOS used to move that number in six unrelated files at once, so a real difference would have arrived in a crowd of meaningless ones. Anything that wants to measure cycles should say so in a test of its own.
-
-To rebuild all four tools with the address and undefined behaviour sanitizers and run the suite under them:
+Records the current output as the expected result, after you have checked that it is
+correct.
 
 ```
 make sanitize
 ```
 
-This catches reads and writes past the end of an array, use after free, leaks, and undefined arithmetic. It also fills fresh allocations with a junk pattern, which turns a read of uninitialised memory from something that quietly works into something the tests notice. It runs everything `make test` runs, takes about twice as long, and puts the ordinary binaries back when it finishes.
+Rebuilds all four tools with the address and undefined behaviour sanitizers and runs the
+whole suite under them. It catches reads and writes past the end of an array, use after
+free, leaks, and undefined arithmetic, takes about twice as long, and puts the ordinary
+binaries back when it finishes.
+
+Everything is built inside `Tests/build`, so running the suite never overwrites anything in
+`Programs/`. To run only some of the tests, call the runner directly with their names:
+
+```
+./Tests/run.sh hello 8bitFibonacci
+```
+
+What each script can and cannot answer, how to add a test, and where the suite is blind are
+in the [SplitBit Test Manual](SplitBit%20Test%20Manual.md).
 
 ## Documentation:
 
-Three documents, divided by what they are about rather than by who reads them.
+Four documents, divided by what they are about rather than by who reads them.
 
 **[SplitBit Programming Manual](SplitBit%20Programming%20Manual.md)** describes **the machine**: the instruction set, the registers, the vector table, interrupts, devices, the memory controller, the console, storage, and faults. Everything here is true of any SplitBit, whatever is running on it.
 
@@ -550,6 +548,8 @@ Three documents, divided by what they are about rather than by who reads them.
 
 [Programs/CosmOS/README.md](Programs/CosmOS/README.md) describes **the operating system**: its shell, its applications, what a program may ask it for, and the libraries it owns. A different system on the same machine would answer all of that differently, which is why it is documented with the system rather than with the CPU.
 
+**[SplitBit Test Manual](SplitBit%20Test%20Manual.md)** describes **the test suite**: what each of its scripts claims, which of those claims are worth the most, how to add to it, and what it is blind to. It is about this repository rather than about the machine, which is why it comes last.
+
 ## License:
 
 Apache License, Version 2.0. You may obtain a copy at [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0).