Write the Test Manual, and make the suite check it

The test system had grown to seven scripts making five genuinely different
kinds of claim, and nothing said which was which. A recorded transcript and a
byte-for-byte comparison against a second implementation both print [ok  ] and
are worth wildly different amounts, so the fourth manual exists to say so: what
each script can and cannot answer, why every determinism rule is there, how to
add a test, and - the part written nowhere else - where the suite is blind.

That last section is the reason for the document. Three buffer overruns into
adjacent variables were all found by a person using the machine and none by the
suite, the sanitizers cannot see them because emulated Data Memory is one
legitimate host array, and there is no second opinion about the CPU at all. A
document listing only strengths teaches the wrong lesson.

The bullets describing each script move out of the README, so docs.sh now reads
the manual for them, and five more numbers in it are settled from the source
rather than trusted: the shape of the manifest, the xfail count, how many
fixture disks makedisks.sh builds, how large the lint baseline is, and the tool
count in either document. Each of the new checks was broken on purpose and
watched to report before being kept, which is the discipline the manual itself
argues for.

Also drops the stale "70 instructions" from instructiontable.py's docstring.
There are 72, and a number that carries no meaning is better removed than
corrected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW

README.md

@@ -504,45 +504,43 @@ $(BUILD)/%.bin: %.asm
 make test
 ```
 
-The suite assembles and runs every program in `Programs/` and compares the results against recorded output. Tests are defined in `Tests/manifest`, one line per program. To record the current output as the expected result, after you have checked that it is correct:
+Builds the four tools, checks they build clean under strict ISO C, and runs seven scripts.
+`Tests/run.sh` assembles and runs every program in `Programs/` and compares the results
+against recorded output; six more ask the questions a recorded file cannot answer. Between
+them they check the two assemblers against each other byte for byte, the two SBFS
+implementations against each other on the same disk, the disk tool against the format, the
+linter against a fixture written to trip it, the terminal behaviour that a pipe makes
+invisible, and the manuals against the code.
 
 ```
 make bless
 ```
 
-Programs are built inside `Tests/build`, so running the suite never overwrites anything in `Programs/`. To run only some of the tests, call the runner directly with their names:
+Records the current output as the expected result, after you have checked that it is
-
+correct.
-```
-./Tests/run.sh hello 8bitFibonacci
-```
-
-The disk images tests read from are built first by `Tests/makedisks.sh`, using SplitDisk. A
-test that reads one is therefore checked against a filesystem written by different code from
-the same written specification, rather than against itself.
-
-`Tests/run.sh` drives that comparison. Six more scripts run alongside it, and each exists
-because a recorded file cannot answer its question:
-
-- **`Tests/disk.sh`** checks the disk tool on its own: files of every awkward size onto an image and off again, and the things the format says cannot happen refused rather than half done.
-- **`Tests/terminal.sh`** checks what a recorded file cannot see. Piped output is buffered and flushed at exit, so a prompt shown before its answer is asked for and one shown an hour late produce identical files; and key mode only touches a terminal when there is one. Both have gone wrong here, and both were found by a person whose terminal stopped working rather than by anything in this suite. So it runs the emulator under a pseudo-terminal and asks directly: that a prompt arrives before input is read, that a keystroke arrives without Return, that the terminal is handed back however the machine dies, and that suspending and resuming leave it as they found it. It also asks the one question about cycles that a recorded file cannot, since the count is stripped from every one: whether a program on a slow disk slept through the wait or spun on it. Both print the same characters and take the same elapsed time, and only the split between idle and bus cycles tells them apart.
-- **`Tests/native.sh`** checks the assembler that runs on SplitBit against the one that runs on the host, byte for byte, on a boot image and four loadable programs, and then on CosmOS and on itself, and then on the CosmOS that CosmOS built.
-- **`Tests/agree.sh`** checks the two implementations of SBFS against each other rather than each against itself, by building the same disk with SplitDisk and with CosmOS and comparing the images byte for byte. Every field one of them writes and the other only reads is checked there and nowhere else.
-- **`Tests/lint.sh`** checks SplitLint against a fixture written so that every line of it trips exactly one rule. It compares which warning came out and at which line rather than how many came out in total: a count stays right while the thing behind it goes wrong, and breaking one rule's message left the total untouched at twenty three.
-- **`Tests/docs.sh`** checks the manuals against the code: that every instruction has a row and every row is an instruction, that the counts in the headings are right, that every directive is written down, that every service the system implements is described and every service described is implemented, that every routine the manuals promise exists, that CosmOS still fits in the half of the machine its memory map gives it, and that the worked examples still assemble to the bytes printed beside them.
-
-A cycle count is deliberately **not** part of a recorded result. The last line of the emulator's output has the number taken out before anything is compared, keeping only whether the program stopped on its own or ran into its limit, which is behaviour. Two instructions added to CosmOS used to move that number in six unrelated files at once, so a real difference would have arrived in a crowd of meaningless ones. Anything that wants to measure cycles should say so in a test of its own.
-
-To rebuild all four tools with the address and undefined behaviour sanitizers and run the suite under them:
 
 ```
 make sanitize
 ```
 
-This catches reads and writes past the end of an array, use after free, leaks, and undefined arithmetic. It also fills fresh allocations with a junk pattern, which turns a read of uninitialised memory from something that quietly works into something the tests notice. It runs everything `make test` runs, takes about twice as long, and puts the ordinary binaries back when it finishes.
+Rebuilds all four tools with the address and undefined behaviour sanitizers and runs the
+whole suite under them. It catches reads and writes past the end of an array, use after
+free, leaks, and undefined arithmetic, takes about twice as long, and puts the ordinary
+binaries back when it finishes.
+
+Everything is built inside `Tests/build`, so running the suite never overwrites anything in
+`Programs/`. To run only some of the tests, call the runner directly with their names:
+
+```
+./Tests/run.sh hello 8bitFibonacci
+```
+
+What each script can and cannot answer, how to add a test, and where the suite is blind are
+in the [SplitBit Test Manual](SplitBit%20Test%20Manual.md).
 
 ## Documentation:
 
-Three documents, divided by what they are about rather than by who reads them.
+Four documents, divided by what they are about rather than by who reads them.
 
 **[SplitBit Programming Manual](SplitBit%20Programming%20Manual.md)** describes **the machine**: the instruction set, the registers, the vector table, interrupts, devices, the memory controller, the console, storage, and faults. Everything here is true of any SplitBit, whatever is running on it.
 
@@ -550,6 +548,8 @@ Three documents, divided by what they are about rather than by who reads them.
 
 [Programs/CosmOS/README.md](Programs/CosmOS/README.md) describes **the operating system**: its shell, its applications, what a program may ask it for, and the libraries it owns. A different system on the same machine would answer all of that differently, which is why it is documented with the system rather than with the CPU.
 
+**[SplitBit Test Manual](SplitBit%20Test%20Manual.md)** describes **the test suite**: what each of its scripts claims, which of those claims are worth the most, how to add to it, and what it is blind to. It is about this repository rather than about the machine, which is why it comes last.
+
 ## License:
 
 Apache License, Version 2.0. You may obtain a copy at [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0).

SplitBit Test Manual.md

@@ -0,0 +1,463 @@
+# The SplitBit Test Manual
+
+This describes the test suite: what it is for, what each part of it can and cannot answer,
+how to add to it, and where it is blind. It is the fourth of the manuals, and the only one
+about the repository rather than about the machine.
+
+The other three describe things a program can rely on. This one describes the reason to
+believe them.
+
+## What The Suite Claims:
+
+The suite is not one thing. It is seven scripts making five different kinds of claim, and
+knowing which claim you are relying on is the whole point of this document. A recorded
+transcript and a byte-for-byte comparison against a second implementation both print
+`[ok  ]`, and they are worth wildly different amounts.
+
+The strongest thing the suite says is this: **the assembler that runs on SplitBit and the
+assembler that runs on the host produce identical bytes, and the filesystem written by the
+tool and the filesystem written by the machine are identical disks.** Those are two
+programs written from one specification, sharing no code, checking each other. Nothing else
+here is that strong.
+
+The weakest thing it says is that a program prints what it printed last time. That is worth
+having and it is worth having a lot of, but it only ever catches change. It cannot catch a
+thing that was wrong the first time it was recorded.
+
+Everything in between is somewhere on that line.
+
+## Running It:
+
+```
+make test
+```
+
+Builds the four tools, checks they compile under strict ISO C, and runs the scripts in
+order. Takes a few seconds. Everything must pass; there are no expected failures at the
+level of the suite, only tests that record an expected failure of the assembler.
+
+```
+make sanitize
+```
+
+The same suite with the four tools rebuilt under AddressSanitizer and
+UndefinedBehaviorSanitizer. See [The Sanitizer Run](#the-sanitizer-run).
+
+Individual scripts can be run on their own, from anywhere:
+
+```
+./Tests/run.sh                  Every program in the manifest.
+./Tests/run.sh hello waitTest   Only the named ones.
+./Tests/run.sh --bless          Record current output as expected. See below.
+./Tests/disk.sh                 The disk tool against the format.
+./Tests/terminal.sh             The things a recorded file cannot see.
+./Tests/native.sh               The two assemblers against each other.
+./Tests/agree.sh                The two filesystems against each other.
+./Tests/lint.sh                 SplitLint against its fixture and the corpus baseline.
+./Tests/lint.sh --bless         Record the corpus baseline. See below.
+./Tests/docs.sh                 The manuals against the code.
+```
+
+`Tests/makedisks.sh` is not in that list because it checks nothing. It builds the disk
+images the other scripts read, and `run.sh` calls it.
+
+Everything is built into `Tests/build`, which is removed and remade at the start of every
+run. **The suite never writes into `Programs/`.** A binary sitting next to its source came
+from `make`, not from here.
+
+## The Five Kinds Of Check:
+
+### 1. Recorded output
+
+`Tests/run.sh` assembles each program named in `Tests/manifest`, runs it, and compares
+everything it printed against a file in `Tests/expected`. 165 tests, of which 103 run, 35
+only assemble, 16 are expected to fail to assemble, and 11 boot from ROM with no image
+given at all.
+
+This is the bulk of the suite and the shallowest part of it. It answers "did this change"
+and nothing else. A recorded file is only as correct as the day somebody looked at it,
+which is why blessing has a protocol.
+
+The 16 `xfail` tests are worth calling out separately: each one is a piece of source the
+assembler is supposed to refuse, and the recorded result is the refusal message. They catch
+two things at once - an error that stops being detected, and an error message that changes
+without anybody meaning it to.
+
+### 2. A second implementation
+
+`Tests/native.sh` and `Tests/agree.sh` are the two checks that do not compare the code
+against a memory of itself.
+
+`native.sh` assembles the same source with the host assembler and with `Asm.sbx` running on
+the emulated machine, and compares the two binaries byte for byte. **The only honest test of
+an assembler is the bytes it produces:** "it ran" and "the sizes look right" both pass for a
+binary with a label one byte out, which is a program that jumps into the middle of an
+instruction. It does a boot image, four loadable programs, CosmOS, and the assembler
+itself - and then the CosmOS that CosmOS built, which is where it stops, because a second
+generation that matches the first is a fixed point.
+
+`agree.sh` builds the same directory tree twice, once with SplitDisk and once by driving
+CosmOS, and compares the images byte for byte. Every field one implementation writes and
+the other only reads is checked here and nowhere else: which entry a file lands in, which
+block, what a directory's unused fields hold, the version in the superblock, the free
+count. A disagreement in any of those is a disk one of them can read and the other cannot,
+and the usual way that gets discovered is somebody's file coming back wrong months later.
+
+Both scripts depend on both sides being driven in the same order, because both allocate
+first fit and both take the first free entry. Given the same operations in the same
+sequence they should reach the same bytes, and any difference is real rather than an
+artefact of the script.
+
+### 3. Named properties
+
+`Tests/disk.sh` and `Tests/lint.sh` do not compare transcripts. Each check is a named
+assertion that either holds or does not, and the name is the failure message.
+
+`disk.sh` puts files of every awkward size onto an image and takes them off again -
+nothing at all, less than a block, exactly a block, a part block, an exact multiple - and
+then asks for the things the format says cannot happen and requires them to be refused
+rather than half done. Roughly half of its checks are `refuses`, which is the shape
+worth copying: **a tool that never says no is not finished.**
+
+`lint.sh` builds a fixture in which every line trips exactly one rule, and checks which
+warning came out at which line. It used to compare a total, and a total is a number that
+stays right while the thing behind it goes wrong: a change that stopped one rule firing and
+made another fire twice would have passed without a murmur. The fixture is written inside
+the script rather than kept as a file because a reader wants the pattern and the
+expectation side by side.
+
+### 4. What a recorded file cannot see
+
+`Tests/terminal.sh` runs the emulator under a pseudo-terminal, which is what makes its
+questions askable at all. Two whole classes of behaviour are invisible to every other check
+here:
+
+**When something is printed.** Piped output is fully buffered and flushed when the process
+ends, so a prompt shown before its answer is read and a prompt shown an hour late produce
+byte-identical files. A prompt printed *after* the thing it was asking for is invisible to
+the entire rest of the suite.
+
+**What happens to the terminal.** Key mode only touches a terminal when there is one, so
+with input from a file there is nothing to put into another state and nothing to put back.
+A machine that leaves the terminal with echo off passes every other test in this
+repository.
+
+Both of those went wrong here, and both were found by a person whose terminal stopped
+working rather than by anything in this suite. That is why the script exists. It checks
+that a prompt arrives before input is read, that a keystroke arrives without Return, that
+the terminal is handed back however the machine dies - SIGHUP, SIGINT, SIGQUIT, SIGABRT,
+SIGSEGV, SIGTERM - and that suspending and resuming leave it as they found it.
+
+It also asks the one question about cycles that nothing else can, since the count is
+stripped from every recorded result: whether a program on a slow disk slept through the
+wait or spun on it. Both print the same characters and take the same elapsed time. Only the
+split between idle and bus cycles tells them apart.
+
+### 5. The documents against the code
+
+`Tests/docs.sh` settles, every time the suite runs, every claim in the manuals that can be
+settled by looking at the source. Documentation goes stale quietly: an instruction added
+without a table row, or a count in a heading that nobody updated, is wrong in a way nothing
+notices until somebody trusts it.
+
+It checks that every tracked file is plain ASCII; that every link in every document goes
+somewhere, and carries `%20` rather than a raw space; that every instruction has a row and
+every row is an instruction; that the counts in the group headings are right, and the count
+of instructions that take a Data Pointer; that every device class has a row in the Devices
+table; that the vector ranges the manuals quote are the ones the assembler uses; that the
+loadable header table matches the offsets the assembler writes; that every console status
+bit is described; that every service CosmOS implements has a row and every row is
+implemented; that every application the CosmOS README describes exists; that the monitor's
+instruction table is the assembler's, generated rather than typed; that every directive is
+written down; that every routine the manuals promise is defined; that CosmOS still fits in
+the half of the machine its memory map gives it; that the sizes quoted for applications are
+their real sizes; and that the worked examples still assemble to the bytes printed beside
+them.
+
+It also checks this document: that every script the suite runs has a bullet here saying why
+it exists, and that the number quoted is the number there are.
+
+**A manual is a claim somebody may check by typing it.** Everything here is one that gets
+checked automatically instead.
+
+## Determinism:
+
+A test that passes four times out of five is worse than no test, because the failure gets
+attributed to the suite rather than to the code. Several rules exist only to keep results
+identical from one run to the next, and each of them is a bug that already happened.
+
+**The cycle count is stripped from every recorded result.** The last line of the emulator's
+output has the number taken out before anything is compared, keeping only *whether* the
+program stopped on its own or ran into its limit, which is behaviour. Two instructions
+added to CosmOS used to move that number in six unrelated files at once, so a real
+difference would have arrived in a crowd of meaningless ones and had to be picked out by
+hand. Anything that genuinely wants to measure cycles should say so out loud in a test of
+its own, the way `terminal.sh` does.
+
+The substitution is deliberately **not anchored to the start of a line**. A program whose
+last output has no newline on it leaves the cursor mid-line and the halt message is printed
+there, so the count sat inside a line rather than at the head of one and survived.
+`replCalculator` is the program that does that, and it was the only test to churn when the
+machine started charging for memory accesses instead of counting instructions.
+
+**Every program runs with `--fast`.** The emulated clock rate has no bearing on what a
+program prints, and waiting out real time makes the suite slower for nothing.
+
+**A program that never halts is bounded by `--cycles`, not by wall clock.** A cycle limit
+produces the same output on a fast machine and a slow one. A time limit does not.
+
+**Every run has a ten second timeout** on top of that, which is a failure rather than a
+result. A test that hangs says so instead of hanging the suite.
+
+**A scratch disk is removed before every run**, so nothing a test writes can be seen by the
+next one. The emulator makes a blank image in its place.
+
+**The build directory is removed and remade at the start of every run**, so a stale binary
+cannot pass for a fresh one.
+
+**The exit status is recorded with the output.** What a program exits with is part of what
+it does. A program that faults is supposed to exit non-zero, and that should be as pinned
+down as what it printed.
+
+## The Manifest:
+
+`Tests/manifest` is one test per line, fields separated by `|`. Blank lines and lines
+starting with `#` are ignored, and the file is heavily commented by section.
+
+```
+name | source | mode | stdin | limit | disk
+```
+
+**name** is what the test is called and what `Tests/expected/<name>.out` is named after.
+
+**source** is relative to `Programs/`. Everything assembles from there with `Libraries/`
+and `CosmOS/Source/` on the include path. CosmOS is on the path because it owns the
+filesystem library and the service names, which test programs outside it include.
+
+**mode** is one of:
+
+| Mode | What happens |
+| --- | --- |
+| `run` | Assemble, execute, compare all output against the recorded result |
+| `assemble` | Assemble only, for library files with no entry point to run |
+| `xfail` | Assembly is expected to fail, and the message is recorded |
+| `rom` | As `run`, but no image is given, so the machine boots from its ROM |
+
+`xfail` records a known breakage, so that fixing one is noticed and so that an accidental
+new one is too. `rom` is what a machine with no debugger attached does: the emulator
+shadows its built-in stage one into Program Memory and reads the disk for everything else.
+
+**stdin** names a file in `Tests/input`, or `-` for none.
+
+**limit** is a cycle count for programs that never halt on their own, or `-`.
+
+**disk** names an image to attach, or `-` for a machine with no disk, which is most of
+them. Four forms:
+
+| Form | Meaning |
+| --- | --- |
+| `name.img` | Scratch. Removed before the run, so the emulator makes a blank one |
+| `disks/name.img` | A fixture built by `makedisks.sh`, used as it stands |
+| `...:ro` | Attached write protected, so a test can check the *device* bars writes |
+| `...@N` | Given a latency of N cycles, so a test can check the filesystem waits |
+
+The `@N` form deserves a note. Every other test runs with the disk's answer available
+before the next instruction, which is the one condition under which failing to wait looks
+exactly like working.
+
+## Fixture Disks:
+
+`Tests/makedisks.sh` builds 26 images with SplitDisk before anything runs, into
+`Tests/build/disks`. **That is the point of them.** A SplitBit program reading one of these
+is being checked against a filesystem written by different code from the same written
+specification, rather than against itself.
+
+They are shaped to be awkward on purpose. `sbfs.img` has two directory blocks and eight
+files that fill the first one exactly, so everything after that lands in the second and the
+walk from block to block gets exercised rather than assumed. It carries a file longer than
+a block whose pattern repeats every twenty-six bytes, which makes a misplaced block obvious
+to read; an empty file; and a name of exactly the maximum length. `deep.img` has eight
+directory blocks. `chain.img` carries a different payload in each of its two boot slots, so
+a test reading the right word is reading the slot that was chosen rather than whatever
+happened to be left in Program Memory, and `chainAlt.img` is the same disk with the other
+slot selected. `selfboot.img` has stage two in a boot slot and the system as an ordinary
+file, which is the whole chain end to end, and `nosystem.img` has the second stage with
+nothing for it to find, so that a loader which cannot start anything says so rather than
+jumping somewhere.
+
+`cfgbare.img`, `cfgbroken.img` and `cfgfallback.img` are the same disk three ways,
+differing only in `/System/Boot/boot.cfg`. Nothing else about any of them changes, which is
+what makes those tests of the **file** rather than of the machinery under it. That is the
+shape to copy when a fixture is added: change one thing, and keep everything else
+identical to a disk that already works.
+
+The script makes its paths absolute before doing anything else, because it changes
+directory partway down. When that was wrong the failure was not an error - it was a disk
+quietly missing some of the files it was supposed to have, which is a much worse thing to
+debug.
+
+## Recording Expected Output:
+
+```
+make bless
+./Tests/run.sh --bless
+```
+
+This overwrites every file in `Tests/expected` with the current output.
+
+**Blessing unexamined output records a bug as correct, permanently and silently.** It is
+the one operation in this repository that can make the suite confidently wrong, and it does
+it without printing anything alarming. The protocol is:
+
+1. Run the suite and read the diffs. `run.sh` prints the first twenty lines of each.
+2. Account for every changed file. Not "that looks like the change I made" - know why each
+   one moved, and know why the ones that did not move did not.
+3. Bless.
+4. Look at `git diff Tests/expected` before committing, which is the last chance to see a
+   recorded result that should not have changed.
+
+A single test can be re-run by name to see one diff on its own, which is usually faster
+than reading all of them:
+
+```
+./Tests/run.sh cosmosCwd
+```
+
+## The Lint Baseline:
+
+`Tests/lint-baseline.txt` records how many warnings of each rule SplitLint finds in each
+file across the whole corpus - 34 file-and-rule pairs. It is checked on every run, and it
+is checked **in both directions.**
+
+A new warning appearing is a regression. A recorded warning *disappearing* is also
+reported, and that is the half people do not expect: it means either that somebody fixed
+something and did not record it, which is fine and takes one command, or that a rule
+stopped firing, which is not fine at all and is otherwise completely silent.
+
+```
+./Tests/lint.sh --bless
+```
+
+records the corpus as it stands, for when warnings have been deliberately fixed or
+deliberately accepted. Same shape as `run.sh --bless`, and it wants the same care.
+
+A warning that is correct but deliberate should be suppressed in the source rather than
+carried in the baseline, with a comment saying `splitlint: <reason>` or
+`splitlint[rule]: <reason>`. The reason is required, so that a deliberate exception says
+what makes it one. A marker that silences nothing is itself reported, as `dead-suppression`,
+which stops the markers outliving the code they were about.
+
+## The Sanitizer Run:
+
+```
+make sanitize
+```
+
+Rebuilds all four tools with `-fsanitize=address,undefined` and runs **the whole suite**
+under them. It catches reads and writes off the end of an array, use after free, leaks, and
+undefined arithmetic. It also fills fresh allocations with a junk pattern, which is what
+turns a read of uninitialised memory from something that quietly works into something the
+tests notice.
+
+It runs everything because it used to not. It built all four tools sanitized and then ran
+only `run.sh` and `terminal.sh`, so SplitDisk was compiled with the sanitizers and never
+exercised, and `native.sh` - which drives the assembler and the emulator harder than
+anything else here - was skipped entirely. Those are exactly where block arithmetic on disk
+images and buffer indexing in two assemblers live. Adding the other scripts cost about six
+seconds.
+
+If the suite fails, the sanitizer binaries are deliberately left in place so the failing
+case can be run again by hand. Plain `make` puts the normal ones back.
+
+Worth running before a release, and after anything that touches memory handling.
+
+## The Strict Build:
+
+`make test` depends on `make strict`, which compiles every source file with
+`-std=c11 -pedantic -Wall -Wextra -Werror` and throws away the object.
+
+This exists because the README says "the sources are ISO C and build clean under
+`-std=c11 -pedantic` with `-Wall -Wextra`", and that is a claim somebody may check by
+typing it. It was false when the target was written: `realpath` went undeclared under a
+feature test macro that did not reach far enough, which the ordinary `-Os` build never saw.
+
+## Adding A Test:
+
+**A program.** Put the source in `Programs/`, add a line to `Tests/manifest`, run
+`./Tests/run.sh <name>` and read what it printed. If it is right, `make bless` - which
+records every test, so make sure the rest of the suite is clean first. If the program needs
+input, put a file in `Tests/input` and name it in the manifest. If it needs a
+disk, name a scratch image; if it needs a disk with something already on it, add it to
+`makedisks.sh` and name it as `disks/whatever.img`.
+
+**A property of a host tool.** Add a `check` or a `refuses` line to `Tests/disk.sh`. The
+name is the failure message, so write it as a statement of what should be true: "no
+removing an occupied one", not "test 47".
+
+**Something a recorded file cannot see.** `Tests/terminal.sh` is the place, and it is the
+hardest script to add to because each check has to arrange a situation and then observe it
+from outside. Small assembly programs are written inline in the script for this, and only
+this - everything else assembles from the repository.
+
+**A linter rule.** Add its line to the fixture in `Tests/lint.sh` next to the other lines
+for its family, with the expected line number and message. Every line of the fixture exists
+to trip exactly one rule; keep it that way, or a rule that stops working will be masked by
+the one next to it.
+
+**A claim in a manual.** If you write a number, a count, or a list into any of the four
+documents, consider whether `Tests/docs.sh` can settle it from the source. Most of them
+can. The ones already there are all claims that were true when written and quietly stopped
+being.
+
+## Checking A Check:
+
+**A new check is not finished until it has been seen to fail.** The discipline is to break
+the thing on purpose, run the check, watch it report, and put the thing back. This has
+caught several checks in this repository that passed for the wrong reason and would have
+passed for ever.
+
+Two of them are worth remembering:
+
+An agreement test passed because both of its cases were on one disk, and the first save ate
+the file that was the second case's *source*. It compared two things that were both empty.
+Splitting them onto separate disks made it fail, which is when it started being a test.
+
+The lint fixture passed with a rule's message broken, because the total was still
+twenty-three.
+
+A check that has only ever been seen to pass is a check you are trusting on faith. That is
+the same category of thing as an unread recorded result.
+
+## What The Suite Does Not Catch:
+
+Written down because a document that lists only strengths teaches the wrong lesson.
+
+**Buffer overruns into an adjacent variable.** This has happened three times: the shell's
+prompt into its command name table, the assembler's ceilings, and the line editor's `Entry`
+buffer into `TextHead` and `ArenaFree`. **All three were found by a person using the
+machine, and none of them by the suite.** They are hard for it structurally: the write
+succeeds, nothing faults, and the damage shows up somewhere else entirely, at a time that
+depends on the memory map. The sanitizers cannot see them either, because the overrun is in
+emulated Data Memory, which is one legitimate host array from end to end. The only defences
+are reading every `#Reserve` that something copies into, and bounding the loop that fills
+it.
+
+**Anything that needs somebody at the keyboard.** `terminal.sh` gets closer than a recorded
+file can, but it drives a pseudo-terminal on a script. Nobody types slowly, changes their
+mind, resizes the window, or holds a key down. The worst bugs of the last stretch - the
+editor misreading a source file, the machine wedging afterwards, and both terminal failures
+above - all came from somebody sitting down and using CosmOS for an hour.
+
+**Performance.** Cycle counts are deliberately excluded from every recorded result, so
+nothing here notices a program becoming four times slower. That is the correct trade for
+now and it is a real gap.
+
+**The emulator being wrong about the machine.** Every check here compares SplitBit against
+SplitBit. `native.sh` and `agree.sh` are second opinions about the *assembler* and the
+*filesystem*, and there is no second opinion about the CPU at all. Timing, bus behaviour,
+what an undriven port reads as, and what happens when two things want memory at once are
+all decided by one implementation and checked against nothing.
+
+**Whether the design is any good.** The suite can say that CosmOS still fits in its half of
+the machine. It cannot say that the memory map is the right one.
+

Tests/docs.sh

@@ -581,42 +581,104 @@ else:
                 " 0x%s - the buffers are on top of the variables"
                 % (ends - 1, first.group(1).upper()))
 
-# ---- Every test script the suite runs has a bullet saying why it exists ----
+# ---- The Test Manual against the suite it describes ----
 #
-# Two claims in the README went stale at once and neither was noticed: it said FIVE more
+# The suite documents itself, and a document about the suite goes stale the same way every
-# scripts run alongside run.sh when there were six, and "all three tools" when there were
+# other one does. Two claims in the README went stale at once before this check existed and
-# four. Both are the kind of number that is written once, is true for months, and is then
+# neither was noticed: it said FIVE more scripts run alongside run.sh when there were six,
-# quietly wrong - which is the entire subject of this file.
+# and "all three tools" when there were four. Both are the kind of number that is written
+# once, is true for months, and is then quietly wrong - which is the entire subject of this
+# file.
 #
-# run.sh is the driver rather than one of the others, and makedisks.sh makes the images
+# The bullets now live in the Test Manual rather than the README, so that is what is read.
-# rather than checking anything, so neither is counted.
+# makedisks.sh is not counted, because it builds the images rather than checking anything;
-# The repository README rather than CosmOS's, which is what `readme` above holds.
+# run.sh is counted, because the manual describes it alongside the rest.
 rootReadme = open("README.md").read()
+manual = open("SplitBit Test Manual.md").read()
+# THE MANUAL IS WRAPPED, so a number and the noun it counts are regularly on two different
+# lines. Every pattern below runs against a copy with its whitespace flattened.
+flat = re.sub(r"\s+", " ", manual)
+
 scripts = sorted(os.path.basename(p) for p in glob.glob("Tests/*.sh")
-                 if os.path.basename(p) not in ("run.sh", "makedisks.sh"))
+                 if os.path.basename(p) != "makedisks.sh")
 words = {"three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8, "nine": 9}
-said = re.search(r"([A-Za-z]+) more scripts run alongside it", rootReadme)
+said = re.search(r"It is ([a-z]+) scripts making", flat)
 if not said:
-    problems.append("the README no longer says how many scripts run alongside run.sh")
+    problems.append("the Test Manual no longer says how many scripts the suite is")
-elif words.get(said.group(1).lower()) != len(scripts):
+elif words.get(said.group(1)) != len(scripts):
-    problems.append("the README says %s scripts run alongside run.sh, and there are %d: %s"
+    problems.append("the Test Manual says the suite is %s scripts, and there are %d: %s"
                     % (said.group(1), len(scripts), ", ".join(scripts)))
 for name in scripts:
-    if ("`Tests/%s`" % name) not in rootReadme:
+    if ("`Tests/%s`" % name) not in manual:
-        problems.append("Tests/%s runs in the suite and the README does not say what it is"
+        problems.append("Tests/%s runs in the suite and the Test Manual does not say what"
-                        " for" % name)
+                        " it is for" % name)
+
+# ---- The shape of the manifest, which the manual states outright ----
+#
+# Five numbers in one sentence, all of them countable from the file they describe. This is
+# the most quotable thing in the manual and the least likely to be recounted by hand.
+modes = {}
+total = 0
+for line in open("Tests/manifest"):
+    line = line.strip()
+    if not line or line.startswith("#"):
+        continue
+    total += 1
+    fields = line.split("|")
+    if len(fields) > 2:
+        modes[fields[2].strip()] = modes.get(fields[2].strip(), 0) + 1
+
+said = re.search(r"(\d+) tests, of which (\d+) run, (\d+) only assemble,"
+                 r" (\d+) are expected to fail to assemble, and (\d+) boot from ROM", flat)
+if not said:
+    problems.append("the Test Manual no longer states the shape of the manifest")
+else:
+    for index, (what, count) in enumerate((("tests", total),
+                                           ("run tests", modes.get("run", 0)),
+                                           ("assemble-only tests", modes.get("assemble", 0)),
+                                           ("xfail tests", modes.get("xfail", 0)),
+                                           ("rom tests", modes.get("rom", 0)))):
+        if int(said.group(index + 1)) != count:
+            problems.append("the Test Manual says there are %s %s, and there are %d"
+                            % (said.group(index + 1), what, count))
+
+said = re.search(r"The (\d+) `xfail` tests", flat)
+if said and int(said.group(1)) != modes.get("xfail", 0):
+    problems.append("the Test Manual says %s xfail tests in one place and %d in another"
+                    % (said.group(1), modes.get("xfail", 0)))
+
+# ---- And the fixtures and the baseline ----
+disks = len(re.findall(r'format "\$DISKS/', open("Tests/makedisks.sh").read()))
+said = re.search(r"builds (\d+) images with SplitDisk", flat)
+if not said:
+    problems.append("the Test Manual no longer says how many fixture disks are built")
+elif int(said.group(1)) != disks:
+    problems.append("the Test Manual says %s fixture disks are built, and makedisks.sh"
+                    " builds %d" % (said.group(1), disks))
+
+pairs = sum(1 for line in open("Tests/lint-baseline.txt") if line.strip())
+said = re.search(r"(\d+) file-and-rule pairs", flat)
+if not said:
+    problems.append("the Test Manual no longer says how large the lint baseline is")
+elif int(said.group(1)) != pairs:
+    problems.append("the Test Manual says the lint baseline holds %s file and rule pairs,"
+                    " and it holds %d" % (said.group(1), pairs))
 
 # ---- And the tool count is the number of things the makefile builds ----
+#
+# Claimed in both documents, so both are read.
 makefile = open("makefile").read()
 built = re.search(r"^all:(.*)$", makefile, re.M)
 if not built:
     problems.append("the makefile no longer has an all target this can count")
 else:
     tools = len(built.group(1).split())
-    for said in re.findall(r"(?:build|rebuild) (?:all )?(?:the )?([a-z]+) tools", rootReadme):
+    for where, text in (("README", rootReadme), ("Test Manual", manual)):
-        if words.get(said) != tools:
+        for said in re.findall(r"(?:build|rebuild)s? (?:all )?(?:the )?([a-z]+) tools",
-            problems.append("the README says the %s tools and the makefile builds %d"
+                               text, re.I):
-                            % (said, tools))
+            if words.get(said.lower()) != tools:
+                problems.append("the %s says the %s tools and the makefile builds %d"
+                                % (where, said, tools))
 
 # ---- The sizes the CosmOS README quotes for its own programs ----
 #

Tests/instructiontable.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 """The instruction table, as the assembler has it.
 
-The monitor needs the same 70 instructions the assembler does, with the same names and the
+The monitor needs the same instructions the assembler does, with the same names and the
 same lengths, and a disassembler that disagreed with the assembler about how long an
 instruction is would not merely print one thing wrong - it would lose its place and print
 everything after it wrong too. So the table is generated from assembly.c rather than typed