SplitBit-Emulator/SplitBit Test Manual.md

# The SplitBit Test Manual

This describes the test suite: what it is for, what each part of it can and cannot answer,
how to add to it, and where it is blind. It is the fourth of the manuals, and the only one
about the repository rather than about the machine.

The other three describe things a program can rely on. This one describes the reason to
believe them.

## What The Suite Claims:

The suite is not one thing. It is seven scripts making five different kinds of claim, and
knowing which claim you are relying on is the whole point of this document. A recorded
transcript and a byte-for-byte comparison against a second implementation both print
`[ok  ]`, and they are worth wildly different amounts.

The strongest thing the suite says is this: **the assembler that runs on SplitBit and the
assembler that runs on the host produce identical bytes, and the filesystem written by the
tool and the filesystem written by the machine are identical disks.** Those are two
programs written from one specification, sharing no code, checking each other. Nothing else
here is that strong.

The weakest thing it says is that a program prints what it printed last time. That is worth
having and it is worth having a lot of, but it only ever catches change. It cannot catch a
thing that was wrong the first time it was recorded.

Everything in between is somewhere on that line.

## Running It:

```
make test
```

Builds the four tools, checks they compile under strict ISO C, and runs the scripts in
order. Takes a few seconds. Everything must pass; there are no expected failures at the
level of the suite, only tests that record an expected failure of the assembler.

```
make sanitize
```

The same suite with the four tools rebuilt under AddressSanitizer and
UndefinedBehaviorSanitizer. See [The Sanitizer Run](#the-sanitizer-run).

Individual scripts can be run on their own, from anywhere:

```
./Tests/run.sh                  Every program in the manifest.
./Tests/run.sh hello waitTest   Only the named ones.
./Tests/run.sh --bless          Record current output as expected. See below.
./Tests/disk.sh                 The disk tool against the format.
./Tests/terminal.sh             The things a recorded file cannot see.
./Tests/native.sh               The two assemblers against each other.
./Tests/agree.sh                The two filesystems against each other.
./Tests/lint.sh                 SplitLint against its fixture and the corpus baseline.
./Tests/lint.sh --bless         Record the corpus baseline. See below.
./Tests/docs.sh                 The manuals against the code.
```

`Tests/makedisks.sh` is not in that list because it checks nothing. It builds the disk
images the other scripts read, and `run.sh` calls it.

Everything is built into `Tests/build`, which is removed and remade at the start of every
run. **The suite never writes into `Programs/`.** A binary sitting next to its source came
from `make`, not from here.

## The Five Kinds Of Check:

### 1. Recorded output

`Tests/run.sh` assembles each program named in `Tests/manifest`, runs it, and compares
everything it printed against a file in `Tests/expected`. 165 tests, of which 103 run, 35
only assemble, 16 are expected to fail to assemble, and 11 boot from ROM with no image
given at all.

This is the bulk of the suite and the shallowest part of it. It answers "did this change"
and nothing else. A recorded file is only as correct as the day somebody looked at it,
which is why blessing has a protocol.

The 16 `xfail` tests are worth calling out separately: each one is a piece of source the
assembler is supposed to refuse, and the recorded result is the refusal message. They catch
two things at once - an error that stops being detected, and an error message that changes
without anybody meaning it to.

### 2. A second implementation

`Tests/native.sh` and `Tests/agree.sh` are the two checks that do not compare the code
against a memory of itself.

`native.sh` assembles the same source with the host assembler and with `Asm.sbx` running on
the emulated machine, and compares the two binaries byte for byte. **The only honest test of
an assembler is the bytes it produces:** "it ran" and "the sizes look right" both pass for a
binary with a label one byte out, which is a program that jumps into the middle of an
instruction. It does a boot image, four loadable programs, CosmOS, and the assembler
itself - and then the CosmOS that CosmOS built, which is where it stops, because a second
generation that matches the first is a fixed point.

`agree.sh` builds the same directory tree twice, once with SplitDisk and once by driving
CosmOS, and compares the images byte for byte. Every field one implementation writes and
the other only reads is checked here and nowhere else: which entry a file lands in, which
block, what a directory's unused fields hold, the version in the superblock, the free
count. A disagreement in any of those is a disk one of them can read and the other cannot,
and the usual way that gets discovered is somebody's file coming back wrong months later.

Both scripts depend on both sides being driven in the same order, because both allocate
first fit and both take the first free entry. Given the same operations in the same
sequence they should reach the same bytes, and any difference is real rather than an
artefact of the script.

### 3. Named properties

`Tests/disk.sh` and `Tests/lint.sh` do not compare transcripts. Each check is a named
assertion that either holds or does not, and the name is the failure message.

`disk.sh` puts files of every awkward size onto an image and takes them off again -
nothing at all, less than a block, exactly a block, a part block, an exact multiple - and
then asks for the things the format says cannot happen and requires them to be refused
rather than half done. Roughly half of its checks are `refuses`, which is the shape
worth copying: **a tool that never says no is not finished.**

`lint.sh` builds a fixture in which every line trips exactly one rule, and checks which
warning came out at which line. It used to compare a total, and a total is a number that
stays right while the thing behind it goes wrong: a change that stopped one rule firing and
made another fire twice would have passed without a murmur. The fixture is written inside
the script rather than kept as a file because a reader wants the pattern and the
expectation side by side.

### 4. What a recorded file cannot see

`Tests/terminal.sh` runs the emulator under a pseudo-terminal, which is what makes its
questions askable at all. Two whole classes of behaviour are invisible to every other check
here:

**When something is printed.** Piped output is fully buffered and flushed when the process
ends, so a prompt shown before its answer is read and a prompt shown an hour late produce
byte-identical files. A prompt printed *after* the thing it was asking for is invisible to
the entire rest of the suite.

**What happens to the terminal.** Key mode only touches a terminal when there is one, so
with input from a file there is nothing to put into another state and nothing to put back.
A machine that leaves the terminal with echo off passes every other test in this
repository.

Both of those went wrong here, and both were found by a person whose terminal stopped
working rather than by anything in this suite. That is why the script exists. It checks
that a prompt arrives before input is read, that a keystroke arrives without Return, that
the terminal is handed back however the machine dies - SIGHUP, SIGINT, SIGQUIT, SIGABRT,
SIGSEGV, SIGTERM - and that suspending and resuming leave it as they found it.

It also asks the one question about cycles that nothing else can, since the count is
stripped from every recorded result: whether a program on a slow disk slept through the
wait or spun on it. Both print the same characters and take the same elapsed time. Only the
split between idle and bus cycles tells them apart.

### 5. The documents against the code

`Tests/docs.sh` settles, every time the suite runs, every claim in the manuals that can be
settled by looking at the source. Documentation goes stale quietly: an instruction added
without a table row, or a count in a heading that nobody updated, is wrong in a way nothing
notices until somebody trusts it.

It checks that every tracked file is plain ASCII; that every link in every document goes
somewhere, and carries `%20` rather than a raw space; that every instruction has a row and
every row is an instruction; that the counts in the group headings are right, and the count
of instructions that take a Data Pointer; that every device class has a row in the Devices
table; that the vector ranges the manuals quote are the ones the assembler uses; that the
loadable header table matches the offsets the assembler writes; that every console status
bit is described; that every service CosmOS implements has a row and every row is
implemented; that every application the CosmOS README describes exists; that the monitor's
instruction table is the assembler's, generated rather than typed; that every directive is
written down; that every routine the manuals promise is defined; that CosmOS still fits in
the half of the machine its memory map gives it; that the sizes quoted for applications are
their real sizes; and that the worked examples still assemble to the bytes printed beside
them.

It also checks this document: that every script the suite runs has a bullet here saying why
it exists, and that the number quoted is the number there are.

**A manual is a claim somebody may check by typing it.** Everything here is one that gets
checked automatically instead.

## Determinism:

A test that passes four times out of five is worse than no test, because the failure gets
attributed to the suite rather than to the code. Several rules exist only to keep results
identical from one run to the next, and each of them is a bug that already happened.

**The cycle count is stripped from every recorded result.** The last line of the emulator's
output has the number taken out before anything is compared, keeping only *whether* the
program stopped on its own or ran into its limit, which is behaviour. Two instructions
added to CosmOS used to move that number in six unrelated files at once, so a real
difference would have arrived in a crowd of meaningless ones and had to be picked out by
hand. Anything that genuinely wants to measure cycles should say so out loud in a test of
its own, the way `terminal.sh` does.

The substitution is deliberately **not anchored to the start of a line**. A program whose
last output has no newline on it leaves the cursor mid-line and the halt message is printed
there, so the count sat inside a line rather than at the head of one and survived.
`replCalculator` is the program that does that, and it was the only test to churn when the
machine started charging for memory accesses instead of counting instructions.

**Every program runs with `--fast`.** The emulated clock rate has no bearing on what a
program prints, and waiting out real time makes the suite slower for nothing.

**A program that never halts is bounded by `--cycles`, not by wall clock.** A cycle limit
produces the same output on a fast machine and a slow one. A time limit does not.

**Every run has a ten second timeout** on top of that, which is a failure rather than a
result. A test that hangs says so instead of hanging the suite.

**A scratch disk is removed before every run**, so nothing a test writes can be seen by the
next one. The emulator makes a blank image in its place.

**The build directory is removed and remade at the start of every run**, so a stale binary
cannot pass for a fresh one.

**The exit status is recorded with the output.** What a program exits with is part of what
it does. A program that faults is supposed to exit non-zero, and that should be as pinned
down as what it printed.

## The Manifest:

`Tests/manifest` is one test per line, fields separated by `|`. Blank lines and lines
starting with `#` are ignored, and the file is heavily commented by section.

```
name | source | mode | stdin | limit | disk
```

**name** is what the test is called and what `Tests/expected/<name>.out` is named after.

**source** is relative to `Programs/`. Everything assembles from there with `Libraries/`
and `CosmOS/Source/` on the include path. CosmOS is on the path because it owns the
filesystem library and the service names, which test programs outside it include.

**mode** is one of:

| Mode | What happens |
| --- | --- |
| `run` | Assemble, execute, compare all output against the recorded result |
| `assemble` | Assemble only, for library files with no entry point to run |
| `xfail` | Assembly is expected to fail, and the message is recorded |
| `rom` | As `run`, but no image is given, so the machine boots from its ROM |

`xfail` records a known breakage, so that fixing one is noticed and so that an accidental
new one is too. `rom` is what a machine with no debugger attached does: the emulator
shadows its built-in stage one into Program Memory and reads the disk for everything else.

**stdin** names a file in `Tests/input`, or `-` for none.

**limit** is a cycle count for programs that never halt on their own, or `-`.

**disk** names an image to attach, or `-` for a machine with no disk, which is most of
them. Four forms:

| Form | Meaning |
| --- | --- |
| `name.img` | Scratch. Removed before the run, so the emulator makes a blank one |
| `disks/name.img` | A fixture built by `makedisks.sh`, used as it stands |
| `...:ro` | Attached write protected, so a test can check the *device* bars writes |
| `...@N` | Given a latency of N cycles, so a test can check the filesystem waits |

The `@N` form deserves a note. Every other test runs with the disk's answer available
before the next instruction, which is the one condition under which failing to wait looks
exactly like working.

## Fixture Disks:

`Tests/makedisks.sh` builds 26 images with SplitDisk before anything runs, into
`Tests/build/disks`. **That is the point of them.** A SplitBit program reading one of these
is being checked against a filesystem written by different code from the same written
specification, rather than against itself.

They are shaped to be awkward on purpose. `sbfs.img` has two directory blocks and eight
files that fill the first one exactly, so everything after that lands in the second and the
walk from block to block gets exercised rather than assumed. It carries a file longer than
a block whose pattern repeats every twenty-six bytes, which makes a misplaced block obvious
to read; an empty file; and a name of exactly the maximum length. `deep.img` has eight
directory blocks. `chain.img` carries a different payload in each of its two boot slots, so
a test reading the right word is reading the slot that was chosen rather than whatever
happened to be left in Program Memory, and `chainAlt.img` is the same disk with the other
slot selected. `selfboot.img` has stage two in a boot slot and the system as an ordinary
file, which is the whole chain end to end, and `nosystem.img` has the second stage with
nothing for it to find, so that a loader which cannot start anything says so rather than
jumping somewhere.

`cfgbare.img`, `cfgbroken.img` and `cfgfallback.img` are the same disk three ways,
differing only in `/System/Boot/boot.cfg`. Nothing else about any of them changes, which is
what makes those tests of the **file** rather than of the machinery under it. That is the
shape to copy when a fixture is added: change one thing, and keep everything else
identical to a disk that already works.

The script makes its paths absolute before doing anything else, because it changes
directory partway down. When that was wrong the failure was not an error - it was a disk
quietly missing some of the files it was supposed to have, which is a much worse thing to
debug.

## Recording Expected Output:

```
make bless
./Tests/run.sh --bless
```

This overwrites every file in `Tests/expected` with the current output.

**Blessing unexamined output records a bug as correct, permanently and silently.** It is
the one operation in this repository that can make the suite confidently wrong, and it does
it without printing anything alarming. The protocol is:

1. Run the suite and read the diffs. `run.sh` prints the first twenty lines of each.
2. Account for every changed file. Not "that looks like the change I made" - know why each
   one moved, and know why the ones that did not move did not.
3. Bless.
4. Look at `git diff Tests/expected` before committing, which is the last chance to see a
   recorded result that should not have changed.

A single test can be re-run by name to see one diff on its own, which is usually faster
than reading all of them:

```
./Tests/run.sh cosmosCwd
```

## The Lint Baseline:

`Tests/lint-baseline.txt` records how many warnings of each rule SplitLint finds in each
file across the whole corpus - 34 file-and-rule pairs. It is checked on every run, and it
is checked **in both directions.**

A new warning appearing is a regression. A recorded warning *disappearing* is also
reported, and that is the half people do not expect: it means either that somebody fixed
something and did not record it, which is fine and takes one command, or that a rule
stopped firing, which is not fine at all and is otherwise completely silent.

```
./Tests/lint.sh --bless
```

records the corpus as it stands, for when warnings have been deliberately fixed or
deliberately accepted. Same shape as `run.sh --bless`, and it wants the same care.

A warning that is correct but deliberate should be suppressed in the source rather than
carried in the baseline, with a comment saying `splitlint: <reason>` or
`splitlint[rule]: <reason>`. The reason is required, so that a deliberate exception says
what makes it one. A marker that silences nothing is itself reported, as `dead-suppression`,
which stops the markers outliving the code they were about.

## The Sanitizer Run:

```
make sanitize
```

Rebuilds all four tools with `-fsanitize=address,undefined` and runs **the whole suite**
under them. It catches reads and writes off the end of an array, use after free, leaks, and
undefined arithmetic. It also fills fresh allocations with a junk pattern, which is what
turns a read of uninitialised memory from something that quietly works into something the
tests notice.

It runs everything because it used to not. It built all four tools sanitized and then ran
only `run.sh` and `terminal.sh`, so SplitDisk was compiled with the sanitizers and never
exercised, and `native.sh` - which drives the assembler and the emulator harder than
anything else here - was skipped entirely. Those are exactly where block arithmetic on disk
images and buffer indexing in two assemblers live. Adding the other scripts cost about six
seconds.

If the suite fails, the sanitizer binaries are deliberately left in place so the failing
case can be run again by hand. Plain `make` puts the normal ones back.

Worth running before a release, and after anything that touches memory handling.

## The Strict Build:

`make test` depends on `make strict`, which compiles every source file with
`-std=c11 -pedantic -Wall -Wextra -Werror` and throws away the object.

This exists because the README says "the sources are ISO C and build clean under
`-std=c11 -pedantic` with `-Wall -Wextra`", and that is a claim somebody may check by
typing it. It was false when the target was written: `realpath` went undeclared under a
feature test macro that did not reach far enough, which the ordinary `-Os` build never saw.

## Adding A Test:

**A program.** Put the source in `Programs/`, add a line to `Tests/manifest`, run
`./Tests/run.sh <name>` and read what it printed. If it is right, `make bless` - which
records every test, so make sure the rest of the suite is clean first. If the program needs
input, put a file in `Tests/input` and name it in the manifest. If it needs a
disk, name a scratch image; if it needs a disk with something already on it, add it to
`makedisks.sh` and name it as `disks/whatever.img`.

**A property of a host tool.** Add a `check` or a `refuses` line to `Tests/disk.sh`. The
name is the failure message, so write it as a statement of what should be true: "no
removing an occupied one", not "test 47".

**Something a recorded file cannot see.** `Tests/terminal.sh` is the place, and it is the
hardest script to add to because each check has to arrange a situation and then observe it
from outside. Small assembly programs are written inline in the script for this, and only
this - everything else assembles from the repository.

**A linter rule.** Add its line to the fixture in `Tests/lint.sh` next to the other lines
for its family, with the expected line number and message. Every line of the fixture exists
to trip exactly one rule; keep it that way, or a rule that stops working will be masked by
the one next to it.

**A claim in a manual.** If you write a number, a count, or a list into any of the four
documents, consider whether `Tests/docs.sh` can settle it from the source. Most of them
can. The ones already there are all claims that were true when written and quietly stopped
being.

## Checking A Check:

**A new check is not finished until it has been seen to fail.** The discipline is to break
the thing on purpose, run the check, watch it report, and put the thing back. This has
caught several checks in this repository that passed for the wrong reason and would have
passed for ever.

Two of them are worth remembering:

An agreement test passed because both of its cases were on one disk, and the first save ate
the file that was the second case's *source*. It compared two things that were both empty.
Splitting them onto separate disks made it fail, which is when it started being a test.

The lint fixture passed with a rule's message broken, because the total was still
twenty-three.

A check that has only ever been seen to pass is a check you are trusting on faith. That is
the same category of thing as an unread recorded result.

## What The Suite Does Not Catch:

Written down because a document that lists only strengths teaches the wrong lesson.

**Buffer overruns into an adjacent variable.** This has happened three times: the shell's
prompt into its command name table, the assembler's ceilings, and the line editor's `Entry`
buffer into `TextHead` and `ArenaFree`. **All three were found by a person using the
machine, and none of them by the suite.** They are hard for it structurally: the write
succeeds, nothing faults, and the damage shows up somewhere else entirely, at a time that
depends on the memory map. The sanitizers cannot see them either, because the overrun is in
emulated Data Memory, which is one legitimate host array from end to end. The only defences
are reading every `#Reserve` that something copies into, and bounding the loop that fills
it.

**Anything that needs somebody at the keyboard.** `terminal.sh` gets closer than a recorded
file can, but it drives a pseudo-terminal on a script. Nobody types slowly, changes their
mind, resizes the window, or holds a key down. The worst bugs of the last stretch - the
editor misreading a source file, the machine wedging afterwards, and both terminal failures
above - all came from somebody sitting down and using CosmOS for an hour.

**Performance.** Cycle counts are deliberately excluded from every recorded result, so
nothing here notices a program becoming four times slower. That is the correct trade for
now and it is a real gap.

**The emulator being wrong about the machine.** Every check here compares SplitBit against
SplitBit. `native.sh` and `agree.sh` are second opinions about the *assembler* and the
*filesystem*, and there is no second opinion about the CPU at all. Timing, bus behaviour,
what an undriven port reads as, and what happens when two things want memory at once are
all decided by one implementation and checked against nothing.

**Whether the design is any good.** The suite can say that CosmOS still fits in its half of
the machine. It cannot say that the memory map is the right one.