SplitBit-Emulator/SplitBit Test Manual.md

The SplitBit Test Manual

This describes the test suite: what it is for, what each part of it can and cannot answer, how to add to it, and where it is blind. It is the fourth of the manuals, and the only one about the repository rather than about the machine.

The other three describe things a program can rely on. This one describes the reason to believe them.

What The Suite Claims:

The suite is not one thing. It is ten scripts making five different kinds of claim, and knowing which claim you are relying on is the whole point of this document. A recorded transcript and a byte-for-byte comparison against a second implementation both print [ok ], and they are worth wildly different amounts.

The strongest thing the suite says is this: on the source it is given, the assembler that runs on SplitBit and the one that runs on the host produce identical bytes; and on the operations it performs, the filesystem written by the tool and the filesystem written by the machine are identical disks. Those are two programs written from one specification, sharing no code, checking each other.

Note the qualifiers, because they are the whole difference between a strong claim and an untrue one. This is evidence about a corpus and a sequence of operations, not a proof about every program that could be assembled or every disk that could be built. No suite says more than that. But nothing else here is even that strong.

The weakest thing it says is that a program prints what it printed last time. That is worth having and it is worth having a lot of, but it only ever catches change. It cannot catch a thing that was wrong the first time it was recorded.

Everything in between is somewhere on that line.

Running It:

make test

Builds the four tools - and Voyager, where Raylib is installed - checks they compile under strict ISO C, and runs the scripts in order. Takes a few seconds. Everything must pass; there are no expected failures at the level of the suite, only tests that record an expected failure of the assembler.

make sanitize

The same suite with the four tools rebuilt under AddressSanitizer and UndefinedBehaviorSanitizer. See The Sanitizer Run.

Individual scripts can be run on their own, from anywhere:

./Tests/run.sh                  Every program in the manifest.
./Tests/run.sh hello waitTest   Only the named ones.
./Tests/run.sh --bless          Record current output as expected. See below.
./Tests/voyager.sh              The same manifest, through the other front end.
./Tests/disk.sh                 The disk tool against the format.
./Tests/cycles.sh               What the memory controller charges.
./Tests/video.sh                What the video device draws.
./Tests/terminal.sh             The things a recorded file cannot see.
./Tests/native.sh               The two assemblers against each other.
./Tests/agree.sh                The two filesystems against each other.
./Tests/lint.sh                 SplitLint against its fixture and the corpus baseline.
./Tests/lint.sh --bless         Record the corpus baseline. See below.
./Tests/docs.sh                 The manuals against the code.

Tests/makedisks.sh is not in that list because it checks nothing. It builds the disk images the other scripts read, and run.sh calls it.

Everything is built into Tests/build, which is removed and remade at the start of every run. The suite never writes into Programs/. A binary sitting next to its source came from make, not from here.

The Five Kinds Of Check:

1. Recorded output

Tests/run.sh assembles each program named in Tests/manifest, runs it, and compares everything it printed against a file in Tests/expected. 169 tests, of which 107 run, 35 only assemble, 16 are expected to fail to assemble, and 11 boot from ROM with no image given at all.

This is the bulk of the suite and the shallowest part of it. It answers "did this change" and nothing else. A recorded file is only as correct as the day somebody looked at it, which is why blessing has a protocol.

The 16 xfail tests are worth calling out separately: each one is a piece of source the assembler is supposed to refuse, and the recorded result is the refusal message. They catch two things at once - an error that stops being detected, and an error message that changes without anybody meaning it to.

2. A second implementation

Tests/native.sh, Tests/agree.sh and Tests/voyager.sh are the checks that do not compare the code against a memory of itself.

native.sh assembles the same source with the host assembler and with Asm.sbx running on the emulated machine, and compares the two binaries byte for byte. The only honest test of an assembler is the bytes it produces: "it ran" and "the sizes look right" both pass for a binary with a label one byte out, which is a program that jumps into the middle of an instruction. It does a boot image, four loadable programs, CosmOS, and the assembler itself - and then the CosmOS that CosmOS built, which is where it stops, because a second generation that matches the first is a fixed point.

agree.sh builds the same directory tree twice, once with SplitDisk and once by driving CosmOS, and compares the images byte for byte. Every field one implementation writes and the other only reads is checked here and nowhere else: which entry a file lands in, which block, what a directory's unused fields hold, the version in the superblock, the free count. A disagreement in any of those is a disk one of them can read and the other cannot, and the usual way that gets discovered is somebody's file coming back wrong months later.

native.sh and agree.sh both depend on the two sides being driven in the same order, because both allocate first fit and both take the first free entry. Given the same operations in the same sequence they should reach the same bytes, and any difference is real rather than an artefact of the script.

Tests/voyager.sh is the cheapest of the three, because it reuses the runner rather than repeating it. SplitBit and Voyager share every line of the machine and differ only in what they present - a terminal, or a window and a speaker - and the way to keep that true is to make the claim testable. So it runs the whole manifest through Voyager with --headless, held to the same recorded results SplitBit is held to. The claim is not that the two look alike: it is that one satisfies every recording the other does, byte for byte, exit status included.

Tests/run.sh takes the machine to run from SPLITBIT_EMULATOR, which is what makes that possible without a second copy of the runner. A copy would drift, and the first thing to go would be whichever awkward case got added to only one of them.

Voyager not being built is not a failure. It needs Raylib and nothing else here does, which is the whole point of there being two binaries; a suite that failed on a machine with no graphics library would be enforcing exactly the dependency the split exists to avoid. The script says it was skipped, and passes.

3. Named properties

Tests/disk.sh and Tests/lint.sh do not compare transcripts. Each check is a named assertion that either holds or does not, and the name is the failure message.

disk.sh puts files of every awkward size onto an image and takes them off again - nothing at all, less than a block, exactly a block, a part block, an exact multiple - and then asks for the things the format says cannot happen and requires them to be refused rather than half done. Roughly half of its checks are refuses, which is the shape worth copying: a tool that never says no is not finished.

Tests/video.sh belongs here too, and exists for the same reason as the two above: the suite has no display, and a screen nothing can look at is a screen nothing checks. The device renders into a buffer that is a pure function of video memory, and the machine can be asked to save it with --screen, so every check runs a program, saves the picture and reads pixels back out of it. No window, no display server, and the same answer every time.

It checks named behaviours rather than a recorded image, which for a screen matters more than usual. A recorded image would say "something changed" and leave which of the palette, the tile, the attribute, the map or the scroll register broke to be found by hand. Instead each check is one claim: that a tile lands where it is put and stops at the cell edge, that the palette is what colours it, that the attribute nibble adds sixteen, that scrolling moves which row is on top, that the map wraps, and that an impossible mode is refused without stopping the machine.

Half of it is about the console rather than the device. Those programs ask the video device for nothing at all: they write bytes to port 0x00, the way every SplitBit program always has, and the picture is what is checked. A character lands at the cursor and the cursor moves along, a newline starts the next row, backspace rubs out, the line wraps at the last column, and the screen scrolls by moving the origin rather than by moving 1,920 bytes - which is checked by reading the register back, because a console that blitted rows instead would look identical on the screen and cost twelve percent of a frame a line.

Breaking the additive nibble fails exactly one check. Breaking the scroll origin fails exactly two. Moving every cell one pixel sideways fails the four about placement. That is what a screen test is supposed to do.

lint.sh builds a fixture in which every line trips exactly one rule, and checks which warning came out at which line. It used to compare a total, and a total is a number that stays right while the thing behind it goes wrong: a change that stopped one rule firing and made another fire twice would have passed without a murmur. The fixture is written inside the script rather than kept as a file because a reader wants the pattern and the expectation side by side.

4. What a recorded file cannot see

Tests/terminal.sh runs the emulator under a pseudo-terminal, which is what makes its questions askable at all. Two whole classes of behaviour are invisible to every other check here:

When something is printed. Piped output is fully buffered and flushed when the process ends, so a prompt shown before its answer is read and a prompt shown an hour late produce byte-identical files. A prompt printed after the thing it was asking for is invisible to the entire rest of the suite.

What happens to the terminal. Key mode only touches a terminal when there is one, so with input from a file there is nothing to put into another state and nothing to put back. A machine that leaves the terminal with echo off passes every other test in this repository.

Both of those went wrong here, and both were found by a person whose terminal stopped working rather than by anything in this suite. That is why the script exists. It checks that a prompt arrives before input is read, that a keystroke arrives without Return, that the terminal is handed back however the machine dies - SIGHUP, SIGINT, SIGQUIT, SIGABRT, SIGSEGV, SIGTERM - and that suspending and resuming leave it as they found it.

It also asks the one question about waiting that nothing else can, since the count is stripped from every recorded result: whether a program on a slow disk slept through the wait or spun on it. Both print the same characters and take the same elapsed time. Only the split between idle and bus cycles tells them apart.

Tests/cycles.sh is the other half of the same bargain, and exists because the determinism rules below throw the cycle count away. It measures what the memory controller charges for moving memory - which is real time out of a program's budget, and is invisible everywhere else in this suite.

It pins the rate rather than a total. Each case runs twice, from programs whose instructions are identical except for the byte written to the Command port: once asking for the transfer, and once asking for GuardOff, which lowers a fence that was never raised and costs nothing beyond the port write. The difference between the two runs is the transfer and nothing else - no instruction count, no setup, no startup - so the check survives every change to the machine that is not a change to what a transfer costs.

5. The documents against the code

Tests/docs.sh settles, every time the suite runs, every claim in the manuals that can be settled by looking at the source. Documentation goes stale quietly: an instruction added without a table row, or a count in a heading that nobody updated, is wrong in a way nothing notices until somebody trusts it.

It checks that every tracked file is plain ASCII; that every link in every document goes somewhere, and carries %20 rather than a raw space; that every instruction has a row and every row is an instruction; that the counts in the group headings are right, and the count of instructions that take a Data Pointer; that every device class has a row in the Devices table; that the vector ranges the manuals quote are the ones the assembler uses; that the loadable header table matches the offsets the assembler writes; that every console status bit is described; that every service CosmOS implements has a row and every row is implemented; that every application the CosmOS README describes exists; that the monitor's instruction table is the assembler's, generated rather than typed; that every directive is written down; that every routine the manuals promise is defined; that CosmOS still fits in the half of the machine its memory map gives it; that the sizes quoted for applications are their real sizes; and that the worked examples still assemble to the bytes printed beside them.

It also checks this document: that every script the suite runs has a bullet here saying why it exists, and that the number quoted is the number there are.

A manual is a claim somebody may check by typing it. Everything here is one that gets checked automatically instead.

Determinism:

A test that passes four times out of five is worse than no test, because the failure gets attributed to the suite rather than to the code. Several rules exist only to keep results identical from one run to the next, and each of them is a bug that already happened.

The cycle count is stripped from every recorded result. The last line of the emulator's output has the number taken out before anything is compared, keeping only whether the program stopped on its own or ran into its limit, which is behaviour. Two instructions added to CosmOS used to move that number in six unrelated files at once, so a real difference would have arrived in a crowd of meaningless ones and had to be picked out by hand. Anything that genuinely wants to measure cycles should say so out loud in a test of its own, the way terminal.sh does.

The substitution is deliberately not anchored to the start of a line. A program whose last output has no newline on it leaves the cursor mid-line and the halt message is printed there, so the count sat inside a line rather than at the head of one and survived. replCalculator is the program that does that, and it was the only test to churn when the machine started charging for memory accesses instead of counting instructions.

Every program runs with --fast. The emulated clock rate has no bearing on what a program prints, and waiting out real time makes the suite slower for nothing.

A program that never halts is bounded by --cycles, not by wall clock. A cycle limit produces the same output on a fast machine and a slow one. A time limit does not.

Every run has a ten second timeout on top of that, which is a failure rather than a result. A test that hangs says so instead of hanging the suite.

A scratch disk is removed before every run, so nothing a test writes can be seen by the next one. The emulator makes a blank image in its place.

The build directory is removed and remade at the start of every run, so a stale binary cannot pass for a fresh one.

The exit status is recorded with the output. What a program exits with is part of what it does. A program that faults is supposed to exit non-zero, and that should be as pinned down as what it printed.

The Manifest:

Tests/manifest is one test per line, fields separated by |. Blank lines and lines starting with # are ignored, and the file is heavily commented by section.

name | source | mode | stdin | limit | disk

name is what the test is called and what Tests/expected/<name>.out is named after.

source is relative to Programs/. Everything assembles from there with Libraries/ and CosmOS/Source/ on the include path. CosmOS is on the path because it owns the filesystem library and the service names, which test programs outside it include.

mode is one of:

Mode	What happens
run	Assemble, execute, compare all output against the recorded result
assemble	Assemble only, for library files with no entry point to run
xfail	Assembly is expected to fail, and the message is recorded
rom	As run, but no image is given, so the machine boots from its ROM

xfail records a known breakage, so that fixing one is noticed and so that an accidental new one is too. rom is what a machine with no debugger attached does: the emulator shadows its built-in stage one into Program Memory and reads the disk for everything else.

stdin names a file in Tests/input, or - for none.

limit is a cycle count for programs that never halt on their own, or -.

disk names an image to attach, or - for a machine with no disk, which is most of them. Four forms:

Form	Meaning
name.img	Scratch. Removed before the run, so the emulator makes a blank one
disks/name.img	A fixture built by makedisks.sh, used as it stands
...:ro	Attached write protected, so a test can check the device bars writes
...@N	Given a latency of N cycles, so a test can check the filesystem waits

The @N form deserves a note. Every other test runs with the disk's answer available before the next instruction, which is the one condition under which failing to wait looks exactly like working.

keys names a file in Tests/input to be fed to the console as a keyboard rather than as standard input, and the difference between those is the whole reason the field exists.

Standard input reaches a console that believes a terminal is doing the line editing, which is true when there is one: the terminal collects a line, rubs out a backspace, and hands over the finished thing at Return. Behind a window there is no terminal, so the console does that itself, and that is real logic which nothing could reach. It broke twice in two days and a person typing found it both times - once as keys that never arrived, once as a corrected line that reached the shell with the backspaces still in it, looking perfectly right on screen and matching no command at all.

A keyboard file installs the same hook a window does, so the same path runs. It does not test the window: Voyager's own key queue is still out of reach, and so is anything about presenting frames. It tests the console, which is where the logic is.

Fixture Disks:

Tests/makedisks.sh builds 26 images with SplitDisk before anything runs, into Tests/build/disks. That is the point of them. A SplitBit program reading one of these is being checked against a filesystem written by different code from the same written specification, rather than against itself.

They are shaped to be awkward on purpose. sbfs.img has two directory blocks and eight files that fill the first one exactly, so everything after that lands in the second and the walk from block to block gets exercised rather than assumed. It carries a file longer than a block whose pattern repeats every twenty-six bytes, which makes a misplaced block obvious to read; an empty file; and a name of exactly the maximum length. deep.img has eight directory blocks. chain.img carries a different payload in each of its two boot slots, so a test reading the right word is reading the slot that was chosen rather than whatever happened to be left in Program Memory, and chainAlt.img is the same disk with the other slot selected. selfboot.img has stage two in a boot slot and the system as an ordinary file, which is the whole chain end to end, and nosystem.img has the second stage with nothing for it to find, so that a loader which cannot start anything says so rather than jumping somewhere.

cfgbare.img, cfgbroken.img and cfgfallback.img are the same disk three ways, differing only in /System/Boot/boot.cfg. Nothing else about any of them changes, which is what makes those tests of the file rather than of the machinery under it. That is the shape to copy when a fixture is added: change one thing, and keep everything else identical to a disk that already works.

The script makes its paths absolute before doing anything else, because it changes directory partway down. When that was wrong the failure was not an error - it was a disk quietly missing some of the files it was supposed to have, which is a much worse thing to debug.

Recording Expected Output:

make bless
./Tests/run.sh --bless

This overwrites every file in Tests/expected with the current output.

Blessing unexamined output records a bug as correct, permanently and silently. It is the one operation in this repository that can make the suite confidently wrong, and it does it without printing anything alarming. The protocol is:

1. Run the suite and read the diffs. run.sh prints the first twenty lines of each.
2. Account for every changed file. Not "that looks like the change I made" - know why each one moved, and know why the ones that did not move did not.
3. Bless.
4. Look at git diff Tests/expected before committing, which is the last chance to see a recorded result that should not have changed.

A single test can be re-run by name to see one diff on its own, which is usually faster than reading all of them:

./Tests/run.sh cosmosCwd

The Lint Baseline:

Tests/lint-baseline.txt records how many warnings of each rule SplitLint finds in each file across the whole corpus - 34 file-and-rule pairs. It is checked on every run, and it is checked in both directions.

A new warning appearing is a regression. A recorded warning disappearing is also reported, and that is the half people do not expect: it means either that somebody fixed something and did not record it, which is fine and takes one command, or that a rule stopped firing, which is not fine at all and is otherwise completely silent.

./Tests/lint.sh --bless

records the corpus as it stands, for when warnings have been deliberately fixed or deliberately accepted. Same shape as run.sh --bless, and it wants the same care.

A warning that is correct but deliberate should be suppressed in the source rather than carried in the baseline, with a comment saying splitlint: <reason> or splitlint[rule]: <reason>. The reason is required, so that a deliberate exception says what makes it one. A marker that silences nothing is itself reported, as dead-suppression, which stops the markers outliving the code they were about.

The Sanitizer Run:

make sanitize

Rebuilds all four tools with -fsanitize=address,undefined and runs the whole suite under them. What it reliably catches is invalid access: reads and writes off the end of an array, use after free, leaks, and arithmetic the standard does not define.

AddressSanitizer also fills fresh heap allocations with a junk pattern, and it is worth knowing why that buys almost nothing here. It is a default of the toolchain rather than anything this build configures, so it is not something to rely on; and there are six heap allocations in the whole repository, all of them in the assembler, the largest a deliberate calloc. The machine's Program and Data memories are static arrays, which the sanitizers neither fill nor bound-check - which is the same fact, seen from a different side, as the overrun blind spot below.

It runs everything because it used to not. It built all four tools sanitized and then ran only run.sh and terminal.sh, so SplitDisk was compiled with the sanitizers and never exercised, and native.sh - which drives the assembler and the emulator harder than anything else here - was skipped entirely. Those are exactly where block arithmetic on disk images and buffer indexing in two assemblers live. Adding the other scripts cost about six seconds.

If the suite fails, the sanitizer binaries are deliberately left in place so the failing case can be run again by hand. Plain make puts the normal ones back.

Worth running before a release, and after anything that touches memory handling.

The Strict Build:

make test depends on make strict, which compiles every source file with -std=c11 -pedantic -Wall -Wextra -Werror and -D_XOPEN_SOURCE=700, then throws away the object.

That last flag is part of the check rather than a hole in it, and the distinction matters: this is strict C11 with the POSIX interfaces the code actually uses explicitly selected, not freestanding ISO C. The sources call realpath, strdup, dirname and getopt, and asking for POSIX.1-2008 by name is what makes a strict C11 build declare them rather than guess.

This exists because the README says "the sources are ISO C and build clean under -std=c11 -pedantic with -Wall -Wextra", and that is a claim somebody may check by typing it. It was false when the target was written: realpath went undeclared under a feature test macro that did not reach far enough, which the ordinary -Os build never saw.

Adding A Test:

A program. Put the source in Programs/, add a line to Tests/manifest, run ./Tests/run.sh <name> and read what it printed. If it is right, make bless - which records every test, so make sure the rest of the suite is clean first. If the program needs input, put a file in Tests/input and name it in the manifest. If it needs a disk, name a scratch image; if it needs a disk with something already on it, add it to makedisks.sh and name it as disks/whatever.img.

A property of a host tool. Add a check or a refuses line to Tests/disk.sh. The name is the failure message, so write it as a statement of what should be true: "no removing an occupied one", not "test 47".

Something a recorded file cannot see. Tests/terminal.sh is the place, and it is the hardest script to add to because each check has to arrange a situation and then observe it from outside. Small assembly programs are written inline in the script for this, and only this - everything else assembles from the repository.

A linter rule. Add its line to the fixture in Tests/lint.sh next to the other lines for its family, with the expected line number and message. Every line of the fixture exists to trip exactly one rule; keep it that way, or a rule that stops working will be masked by the one next to it.

A claim in a manual. If you write a number, a count, or a list into any of the four documents, consider whether Tests/docs.sh can settle it from the source. Most of them can. The ones already there are all claims that were true when written and quietly stopped being.

Checking A Check:

A new check is not finished until it has been seen to fail. The discipline is to break the thing on purpose, run the check, watch it report, and put the thing back. This has caught several checks in this repository that passed for the wrong reason and would have passed for ever.

Two of them are worth remembering:

An agreement test passed because both of its cases were on one disk, and the first save ate the file that was the second case's source. It compared two things that were both empty. Splitting them onto separate disks made it fail, which is when it started being a test.

The lint fixture passed with a rule's message broken, because the total was still twenty-three.

A check that has only ever been seen to pass is a check you are trusting on faith. That is the same category of thing as an unread recorded result.

What The Suite Does Not Catch:

Written down because a document that lists only strengths teaches the wrong lesson.

Buffer overruns into an adjacent variable. This has happened three times: the shell's prompt into its command name table, the assembler's ceilings, and the line editor's Entry buffer into TextHead and ArenaFree. All three were found by a person using the machine, and none of them by the suite. They are hard for it structurally: the write succeeds, nothing faults, and the damage shows up somewhere else entirely, at a time that depends on the memory map. The sanitizers cannot see them either, because the overrun is in emulated Data Memory, which is one legitimate host array from end to end. The only defences are reading every #Reserve that something copies into, and bounding the loop that fills it.

Anything that needs somebody at the keyboard. terminal.sh gets closer than a recorded file can, but it drives a pseudo-terminal on a script. Nobody types slowly, changes their mind, resizes the window, or holds a key down. The worst bugs of the last stretch - the editor misreading a source file, the machine wedging afterwards, and both terminal failures above - all came from somebody sitting down and using CosmOS for an hour.

Performance. Cycle counts are deliberately excluded from every recorded result, so nothing here notices a program becoming four times slower. That is the correct trade for now and it is a real gap.

The emulator being wrong about the machine. Every check here compares SplitBit against SplitBit. native.sh and agree.sh are second opinions about the assembler and the filesystem, and there is no second opinion about the CPU at all. Timing, bus behaviour, what an undriven port reads as, and what happens when two things want memory at once are all decided by one implementation and checked against nothing.

Whether the design is any good. The suite can say that CosmOS still fits in its half of the machine. It cannot say that the memory map is the right one.