jake/SplitBit-Emulator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Write the Test Manual, and make the suite check it

Browse Source 
The test system had grown to seven scripts making five genuinely different
kinds of claim, and nothing said which was which. A recorded transcript and a
byte-for-byte comparison against a second implementation both print [ok  ] and
are worth wildly different amounts, so the fourth manual exists to say so: what
each script can and cannot answer, why every determinism rule is there, how to
add a test, and - the part written nowhere else - where the suite is blind.

That last section is the reason for the document. Three buffer overruns into
adjacent variables were all found by a person using the machine and none by the
suite, the sanitizers cannot see them because emulated Data Memory is one
legitimate host array, and there is no second opinion about the CPU at all. A
document listing only strengths teaches the wrong lesson.

The bullets describing each script move out of the README, so docs.sh now reads
the manual for them, and five more numbers in it are settled from the source
rather than trusted: the shape of the manifest, the xfail count, how many
fixture disks makedisks.sh builds, how large the lint baseline is, and the tool
count in either document. Each of the new checks was broken on purpose and
watched to report before being kept, which is the discipline the manual itself
argues for.

Also drops the stale "70 instructions" from instructiontable.py's docstring.
There are 72, and a number that carries no meaning is better removed than
corrected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW
This commit is contained in:
Anachronaut
2026-08-27 23:52:12 -04:00
co-authored by Claude Opus 5
parent 79727044b7
commit d6bc416698
4 changed files with 572 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Tests/docs.sh
+82 -20
View File
@@ -581,42 +581,104 @@ else:
" 0x%s - the buffers are on top of the variables"
% (ends - 1, first.group(1).upper()))
# ---- Every test script the suite runs has a bullet saying why it exists ----
# ---- The Test Manual against the suite it describes ----
#
# Two claims in the README went stale at once and neither was noticed: it said FIVE more
# scripts run alongside run.sh when there were six, and "all three tools" when there were
# four. Both are the kind of number that is written once, is true for months, and is then
# quietly wrong - which is the entire subject of this file.
# The suite documents itself, and a document about the suite goes stale the same way every
# other one does. Two claims in the README went stale at once before this check existed and
# neither was noticed: it said FIVE more scripts run alongside run.sh when there were six,
# and "all three tools" when there were four. Both are the kind of number that is written
# once, is true for months, and is then quietly wrong - which is the entire subject of this
# file.
#
# run.sh is the driver rather than one of the others, and makedisks.sh makes the images
# rather than checking anything, so neither is counted.
# The repository README rather than CosmOS's, which is what `readme` above holds.
# The bullets now live in the Test Manual rather than the README, so that is what is read.
# makedisks.sh is not counted, because it builds the images rather than checking anything;
# run.sh is counted, because the manual describes it alongside the rest.
rootReadme = open("README.md").read()
manual = open("SplitBit Test Manual.md").read()
# THE MANUAL IS WRAPPED, so a number and the noun it counts are regularly on two different
# lines. Every pattern below runs against a copy with its whitespace flattened.
flat = re.sub(r"\s+", " ", manual)
scripts = sorted(os.path.basename(p) for p in glob.glob("Tests/*.sh")
if os.path.basename(p) not in ("run.sh", "makedisks.sh"))
if os.path.basename(p) != "makedisks.sh")
words = {"three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8, "nine": 9}
said = re.search(r"([A-Za-z]+) more scripts run alongside it", rootReadme)
said = re.search(r"It is ([a-z]+) scripts making", flat)
if not said:
problems.append("the README no longer says how many scripts run alongside run.sh")
elif words.get(said.group(1).lower()) != len(scripts):
problems.append("the README says %s scripts run alongside run.sh, and there are %d: %s"
problems.append("the Test Manual no longer says how many scripts the suite is")
elif words.get(said.group(1)) != len(scripts):
problems.append("the Test Manual says the suite is %s scripts, and there are %d: %s"
% (said.group(1), len(scripts), ", ".join(scripts)))
for name in scripts:
if ("`Tests/%s`" % name) not in rootReadme:
problems.append("Tests/%s runs in the suite and the README does not say what it is"
" for" % name)
if ("`Tests/%s`" % name) not in manual:
problems.append("Tests/%s runs in the suite and the Test Manual does not say what"
" it is for" % name)
# ---- The shape of the manifest, which the manual states outright ----
#
# Five numbers in one sentence, all of them countable from the file they describe. This is
# the most quotable thing in the manual and the least likely to be recounted by hand.
modes = {}
total = 0
for line in open("Tests/manifest"):
line = line.strip()
if not line or line.startswith("#"):
continue
total += 1
fields = line.split("|")
if len(fields) > 2:
modes[fields[2].strip()] = modes.get(fields[2].strip(), 0) + 1
said = re.search(r"(\d+) tests, of which (\d+) run, (\d+) only assemble,"
r" (\d+) are expected to fail to assemble, and (\d+) boot from ROM", flat)
if not said:
problems.append("the Test Manual no longer states the shape of the manifest")
else:
for index, (what, count) in enumerate((("tests", total),
("run tests", modes.get("run", 0)),
("assemble-only tests", modes.get("assemble", 0)),
("xfail tests", modes.get("xfail", 0)),
("rom tests", modes.get("rom", 0)))):
if int(said.group(index + 1)) != count:
problems.append("the Test Manual says there are %s %s, and there are %d"
% (said.group(index + 1), what, count))
said = re.search(r"The (\d+) `xfail` tests", flat)
if said and int(said.group(1)) != modes.get("xfail", 0):
problems.append("the Test Manual says %s xfail tests in one place and %d in another"
% (said.group(1), modes.get("xfail", 0)))
# ---- And the fixtures and the baseline ----
disks = len(re.findall(r'format "\$DISKS/', open("Tests/makedisks.sh").read()))
said = re.search(r"builds (\d+) images with SplitDisk", flat)
if not said:
problems.append("the Test Manual no longer says how many fixture disks are built")
elif int(said.group(1)) != disks:
problems.append("the Test Manual says %s fixture disks are built, and makedisks.sh"
" builds %d" % (said.group(1), disks))
pairs = sum(1 for line in open("Tests/lint-baseline.txt") if line.strip())
said = re.search(r"(\d+) file-and-rule pairs", flat)
if not said:
problems.append("the Test Manual no longer says how large the lint baseline is")
elif int(said.group(1)) != pairs:
problems.append("the Test Manual says the lint baseline holds %s file and rule pairs,"
" and it holds %d" % (said.group(1), pairs))
# ---- And the tool count is the number of things the makefile builds ----
#
# Claimed in both documents, so both are read.
makefile = open("makefile").read()
built = re.search(r"^all:(.*)$", makefile, re.M)
if not built:
problems.append("the makefile no longer has an all target this can count")
else:
tools = len(built.group(1).split())
for said in re.findall(r"(?:build|rebuild) (?:all )?(?:the )?([a-z]+) tools", rootReadme):
if words.get(said) != tools:
problems.append("the README says the %s tools and the makefile builds %d"
% (said, tools))
for where, text in (("README", rootReadme), ("Test Manual", manual)):
for said in re.findall(r"(?:build|rebuild)s? (?:all )?(?:the )?([a-z]+) tools",
text, re.I):
if words.get(said.lower()) != tools:
problems.append("the %s says the %s tools and the makefile builds %d"
% (where, said, tools))
# ---- The sizes the CosmOS README quotes for its own programs ----
#