Make xfail compare the diagnostic, not just the failure

The Test Manual said an xfail test records the assembler's refusal message and so
catches both an error that stops being detected and a message that changes without
anybody meaning it to. It did not. run.sh checked only that the assembler exited
non zero, printed the first line for a person to read, and compared nothing; --bless
recorded nothing for these sixteen tests at all.

So an xfail passed four different ways that look identical from outside: the intended
error fired, an unrelated error fired, the message changed, or the assembler fell over
on its way to the point. That is the documentation describing behaviour the code does
not have, which is the exact failure Tests/docs.sh exists to prevent, in the manual
that argues for knowing what your evidence is worth.

The diagnostic is now stripped of colour, given the same [exit N] line every other
recorded result carries, and compared through check() like anything else. Sixteen
results recorded; every existing one is byte for byte unchanged. Verified the way the
manual asks: one diagnostic was broken on purpose, its test failed with the changed
line in the diff, and its neighbour passed.

Found by ChatGPT reviewing the manual.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW
This commit is contained in:
Anachronaut
2026-08-28 18:07:35 -04:00
co-authored by Claude Opus 5
parent 999451e935
commit 8fbbeb6ec9
18 changed files with 95 additions and 6 deletions
+4 -2
View File
@@ -12,8 +12,10 @@
# Modes:
# run Assemble, execute, compare all output against Tests/expected/<name>.out
# assemble Assemble only. For library files that have no entry point to run
# xfail Assembly is expected to fail. Records a known breakage so that
# fixing one is noticed, and so an accidental new one is too
# xfail Assembly is expected to fail, and the assembler's diagnostic is recorded
# and compared like any other output. Catches a known breakage that stops
# being detected, an accidental new one, and a message that changes
# without anybody meaning it to
#
# limit is a cycle count, for programs that never halt on their own. It is passed
# to the emulator as --cycles, which bounds the run by cycles rather than by wall