Make xfail compare the diagnostic, not just the failure
The Test Manual said an xfail test records the assembler's refusal message and so catches both an error that stops being detected and a message that changes without anybody meaning it to. It did not. run.sh checked only that the assembler exited non zero, printed the first line for a person to read, and compared nothing; --bless recorded nothing for these sixteen tests at all. So an xfail passed four different ways that look identical from outside: the intended error fired, an unrelated error fired, the message changed, or the assembler fell over on its way to the point. That is the documentation describing behaviour the code does not have, which is the exact failure Tests/docs.sh exists to prevent, in the manual that argues for knowing what your evidence is worth. The diagnostic is now stripped of colour, given the same [exit N] line every other recorded result carries, and compared through check() like anything else. Sixteen results recorded; every existing one is byte for byte unchanged. Verified the way the manual asks: one diagnostic was broken on purpose, its test failed with the changed line in the diff, and its neighbour passed. Found by ChatGPT reviewing the manual. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW
This commit is contained in:
co-authored by
Claude Opus 5
parent
999451e935
commit
8fbbeb6ec9
+28
-4
@@ -141,13 +141,27 @@ assemble() {
|
||||
# test programs outside it include.
|
||||
local name="$1" src="$2"
|
||||
local bin="$BUILD/$name.bin"
|
||||
if ( cd "$PROGRAMS" && "$ASSEMBLER" -I Libraries -I CosmOS/Source -o "$bin" "$src" ) >"$BUILD/.assemble.log" 2>&1; then
|
||||
( cd "$PROGRAMS" && "$ASSEMBLER" -I Libraries -I CosmOS/Source -o "$bin" "$src" ) \
|
||||
>"$BUILD/.assemble.log" 2>&1
|
||||
# WRITTEN TO A FILE RATHER THAN A VARIABLE, because this function is called inside a
|
||||
# command substitution below and that runs it in a subshell, where anything it set
|
||||
# would be thrown away. A file is the one channel that survives either call site.
|
||||
echo $? > "$BUILD/.assemble.status"
|
||||
if [ "$(cat "$BUILD/.assemble.status")" -eq 0 ]; then
|
||||
echo "$bin"
|
||||
return 0
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
# Takes the colour out of a diagnostic, in place. The assembler paints its errors whether
|
||||
# or not anything is a terminal, so a recorded refusal would otherwise carry escape
|
||||
# sequences - and a recorded result full of them is unreadable in a diff, which is the one
|
||||
# moment it has to be read.
|
||||
uncolour() {
|
||||
sed -i -E 's/\x1b\[[0-9;]*m//g' "$1"
|
||||
}
|
||||
|
||||
while IFS='|' read -r name src mode stdin limit disk; do
|
||||
name="$(trim "$name")"
|
||||
[ -z "$name" ] && continue
|
||||
@@ -163,10 +177,20 @@ while IFS='|' read -r name src mode stdin limit disk; do
|
||||
if assemble "$name" "$src" >/dev/null; then
|
||||
FAIL=$((FAIL + 1)); FAILED_NAMES+=("$name")
|
||||
report "FAIL" "$name" "expected assembly to fail, but it succeeded"
|
||||
else
|
||||
PASS=$((PASS + 1))
|
||||
report "ok" "$name" "fails as recorded: $(head -1 "$BUILD/.assemble.log" | tr -d '\033' | sed 's/\[[0-9;]*m//g')"
|
||||
continue
|
||||
fi
|
||||
# WHAT IT SAID, not merely that it said something. Checking only that the assembler
|
||||
# exited non zero passed for the intended error, for an unrelated error, for a
|
||||
# changed message, and for the assembler falling over on its way to the point - all
|
||||
# four look identical from outside. So the diagnostic is recorded and compared like
|
||||
# any other output, which is what the Test Manual has always claimed happened here.
|
||||
OUT="$BUILD/.out"
|
||||
cp "$BUILD/.assemble.log" "$OUT"
|
||||
uncolour "$OUT"
|
||||
# The same last line every other recorded result carries, and for the same reason:
|
||||
# which non zero status a refusal exits with is part of what it does.
|
||||
printf '[exit %d]\n' "$(cat "$BUILD/.assemble.status")" >> "$OUT"
|
||||
check "$name" "$OUT"
|
||||
continue
|
||||
fi
|
||||
|
||||
|
||||
Reference in New Issue
Block a user