From 2e9cb9af7e2bff55dc4e2f43fb725294e501f8ca Mon Sep 17 00:00:00 2001 From: Anachronaut Date: Sun, 6 Sep 2026 21:33:15 -0400 Subject: [PATCH] Comments are thrown away by the reader, not read one byte at a time A comment used to walk through tokGet and srcNext and be classified, character by character, on the way to being discarded. Comments are most of what this assembler reads: 87 per cent of what Say.asm pulls in once services.asm is counted, 59 per cent of CosmOS. The house style here is dense commentary, so the assembler is penalised more than most by its own sources. The saving is not the classifying, it is the BOOKKEEPING. srcNext loads and stores the walking pointer through memory for every character and asks numCompare whether the buffer is used up. srcSkipComment keeps the pointer in a data pointer for a whole run and the newline in B, so a comment byte costs a load, a compare and two steps. A run is capped at 255 so one byte can count it, which is the only reason it loops. Measured, each version with its own rebuilt images: without with hello 791,957 586,184 Say 9,924,401 4,506,702 2.20x Files 13,197,710 7,188,351 1.84x Keys 23,091,447 15,069,880 1.53x cosmos 886,498,996 789,982,899 1.12x Which tracks the comment ratios: Say gains most and cosmos least, in proportion to how much of each is prose. ---- And two mistakes worth keeping ---- The scratch went among the READER'S STATE, which is a block copied whole by a count written down somewhere else - so every saved file lost the last four bytes of itself and an include came back with its pointer wrong. The comment above that block says not to do this, in capitals. That is twice this week: scriptCopyState had the same shape this morning. And a file that ends inside a comment has to put back the file that included it, exactly as srcAtEnd does for a character. NOTHING IN THIS REPOSITORY ENDS THAT WAY - every source here ends on a line of code with a newline after it - so break.sh could not catch an error in that path because nothing reached it. tail.asm is generated with no newline on its last line for that reason, and usestail.asm names a label after the include, which is what goes missing when the include never returns. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW --- Programs/CosmOS/Assembler/source.asm | 154 +++++++++++++++++++++++++++ Programs/CosmOS/Assembler/token.asm | 16 +-- Tests/native.sh | 32 +++++- 3 files changed, 195 insertions(+), 7 deletions(-) diff --git a/Programs/CosmOS/Assembler/source.asm b/Programs/CosmOS/Assembler/source.asm index 8993e7b..89eed7a 100644 --- a/Programs/CosmOS/Assembler/source.asm +++ b/Programs/CosmOS/Assembler/source.asm @@ -105,6 +105,143 @@ srcRewindNo: ADD ; Q is not zero: it is not. RET +; ---- Everything up to and including the next newline, thrown away ---- +; +; Q is zero if a newline was found, and something else if the file ended first. +; +; THIS IS MOST OF WHAT THE ASSEMBLER READS. Measured on this repository: 87 per cent of what +; Say.asm pulls in is comment and blank line, and 59 per cent of CosmOS - the house style is +; dense commentary, so the assembler is penalised more than most by its own sources. Every one +; of those bytes used to go through tokGet and srcNext and be classified and thrown away. +; +; The saving is not the classifying, it is the BOOKKEEPING. srcNext loads and stores the +; walking pointer through memory for every single character, and asks numCompare whether the +; buffer is used up. Here the pointer lives in DP0 for a whole run and the newline lives in B, +; so a comment byte costs a load, a compare and two steps. +; +; A run is capped at 255 so that one byte can count it, which is why the outer loop exists at +; all: a full block is 256, one more than a byte holds. +srcSkipComment: + SETD.0 SrcEnded + LDA.0 + BNA srcSkipEnded + +srcSkipBlock: + ; What is left of the buffer. + SETD.0 SrcSpan + SETD.2 SrcCount + CALL numSet + SETD.0 SrcSpan + SETD.2 SrcAt + CALL numTake + + SETD.0 SrcSpan + LDA.0 + INCD.0 + LDB.0 + OR + BRQ srcSkipRefill ; Used up, so the next block or the end. + + SETD.0 SrcSpan + LDA.0 + BNA srcSkipCapped ; A high half at all means 256, which is one too many to count. + INCD.0 + LDA.0 + BRI srcSkipCounted +srcSkipCapped: + INIA 0xFF +srcSkipCounted: + SETD.0 SrcRun + STA.0 + SETD.0 SrcTaken + STA.0 ; Where the run began, to work out what it swallowed. + + SETD.2 SrcPointer + LDD.0.2 + SETD.1 SrcRun + INIB 0x0A ; The newline, held in B for the whole run. + +srcSkipChar: + LDA.0 + XOR + BRQ srcSkipNewline + INCD.0 + LDA.1 + DECA + STA.1 + BNA srcSkipChar + + ; The run ended without a newline in it, so the rest of the block is comment too. + SETD.2 SrcPointer + STD.0.2 + CALL srcSkipAccount + BRI srcSkipBlock + +srcSkipNewline: + ; Past the newline itself, which belongs to the comment. + INCD.0 + SETD.1 SrcRun + LDA.1 + DECA + STA.1 + SETD.2 SrcPointer + STD.0.2 + CALL srcSkipAccount + + ; A newline is what makes the next character part of the next line, counted as it goes past + ; exactly as srcNext counts it. + SETD.0 SrcLine + CALL numStep + RSTA + RSTB + CCF + ADD + RET + +srcSkipRefill: + CALL srcLoad + BRQ srcSkipBlock ; Another block of the same file. + + ; ---- A file that runs out inside a comment ---- + ; + ; The same thing srcAtEnd does for a character, and it has to be done the same way here: an + ; included file ending puts the one that included it back on, and the comment carries on + ; into it exactly as it did when this was read a character at a time. + ; + ; NOT DOING THIS WAS THE WHOLE OF THE FIRST VERSION'S BUG. services.asm ends inside a + ; comment, so an include of it never came back, and every label after the #Include line - + ; the whole of the program doing the including - was quietly never defined. It surfaced as + ; "no label of that name is defined anywhere" pointing at a line whose label was three + ; lines from the end of the file. + INIA 0d1 + SETD.0 SrcEnded + STA.0 + SETD.0 SrcDepth + LDA.0 + BRA srcSkipEnded ; Nothing included this, so there is no more source anywhere. + CALL srcPop + BRI srcSkipComment + +srcSkipEnded: + INIA 0x01 + RSTB + CCF + ADD + RET + +; How much of the run went by, added to how far into the buffer we have read. +srcSkipAccount: + SETD.0 SrcTaken + LDA.0 + SETD.0 SrcRun + LDB.0 + CCF + SUB + MVQA + SETD.0 SrcAt + CALL numAddByte + RET + ; The next character of the file, into SrcChar. Q is zero if there was one, and something ; else at the end of the file. srcNext: @@ -576,6 +713,23 @@ SrcStateBytes: SrcDepthLimit: 0d6 +; ---- OUTSIDE the state block above, and that is not an accident ---- +; +; What is left of the buffer, how much of it one run may take, and where that run began. +; They live for the length of one call to srcSkipComment and never across an include, so they +; are not part of what a file remembers while another one is being read. +; +; PUT AMONG THE STATE FIRST, which added four bytes to a block that is copied whole by a +; count written down somewhere else - so every saved file lost the last four bytes of itself +; and an include came back with its buffer pointer wrong. The comment two paragraphs up says +; not to do it, and says it in capitals. +SrcSpan: + 0x00 0x00 +SrcRun: + 0x00 +SrcTaken: + 0x00 + ; Where an include is looked for when it is not beside you. One fixed place rather than a ; list somebody sets, for the same reason the shell has one fixed place for programs: a ; list would need somewhere to live between one boot and the next. diff --git a/Programs/CosmOS/Assembler/token.asm b/Programs/CosmOS/Assembler/token.asm index a2869c0..b586ffb 100644 --- a/Programs/CosmOS/Assembler/token.asm +++ b/Programs/CosmOS/Assembler/token.asm @@ -54,13 +54,17 @@ tokSkip: BNQ tokBegin tokComment: - CALL tokGet + ; ---- Thrown away by the reader, not read one character at a time ---- + ; + ; This used to walk the comment through tokGet and srcNext, classifying every byte of it on + ; the way to discarding it. Comments are most of what this assembler reads - 87 per cent of + ; what a small program pulls in, once the libraries it includes are counted - so it was most + ; of the work, and all of it thrown away. + ; + ; A pushed back character cannot be sitting there: this is only reached from tokSkip, which + ; got the semicolon out of tokGet and did not put anything back. + CALL srcSkipComment BNQ tokEnded - SETD.0 TokChar - LDA.0 - INIB 0x0A - XOR - BNQ tokComment ; Everything up to the newline belongs to the comment. BRI tokSkip tokBegin: diff --git a/Tests/native.sh b/Tests/native.sh index 14db73a..7757b54 100755 --- a/Tests/native.sh +++ b/Tests/native.sh @@ -71,13 +71,37 @@ APPS="Say greet Files Keys" "$TOOL" put "$WORK/native.img" "$ROOT/Programs/CosmOS/Source/services.asm" services.asm >/dev/null "$TOOL" put "$WORK/native.img" "$ROOT/Programs/CosmOS/Source/console.asm" console.asm >/dev/null for app in $APPS; do + [ -f "$ROOT/Programs/CosmOS/Apps/$app.asm" ] || continue "$TOOL" put "$WORK/native.img" "$ROOT/Programs/CosmOS/Apps/$app.asm" "$app.asm" >/dev/null done +# ---- A file that ends in the middle of a comment ---- +# +# Generated rather than committed, because what makes it work is the ABSENCE of a newline on +# the last line and no editor can be trusted to leave that alone. +# +# An included file running out inside a comment is the case the reader has to pop the include +# stack for, and it is the one shape no source in this repository happens to have: every file +# here ends on a line of code with a newline after it. Nothing exercised that path, so an +# error in it could not be caught - and comment skipping is exactly where such an error would +# live, since a comment is the only thing that reads past the end of a line without looking. +# +# usestail.asm names a label AFTER the include, which is what goes missing if the file that +# did the including is not put back: the include never returns, the rest of the program is +# never read, and the label is never defined. +printf '; nothing but a comment, and no newline after it' > "$WORK/tail.asm" +printf '#Include tail.asm\n#Program\n #Base 0x5000\nstart:\n SETD.0 Later\n HALT\n#Data\n #Base 0x3000\nLater:\n 0x00\n' > "$WORK/usestail.asm" +"$TOOL" put "$WORK/native.img" "$WORK/tail.asm" tail.asm >/dev/null +"$TOOL" put "$WORK/native.img" "$WORK/usestail.asm" usestail.asm >/dev/null + { echo "load Asm.sbx" echo "run hello.asm" - for app in $APPS; do echo "run $app.asm"; done + echo "run usestail.asm" + for app in $APPS; do + [ -f "$ROOT/Programs/CosmOS/Apps/$app.asm" ] || continue + echo "run $app.asm" + done echo "exit" } | "$EMU" --fast --cycles 600000000 -D "$WORK/native.img" "$WORK/cosmos.bin" \ > "$WORK/session.txt" 2>&1 @@ -85,6 +109,12 @@ done # ---- It got as far as writing something ---- check "it wrote a file" grep -q "wrote hello.bin" "$WORK/session.txt" +# ---- And a file that ends inside a comment did not swallow the one that included it ---- +# +# The label is defined after the #Include, so it exists only if the reader came back. +check "an include that ends in a comment comes back" \ + grep -q "wrote usestail.sbx" "$WORK/session.txt" + # ---- And the file is the one the host assembler makes ---- "$ASM" -o "$WORK/reference.bin" "$ROOT/Programs/Examples/hello.asm" >/dev/null "$TOOL" get "$WORK/native.img" hello.bin "$WORK/native.bin" >/dev/null 2>&1