AnachronautandClaude Opus 5
2e9cb9af7e
A comment used to walk through tokGet and srcNext and be classified,
character by character, on the way to being discarded. Comments are most
of what this assembler reads: 87 per cent of what Say.asm pulls in once
services.asm is counted, 59 per cent of CosmOS. The house style here is
dense commentary, so the assembler is penalised more than most by its own
sources.
The saving is not the classifying, it is the BOOKKEEPING. srcNext loads
and stores the walking pointer through memory for every character and
asks numCompare whether the buffer is used up. srcSkipComment keeps the
pointer in a data pointer for a whole run and the newline in B, so a
comment byte costs a load, a compare and two steps. A run is capped at
255 so one byte can count it, which is the only reason it loops.
Measured, each version with its own rebuilt images:
without with
hello 791,957 586,184
Say 9,924,401 4,506,702 2.20x
Files 13,197,710 7,188,351 1.84x
Keys 23,091,447 15,069,880 1.53x
cosmos 886,498,996 789,982,899 1.12x
Which tracks the comment ratios: Say gains most and cosmos least, in
proportion to how much of each is prose.
---- And two mistakes worth keeping ----
The scratch went among the READER'S STATE, which is a block copied whole
by a count written down somewhere else - so every saved file lost the
last four bytes of itself and an include came back with its pointer
wrong. The comment above that block says not to do this, in capitals.
That is twice this week: scriptCopyState had the same shape this morning.
And a file that ends inside a comment has to put back the file that
included it, exactly as srcAtEnd does for a character. NOTHING IN THIS
REPOSITORY ENDS THAT WAY - every source here ends on a line of code with
a newline after it - so break.sh could not catch an error in that path
because nothing reached it. tail.asm is generated with no newline on its
last line for that reason, and usestail.asm names a label after the
include, which is what goes missing when the include never returns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW
298 lines
6.6 KiB
NASM
298 lines
6.6 KiB
NASM
; Tokens out of characters.
|
|
;
|
|
; A token is a run of characters with whitespace or a comment on either side, or anything
|
|
; between a pair of quotes. That is the whole of the lexical grammar: SplitBit assembly has
|
|
; no operators, no punctuation and no line continuation, so there is nothing here that has
|
|
; to look ahead more than one character.
|
|
;
|
|
; ONE CHARACTER OF LOOKAHEAD, and it is held here rather than in the reader. A word ends
|
|
; when something that is not part of it turns up, and that something has already been read
|
|
; by the time anyone knows - so it is put in TokPending and taken again next time. Keeping
|
|
; it at this level rather than pushing it back into the reader means the line number needs
|
|
; no arithmetic: srcNext counted the newline when it handed it out, and it stays counted.
|
|
;
|
|
; Zero means "nothing held", which is safe because a source file is text and a text file
|
|
; has no zero bytes in it. A file that did would be rejected as unassemblable long before
|
|
; the difference showed.
|
|
;
|
|
; Written by Anachronaut
|
|
|
|
#Program
|
|
|
|
; The next token, into TokText with a zero after it. Q is zero if there was one.
|
|
;
|
|
; TokLength is how long it is, TokString says whether it arrived in quotes, and TokLine is
|
|
; the line it STARTED on - captured before the token is read, because a token ending in a
|
|
; newline has already moved the reader on to the next line by the time it is finished.
|
|
tokNext:
|
|
SETD.0 TokHeld
|
|
LDA.0
|
|
BRA tokFresh
|
|
RSTA
|
|
STA.0
|
|
RSTA
|
|
RSTB
|
|
CCF
|
|
ADD ; The one that was handed back, exactly as it was.
|
|
RET
|
|
|
|
tokFresh:
|
|
RSTA
|
|
SETD.0 TokString
|
|
STA.0
|
|
|
|
tokSkip:
|
|
CALL tokGet
|
|
BNQ tokEnded
|
|
CALL tokIsSpace
|
|
BRQ tokSkip
|
|
|
|
SETD.0 TokChar
|
|
LDA.0
|
|
INIB 0x3B ; A semicolon starts a comment.
|
|
XOR
|
|
BNQ tokBegin
|
|
|
|
tokComment:
|
|
; ---- Thrown away by the reader, not read one character at a time ----
|
|
;
|
|
; This used to walk the comment through tokGet and srcNext, classifying every byte of it on
|
|
; the way to discarding it. Comments are most of what this assembler reads - 87 per cent of
|
|
; what a small program pulls in, once the libraries it includes are counted - so it was most
|
|
; of the work, and all of it thrown away.
|
|
;
|
|
; A pushed back character cannot be sitting there: this is only reached from tokSkip, which
|
|
; got the semicolon out of tokGet and did not put anything back.
|
|
CALL srcSkipComment
|
|
BNQ tokEnded
|
|
BRI tokSkip
|
|
|
|
tokBegin:
|
|
; Where it starts, for anything that has to complain about it later.
|
|
SETD.0 TokLine
|
|
SETD.2 SrcLine
|
|
CALL numSet
|
|
|
|
RSTA
|
|
SETD.0 TokLength
|
|
STA.0
|
|
SETD.0 TokText
|
|
SETD.1 TokPointer
|
|
STD.0.1
|
|
|
|
SETD.0 TokChar
|
|
LDA.0
|
|
INIB 0x22 ; A quote starts a string.
|
|
XOR
|
|
BNQ tokWord
|
|
|
|
INIA 0d1
|
|
SETD.0 TokString
|
|
STA.0
|
|
|
|
tokStringLoop:
|
|
CALL tokGet
|
|
BNQ tokDone ; The file ended inside a string. Take what there is; the
|
|
; classifier will have something to complain about.
|
|
SETD.0 TokChar
|
|
LDA.0
|
|
INIB 0x22
|
|
XOR
|
|
BRQ tokDone
|
|
CALL tokAppend
|
|
BRI tokStringLoop
|
|
|
|
tokWord:
|
|
CALL tokAppend
|
|
tokWordLoop:
|
|
CALL tokGet
|
|
BNQ tokDone
|
|
CALL tokIsSpace
|
|
BRQ tokHoldDone
|
|
SETD.0 TokChar
|
|
LDA.0
|
|
INIB 0x3B
|
|
XOR
|
|
BRQ tokHoldDone ; A comment butting straight up against a word ends it.
|
|
CALL tokAppend
|
|
BRI tokWordLoop
|
|
|
|
tokHoldDone:
|
|
; Whatever ended the word was not part of it, so it goes back to be looked at again.
|
|
SETD.0 TokChar
|
|
LDA.0
|
|
SETD.0 TokPending
|
|
STA.0
|
|
|
|
tokDone:
|
|
SETD.1 TokPointer
|
|
LDD.0.1
|
|
RSTA
|
|
STA.0 ; The zero that makes it a string.
|
|
RSTA
|
|
RSTB
|
|
CCF
|
|
ADD ; Q is zero: there was a token.
|
|
RET
|
|
|
|
tokEnded:
|
|
RSTA
|
|
SETD.0 TokLength
|
|
STA.0
|
|
SETD.0 TokText
|
|
STA.0
|
|
RSTA
|
|
INIB 0d1
|
|
CCF
|
|
ADD ; Q is not zero: the source is finished.
|
|
RET
|
|
|
|
; Puts the held character back into the file it came from, leaving nothing in hand.
|
|
;
|
|
; This is what #Include calls before the reader puts the current file aside. The character
|
|
; the tokenizer is holding was read from that file and has not been used, so it goes back
|
|
; into it; there is then nothing to carry across the include and nothing to hand back at a
|
|
; moment that might land in the middle of a word.
|
|
tokUnread:
|
|
SETD.0 TokPending
|
|
LDA.0
|
|
BRA tokUnreadDone
|
|
CALL srcStepBack
|
|
RSTA
|
|
SETD.0 TokPending
|
|
STA.0
|
|
tokUnreadDone:
|
|
RET
|
|
|
|
; Hands the token just read back, so that the next tokNext produces it again.
|
|
;
|
|
; ONE TOKEN, and only where nothing has changed it since. The Vector Segment needs it: a
|
|
; name there may be followed by a number, by a handler, or by the next line's name, and
|
|
; which it is cannot be known without looking. Do NOT use it after anything that alters
|
|
; TokText - a label definition with its colon written over would come back as a use of the
|
|
; name rather than as a definition of it.
|
|
tokBack:
|
|
INIA 0d1
|
|
SETD.0 TokHeld
|
|
STA.0
|
|
RET
|
|
|
|
; The next character, into TokChar. Q is zero if there was one. Takes the held one first.
|
|
tokGet:
|
|
SETD.0 TokPending
|
|
LDA.0
|
|
BRA tokGetFresh
|
|
SETD.0 TokChar
|
|
STA.0
|
|
RSTA
|
|
SETD.0 TokPending
|
|
STA.0
|
|
RSTA
|
|
RSTB
|
|
CCF
|
|
ADD
|
|
RET
|
|
|
|
tokGetFresh:
|
|
CALL srcNext
|
|
BNQ tokGetNone
|
|
SETD.0 SrcChar
|
|
LDA.0
|
|
SETD.0 TokChar
|
|
STA.0
|
|
RSTA
|
|
RSTB
|
|
CCF
|
|
ADD
|
|
RET
|
|
|
|
tokGetNone:
|
|
RSTA
|
|
INIB 0d1
|
|
CCF
|
|
ADD
|
|
RET
|
|
|
|
; Adds TokChar to the token being built, unless it is already as long as one may be.
|
|
;
|
|
; A token that runs over is truncated rather than refused, and the classifier refuses it
|
|
; afterwards: nothing 255 characters long is a valid mnemonic, literal or label, so the
|
|
; error that comes out names what was wrong with it rather than only how long it was.
|
|
tokAppend:
|
|
SETD.0 TokLength
|
|
LDA.0
|
|
INIB 0xFF
|
|
XOR
|
|
BRQ tokAppendFull
|
|
SETD.1 TokPointer
|
|
LDD.0.1
|
|
SETD.2 TokChar
|
|
LDA.2
|
|
STA.0
|
|
INCD.0
|
|
STD.0.1
|
|
SETD.0 TokLength
|
|
LDA.0
|
|
INCA
|
|
STA.0
|
|
tokAppendFull:
|
|
RET
|
|
|
|
; Q is zero if TokChar is whitespace: a space, or anything in the run from tab to carriage
|
|
; return, which is what the C library calls a space and what the other assembler uses.
|
|
tokIsSpace:
|
|
SETD.0 TokChar
|
|
LDA.0
|
|
INIB 0x20
|
|
XOR
|
|
BRQ tokSpaceYes
|
|
|
|
SETD.0 TokChar
|
|
LDA.0
|
|
INIB 0x09
|
|
CCF
|
|
SUB
|
|
BRC tokSpaceNo ; Below a tab.
|
|
SETD.0 TokChar
|
|
LDA.0
|
|
INIB 0x0E
|
|
CCF
|
|
SUB
|
|
BNC tokSpaceNo ; Past a carriage return.
|
|
|
|
tokSpaceYes:
|
|
RSTA
|
|
RSTB
|
|
CCF
|
|
ADD
|
|
RET
|
|
|
|
tokSpaceNo:
|
|
RSTA
|
|
INIB 0d1
|
|
CCF
|
|
ADD
|
|
RET
|
|
|
|
#Data
|
|
|
|
TokLine:
|
|
0x00 0x00
|
|
TokPointer:
|
|
0x00 0x00
|
|
TokLength:
|
|
0x00
|
|
TokString:
|
|
0x00
|
|
TokChar:
|
|
0x00
|
|
TokPending:
|
|
0x00
|
|
TokHeld:
|
|
0x00
|
|
|
|
; As long as a token may be, and one more for the zero. The other assembler stops at the
|
|
; same 255, and the limit is worth matching rather than choosing again.
|
|
TokText:
|
|
#Reserve 0d256
|