Files
SplitBit-Emulator/Programs/CosmOS/Assembler/tokenTest.asm
T
AnachronautandClaude Opus 5 c5e4ec3455 M2: the native assembler builds applications
> load Asm.sbx
    > run Say.asm
    wrote Say.sbx: program 46, data 93, labels 7
    > load Say.sbx
    > run built by the machine itself
    it says: built by the machine itself

The machine assembles an application and then runs what it built. Say,
greet and Files all come out byte for byte identical to the C assembler's,
and Tests/native.sh checks all three on every run alongside the boot image
M1 already covered.

WHAT IT TOOK, and it was more than #Include and #Base:

  #Include   The reader is a stack of readers. The current file's whole
             state goes aside - buffer and all, 292 bytes - the new one
             opens, and the end of it pops the old one back. A file goes in
             once; including it twice does nothing, which is what lets two
             libraries depend on a third. The list is forgotten between the
             passes, because the second has to walk the same tree.
  #Base      Cursors start there, so labels hold the addresses the program
             will really have. A program that says where it goes gets the
             SBEX header and a .sbx name; one that says nothing gets SPBT
             and .bin. A program that bases one segment and leaves the
             other unbased with content in it is refused.
  #Reserve   Runs of zeroes, moved over in the first pass and written in
  #Align     the second. How many an #Align comes to depends on where the
             cursor has reached, which is why both passes keep a cursor.
  #Vectors   Names are read and numbered, pinned where the source pins
             them, so SWI osPrintString resolves. Every application needs
             this - a program that calls a service names a vector declared
             in a file it includes.

THE TWO PASSES ARE NOW ONE LOOP, walked twice, with Emitting the only
difference. They have to agree about the length of every token, and the way
they stop agreeing is by being two pieces of code that drifted apart -
which is the exact shape of the bug this assembler found in the C one.
Sharing the body means there is nothing to drift. What is left is checked
anyway: the second pass compares its own totals against the first's and
refuses to write the file if they differ.

THE BUG WORTH RECORDING. The tokenizer holds one character of lookahead,
and at an #Include that character belongs to the file being put aside. It
was carried across and handed back on the way out, which is wrong: a file
runs out in the middle of whatever the tokenizer happens to be doing, so
the character arrives in the middle of a word. `start:` came back as `s`
and then `tart:` - and the result assembled into a perfectly plausible
file. The fix is to undo the read instead, so the character is simply still
there when the file is opened again and the question of when to hand it
back never arises.

Three smaller ones, all old friends: three places took the CONTENTS of a
buffer where they wanted its ADDRESS; vecTakeAuto returned its answer in A,
which a RET puts back; and pass two re-declared every vector because only
the label table was being skipped on the second walk.

sameText moved down into numbers.asm from the label table - four parts want
it now, and a reader test that includes neither labels nor tokens has to
build on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW
2026-08-20 22:50:39 -04:00

177 lines
3.1 KiB
NASM

; The assembler's front end on its own, one line per token.
;
; Prints the line each token started on, what the token turned out to be, how many bytes
; it will come to, and the token itself between brackets so that whitespace at either end
; would show if any ever leaked in.
;
; WHAT A TOKEN IS is the part worth checking here rather than at the far end. A wrong
; classification does not produce a wrong byte in an obvious place - it produces a right
; looking program of the wrong length, with everything after it shifted, and by then the
; only symptom is that a label points at the middle of an instruction.
;
; Written by Anachronaut
#Include services.asm
#Program
#Base 0x2000
start:
SETD.0 Wanted
INIB 0d23
SWI osArgument
SETD.0 Wanted
LDA.0
BRA nothingAsked
SETD.0 Wanted
CALL srcOpen
BNQ noFile
tokenLoop:
CALL tokNext
BNQ tokensDone
SETD.0 TokLine
LDA.0
INCD.0
LDB.0
SWI osPrintNumber
CALL clsToken
BNQ badToken
; Which of the six it turned out to be. The names are four characters and a space, so
; the columns line up without any counting.
SETD.0 TypeNames
SETD.2 ClsType
LDA.2
CALL nameOfType
SETD.1 NamePointer
LDD.0.1
SWI osPrintString
SETD.0 ClsLength
LDA.0
INCD.0
LDB.0
SWI osPrintNumber ; Sixteen bits: a string of 255 characters is 256 bytes long.
SETD.0 OpenMark
SWI osPrintString
SETD.0 TokText
SWI osPrintString
SETD.0 CloseMark
SWI osPrintString
BRI tokenLoop
badToken:
SETD.0 StoppedText
SWI osPrintString
SWI osExit
; The name of type A, out of a table of fixed width entries so that no pointer arithmetic
; is needed beyond a multiply by the width.
nameOfType:
SETD.0 TypeWidth
LDB.0
RSTA
SETD.0 NameLeft
STA.0
SETD.2 ClsType
LDA.2
SETD.0 NameOffset
CALL numZero
nameLoop:
SETD.2 ClsType
LDA.2
SETD.0 NameLeft
LDB.0
CCF
SUB
BRQ nameFound
SETD.0 TypeWidth
LDA.0
SETD.0 NameOffset
CALL numAddByte
SETD.0 NameLeft
LDA.0
INCA
STA.0
BRI nameLoop
nameFound:
; The answer is left in NamePointer rather than in DP0, because a RET puts Data Pointers
; 0 to 2 back as they were: a routine cannot hand back a pointer, only write one down.
SETD.0 TypeNames
SETD.1 NamePointer
STD.0.1
SETD.0 NamePointer
SETD.2 NameOffset
CALL numAdd
RET
tokensDone:
SETD.0 DoneText
SWI osPrintString
SWI osExit
nothingAsked:
SETD.0 AskText
SWI osPrintString
SWI osExit
noFile:
SETD.0 NoFileText
SWI osPrintString
SWI osExit
#Data
#Base 0x1000
Wanted:
#Reserve 0d23
OpenMark:
" ["
CloseMark:
"]
"
; Six names of eleven characters each, counting the zero the assembler puts on the end of
; every string. Written one to a line so that adding a type is adding a line.
TypeNames:
" keyword "
" instr "
" value "
" string "
" label: "
" label "
TypeWidth:
0d11
NameLeft:
0x00
NameOffset:
0x00 0x00
NamePointer:
0x00 0x00
StoppedText:
"---- stopped: the assembler does not understand that
"
DoneText:
"---- no more tokens
"
AskText:
"say which file
"
NoFileText:
"no such file
"
#Include numbers.asm
#Include source.asm
#Include token.asm
#Include classify.asm
#Include table.asm