M2: the native assembler builds applications

> load Asm.sbx
    > run Say.asm
    wrote Say.sbx: program 46, data 93, labels 7
    > load Say.sbx
    > run built by the machine itself
    it says: built by the machine itself

The machine assembles an application and then runs what it built. Say,
greet and Files all come out byte for byte identical to the C assembler's,
and Tests/native.sh checks all three on every run alongside the boot image
M1 already covered.

WHAT IT TOOK, and it was more than #Include and #Base:

  #Include   The reader is a stack of readers. The current file's whole
             state goes aside - buffer and all, 292 bytes - the new one
             opens, and the end of it pops the old one back. A file goes in
             once; including it twice does nothing, which is what lets two
             libraries depend on a third. The list is forgotten between the
             passes, because the second has to walk the same tree.
  #Base      Cursors start there, so labels hold the addresses the program
             will really have. A program that says where it goes gets the
             SBEX header and a .sbx name; one that says nothing gets SPBT
             and .bin. A program that bases one segment and leaves the
             other unbased with content in it is refused.
  #Reserve   Runs of zeroes, moved over in the first pass and written in
  #Align     the second. How many an #Align comes to depends on where the
             cursor has reached, which is why both passes keep a cursor.
  #Vectors   Names are read and numbered, pinned where the source pins
             them, so SWI osPrintString resolves. Every application needs
             this - a program that calls a service names a vector declared
             in a file it includes.

THE TWO PASSES ARE NOW ONE LOOP, walked twice, with Emitting the only
difference. They have to agree about the length of every token, and the way
they stop agreeing is by being two pieces of code that drifted apart -
which is the exact shape of the bug this assembler found in the C one.
Sharing the body means there is nothing to drift. What is left is checked
anyway: the second pass compares its own totals against the first's and
refuses to write the file if they differ.

THE BUG WORTH RECORDING. The tokenizer holds one character of lookahead,
and at an #Include that character belongs to the file being put aside. It
was carried across and handed back on the way out, which is wrong: a file
runs out in the middle of whatever the tokenizer happens to be doing, so
the character arrives in the middle of a word. `start:` came back as `s`
and then `tart:` - and the result assembled into a perfectly plausible
file. The fix is to undo the read instead, so the character is simply still
there when the file is opened again and the question of when to hand it
back never arises.

Three smaller ones, all old friends: three places took the CONTENTS of a
buffer where they wanted its ADDRESS; vecTakeAuto returned its answer in A,
which a RET puts back; and pass two re-declared every vector because only
the label table was being skipped on the second walk.

sameText moved down into numbers.asm from the label table - four parts want
it now, and a reader test that includes neither labels nor tokens has to
build on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW

Programs/CosmOS/Assembler/classify.asm

@@ -20,9 +20,8 @@
 
 ; Works out what TokText is. Q is zero if it is something the assembler understands.
 clsToken:
-  RSTA
   SETD.0 ClsLength
-  STA.0
+  CALL numZero
 
   SETD.0 TokString
   LDA.0
@@ -62,8 +61,7 @@ clsTryValue:
   SETD.0 ClsType
   STA.0
   INIA 0d1
-  SETD.0 ClsLength
+  CALL clsSetLength
-  STA.0
   BRI clsYes
 
 clsIsString:
@@ -72,11 +70,18 @@ clsIsString:
   STA.0
   ; A string is its characters and the zero byte after them, which is why two strings
   ; written in a row are two strings rather than one long one.
-  SETD.0 TokLength
+  ;
-  LDA.0
+  ; SIXTEEN BITS, and this is the token that needs them: a string may be 255 characters,
-  INCA
+  ; which with its zero is 256, and 256 does not fit in a byte. Everything else here is 0,
+  ; 1, 2 or 3.
   SETD.0 ClsLength
-  STA.0
+  CALL numZero
+  SETD.2 TokLength
+  LDA.2
+  SETD.0 ClsLength
+  CALL numAddByte
+  SETD.0 ClsLength
+  CALL numStep
   BRI clsYes
 
 clsTryLabel:
@@ -98,8 +103,7 @@ clsUse:
   SETD.0 ClsType
   STA.0
   INIA 0d2
-  SETD.0 ClsLength
+  CALL clsSetLength
-  STA.0
 
 clsYes:
   RSTA
@@ -187,8 +191,7 @@ clsSelectorsFit:
   SETD.0 ClsWanted
   LDA.0
   INCA
-  SETD.0 ClsLength
+  CALL clsSetLength         ; The opcode and its selectors. The operand is its own token.
-  STA.0                     ; The opcode and its selectors. The operand is its own token.
   RSTA
   RSTB
   CCF
@@ -409,105 +412,29 @@ clsSameDone:
 ; Is TokText a well formed literal? Q is zero if it is, and ClsValue is what it comes to.
 ; Anything beginning with a zero has to be one, so a failure here is an error rather than
 ; an invitation to try the next test.
+;
+; A LITERAL IS ONE BYTE WHEREVER IT GOES, so this is the byte-wide door onto clsWord below.
+; The directives are the wide one: #Base takes an address and #Reserve a count, and neither
+; would fit through here.
 clsValue:
-  SETD.0 TokText
+  CALL clsWord
+  BNQ clsValueNo
+  SETD.0 ClsWord
+  LDA.0
+  BNA clsValueTooBig        ; Something in the high byte, so it will not fit in one.
   INCD.0
   LDA.0
-  INIB 0x78                 ; 'x'
-  XOR
-  BRQ clsValueHex
-  SETD.0 TokText
-  INCD.0
-  LDA.0
-  INIB 0x64                 ; 'd'
-  XOR
-  BRQ clsValueDecimal
-
-  SETD.0 BadPrefix
-  CALL clsComplain
-  BRI clsValueNo
-
-clsValueHex:
-  INIA 0d16
-  SETD.0 ClsBase
-  STA.0
-  BRI clsValueDigits
-
-clsValueDecimal:
-  INIA 0d10
-  SETD.0 ClsBase
-  STA.0
-
-clsValueDigits:
-  SETD.0 TokLength
-  LDA.0
-  INIB 0d3
-  CCF
-  SUB
-  BRC clsValueEmpty         ; Only the prefix, so there are no digits at all.
-
-  RSTA
   SETD.0 ClsValue
   STA.0
-  SETD.0 TokText
-  INCD.0
-  INCD.0
-  SETD.1 ClsWalk
-  STD.0.1
-
-clsValueLoop:
-  SETD.1 ClsWalk
-  LDD.0.1
-  LDA.0
-  BRA clsValueGood
-  CALL clsDigit
-  BNQ clsValueBadDigit
-
-  ; value = value * base + digit, and anything that will not fit in a byte is refused
-  ; rather than wrapped, because a literal is one byte wherever it goes.
-  SETD.0 ClsDigitValue
-  LDA.0
-  SETD.2 ClsValue
-  LDB.2
-  PSHA
-  SETD.0 ClsBase
-  LDA.0
-  CALL clsMultiply
-  BNQ clsValueTooBig
-  POPA
-  SETD.0 ClsProduct
-  LDB.0
-  CCF
-  ADD
-  BRC clsValueTooBig
-  MVQA
-  SETD.0 ClsValue
-  STA.0
-
-  SETD.0 ClsWalk
-  CALL numStep
-  BRI clsValueLoop
-
-clsValueGood:
   RSTA
   RSTB
   CCF
   ADD
   RET
 
-clsValueEmpty:
-  SETD.0 NoDigits
-  CALL clsComplain
-  BRI clsValueNo
-clsValueBadDigit:
-  SETD.0 BadDigit
-  CALL clsComplain
-  BRI clsValueNo
 clsValueTooBig:
-  POPA
   SETD.0 TooBig
   CALL clsComplain
-
 clsValueNo:
   RSTA
   INIB 0d1
@@ -515,6 +442,158 @@ clsValueNo:
   ADD
   RET
 
+; Reads TokText as a sixteen bit number, into ClsWord. Q is zero if it is a well formed one.
+;
+; Both bases are here rather than in two routines because the only difference is which
+; digits count and what to multiply by, and a number is written the same way wherever it
+; appears - an address after #Base, a count after #Reserve, a byte in a segment.
+clsWord:
+  SETD.0 TokText
+  INCD.0
+  LDA.0
+  INIB 0x78                 ; 'x'
+  XOR
+  BRQ clsWordHex
+  SETD.0 TokText
+  INCD.0
+  LDA.0
+  INIB 0x64                 ; 'd'
+  XOR
+  BRQ clsWordDecimal
+
+  SETD.0 BadPrefix
+  CALL clsComplain
+  BRI clsWordNo
+
+clsWordHex:
+  INIA 0d16
+  SETD.0 ClsBase
+  STA.0
+  BRI clsWordDigits
+
+clsWordDecimal:
+  INIA 0d10
+  SETD.0 ClsBase
+  STA.0
+
+clsWordDigits:
+  SETD.0 TokLength
+  LDA.0
+  INIB 0d3
+  CCF
+  SUB
+  BRC clsWordEmpty          ; Only the prefix, so there are no digits at all.
+
+  SETD.0 ClsWord
+  CALL numZero
+  SETD.0 TokText
+  INCD.0
+  INCD.0
+  SETD.1 ClsWalk
+  STD.0.1
+
+clsWordLoop:
+  SETD.1 ClsWalk
+  LDD.0.1
+  LDA.0
+  BRA clsWordGood
+  CALL clsDigit
+  BNQ clsWordBadDigit
+
+  CALL clsWordTimesBase
+  BNQ clsWordTooBig
+
+  ; And the digit on the end. A sum that comes out smaller than what went into it is a sum
+  ; that went past sixteen bits, which is the only test needed and costs one comparison.
+  RSTA
+  SETD.0 ClsDigitWord
+  STA.0
+  INCD.0
+  SETD.2 ClsDigitValue
+  LDA.2
+  STA.0
+  SETD.0 ClsWord
+  SETD.2 ClsDigitWord
+  CALL numAdd
+  SETD.0 ClsWord
+  SETD.2 ClsDigitWord
+  CALL numCompare
+  BRC clsWordTooBig
+
+  SETD.0 ClsWalk
+  CALL numStep
+  BRI clsWordLoop
+
+clsWordGood:
+  RSTA
+  RSTB
+  CCF
+  ADD
+  RET
+
+clsWordEmpty:
+  SETD.0 NoDigits
+  CALL clsComplain
+  BRI clsWordNo
+clsWordBadDigit:
+  SETD.0 BadDigit
+  CALL clsComplain
+  BRI clsWordNo
+clsWordTooBig:
+  SETD.0 TooBigWord
+  CALL clsComplain
+clsWordNo:
+  RSTA
+  INIB 0d1
+  CCF
+  ADD
+  RET
+
+; ClsWord becomes itself times ClsBase. Q is not zero if that went past sixteen bits.
+;
+; By repeated addition, because this machine has no multiply. The base is ten or sixteen,
+; so it is at most sixteen additions per digit, and a number in a source file has four or
+; five digits.
+clsWordTimesBase:
+  SETD.0 ClsAccum
+  CALL numZero
+  SETD.0 ClsMulLeft
+  SETD.2 ClsBase
+  LDA.2
+  STA.0
+
+clsWordMulLoop:
+  SETD.0 ClsMulLeft
+  LDA.0
+  BRA clsWordMulDone
+  DECA
+  STA.0
+  SETD.0 ClsAccum
+  SETD.2 ClsWord
+  CALL numAdd
+  SETD.0 ClsAccum
+  SETD.2 ClsWord
+  CALL numCompare
+  BRC clsWordMulOver        ; It came out smaller than what was added, so it wrapped.
+  BRI clsWordMulLoop
+
+clsWordMulDone:
+  SETD.0 ClsWord
+  SETD.2 ClsAccum
+  CALL numSet
+  RSTA
+  RSTB
+  CCF
+  ADD
+  RET
+
+clsWordMulOver:
+  RSTA
+  INIB 0d1
+  CCF
+  ADD
+  RET
+
 ; The character in A as a digit in ClsBase, into ClsDigitValue. Q is zero if it is one.
 clsDigit:
   SETD.0 ClsHold
@@ -592,44 +671,18 @@ clsDigitNo:
   ADD
   RET
 
-; B times A into ClsProduct. Q is not zero if it would not fit in a byte, which is the
-; only answer a literal can use: there is no wider literal to promote it to.
-clsMultiply:
-  SETD.0 ClsMulLeft
-  STA.0
-  RSTA
-  SETD.0 ClsProduct
-  STA.0
-clsMultiplyLoop:
-  SETD.0 ClsMulLeft
-  LDA.0
-  BRA clsMultiplyDone
-  DECA
-  STA.0
-  SETD.0 ClsProduct
-  LDA.0
-  CCF
-  ADD
-  BRC clsMultiplyOver
-  MVQA
-  SETD.0 ClsProduct
-  STA.0
-  BRI clsMultiplyLoop
-clsMultiplyDone:
-  RSTA
-  RSTB
-  CCF
-  ADD
-  RET
-clsMultiplyOver:
-  RSTA
-  INIB 0d1
-  CCF
-  ADD
-  RET
-
 ; ---- Odds and ends ----
 
+; ClsLength becomes the byte in A. Everything but a string is a small number, and this is
+; how a small number is written into a sixteen bit field.
+clsSetLength:
+  SETD.0 ClsLength
+  RSTB
+  STB.0
+  INCD.0
+  STA.0
+  RET
+
 ; The character in A, folded to upper case, into ClsByte.
 clsUpper:
   SETD.0 ClsHold
@@ -726,7 +779,7 @@ clsComplain:
 ClsType:
   0x00
 ClsLength:
-  0x00
+  0x00 0x00
 ClsOpcode:
   0x00
 ClsShape:
@@ -743,12 +796,16 @@ ClsName:
   #Reserve 0d5
 ClsValue:
   0x00
+ClsWord:
+  0x00 0x00
+ClsAccum:
+  0x00 0x00
+ClsDigitWord:
+  0x00 0x00
 ClsBase:
   0x00
 ClsDigitValue:
   0x00
-ClsProduct:
-  0x00
 ClsMulLeft:
   0x00
 ClsHold:
@@ -786,6 +843,8 @@ BadDigit:
 "that is not a digit in the base the prefix asked for"
 TooBig:
 "a literal too large to fit in one byte"
+TooBigWord:
+"a number too large to fit in sixteen bits"
 TooManySelectors:
 "more Data Pointer selectors than that instruction has pointers to name"
 BadSelector:

Programs/CosmOS/Assembler/labels.asm

@@ -173,7 +173,7 @@ labFindLoop:
   LDD.0.1
   SETD.1 LabSought
   LDD.1.1
-  CALL labSame
+  CALL sameText
   BRQ labFindGot
 
   SETD.0 LabWhich
@@ -225,21 +225,6 @@ labEntryAt:
   CALL numAdd
   RET
 
-; Q is zero if the strings at DP0 and DP1 are the same, both ending in a zero byte.
-labSame:
-  LDA.0
-  LDB.1
-  CCF
-  SUB
-  BNQ labSameDone
-  LDA.0
-  BRA labSameDone           ; They ended together, so they matched all the way.
-  INCD.0
-  INCD.1
-  BRI labSame
-labSameDone:
-  RET
-
 ; How long the string at DP0 is, counting the zero on the end, into LabLength.
 labLength:
   SETD.1 LabLenWalk

Programs/CosmOS/Assembler/numbers.asm

@@ -1,4 +1,5 @@
-; Sixteen bit arithmetic, for an assembler that counts in addresses.
+; The small things every other part of the assembler needs: sixteen bit arithmetic, for
+; something that counts in addresses, and one string comparison.
 ;
 ; sbfs.asm has routines like these and the assembler cannot use them: it does not include
 ; the filesystem, because it reaches the disk through the system's services instead. That
@@ -44,6 +45,25 @@ numAdd:
   STA.0
   RET
 
+; The two byte number at DP0 becomes itself less the one at DP2.
+numTake:
+  DPUP.0 0d01
+  DPUP.2 0d01
+  LDA.0
+  LDB.2
+  CCF
+  SUB
+  MVQA
+  STA.0
+  DPDN.0 0d01
+  DPDN.2 0d01
+  LDA.0
+  LDB.2
+  SUB                       ; Borrows in from the low half.
+  MVQA
+  STA.0
+  RET
+
 ; Adds the byte in A to the two byte number at DP0.
 numAddByte:
   DPUP.0 0d01
@@ -92,6 +112,25 @@ numCompare:
 numCompareDone:
   RET
 
+; Q is zero if the strings at DP0 and DP1 are the same, both ending in a zero byte.
+;
+; Down here rather than with the label table, where it started, because four separate
+; parts want it: labels, vector names, which file has already been included, and which
+; directive a keyword is.
+sameText:
+  LDA.0
+  LDB.1
+  CCF
+  SUB
+  BNQ sameTextDone
+  LDA.0
+  BRA sameTextDone          ; They ended together, so they matched all the way.
+  INCD.0
+  INCD.1
+  BRI sameText
+sameTextDone:
+  RET
+
 ; The two byte number at DP0 becomes zero.
 numZero:
   RSTA

Programs/CosmOS/Assembler/source.asm

@@ -12,6 +12,24 @@
 ; The file is read TWICE, once per pass, and srcRewind is how the second pass starts over.
 ; Nothing is kept between the passes but the label table.
 ;
+; ---- A stack of readers ----
+;
+; #Include splices another file in where it stands, so the reader is a stack: srcInclude
+; puts the current file's whole state aside, opens the new one, and the end of that file
+; pops the old one back and carries on where it left off. Everything above works on "the
+; current file" and does not know the stack is there.
+;
+; THE WHOLE STATE GOES ASIDE, buffer and all, 293 bytes of it. Keeping only the position
+; and re-reading the block on the way back would be cheaper in memory and would cost a disk
+; read per pop; at six levels of nesting the copy costs less than the arithmetic to avoid it.
+; The buffer pointer survives the trip because it points into the buffer, which is always at
+; the same address - the state is saved from and restored to the same variables.
+;
+; A FILE IS INCLUDED ONCE. Including it twice is not an error, it just does nothing, which
+; is what lets two libraries depend on a third. The names are remembered for the length of
+; one pass and forgotten between them, because the second pass has to walk exactly the same
+; tree the first one did.
+;
 ; Written by Anachronaut
 
 #Program
@@ -22,6 +40,20 @@
 ; memory and every later block read has to name the file again - there being no such thing
 ; as an open file to hold on to.
 srcOpen:
+  SETD.1 SrcTopName
+  CALL srcKeepName          ; Kept apart, so that each pass can open it again.
+  CALL srcRestart
+  RET
+
+; Back to the top of the tree: the first file, no includes taken yet, nothing on the stack.
+; This is what starts each pass.
+srcRestart:
+  RSTA
+  SETD.0 SrcDepth
+  STA.0
+  SETD.0 IncCount
+  STA.0
+  SETD.0 SrcTopName
   SETD.1 SrcName
   CALL srcKeepName
   CALL srcRewind
@@ -118,13 +150,262 @@ srcNextDone:
   RET
 
 srcAtEnd:
+  ; This file is finished. If it was included by another, that one is not: it goes back on
+  ; and the next character comes from where it left off, which is what makes an include
+  ; read as though the text had been written there.
   INIA 0d1
   SETD.0 SrcEnded
   STA.0
+  SETD.0 SrcDepth
+  LDA.0
+  BRA srcNothingLeft
+  CALL srcPop
+  BRI srcNext
+
+srcNothingLeft:
   RSTA
   INIB 0d1
   CCF
-  ADD                       ; Q is not zero: the file is finished.
+  ADD                       ; Q is not zero: there is no more source anywhere.
+  RET
+
+; ---- The stack ----
+
+; Opens the file named at DP0 as though its text were written here. Q is zero if the
+; reader is now inside it, or if it had already been included and there is nothing to do.
+srcInclude:
+  SETD.1 IncWanted
+  CALL srcKeepName
+  CALL srcSeenAlready
+  BRQ srcIncludeSkip
+
+  SETD.0 SrcDepth
+  LDA.0
+  SETD.2 SrcDepthLimit
+  LDB.2
+  CCF
+  SUB
+  BNC srcTooDeep
+
+  CALL srcRemember
+  CALL srcPush
+  SETD.0 IncWanted
+  SETD.1 SrcName
+  CALL srcKeepName
+  CALL srcRewind
+  BNQ srcIncludeGone
+  RET                       ; Q is zero, out of srcRewind.
+
+srcIncludeSkip:
+  RSTA
+  RSTB
+  CCF
+  ADD                       ; Already in, so this line does nothing at all.
+  RET
+
+srcIncludeGone:
+  ; The file is not there. The stack is left as it is: the caller stops the assembly, and
+  ; unwinding for the sake of tidiness would only hide where it happened.
+  RSTA
+  INIB 0d1
+  CCF
+  ADD
+  RET
+
+srcTooDeep:
+  SETD.0 TooDeepText
+  SWI osPrintString
+  RSTA
+  INIB 0d1
+  CCF
+  ADD
+  RET
+
+; Puts the current file aside and makes room for another.
+srcPush:
+  CALL srcSlot
+  SETD.0 SrcState
+  SETD.1 SrcSlot
+  LDD.1.1
+  CALL srcCopyState
+  SETD.0 SrcDepth
+  LDA.0
+  INCA
+  STA.0
+  RET
+
+; And takes it back.
+srcPop:
+  SETD.0 SrcDepth
+  LDA.0
+  DECA
+  STA.0
+  CALL srcSlot
+  SETD.1 SrcSlot
+  LDD.0.1
+  SETD.1 SrcState
+  CALL srcCopyState
+  RET
+
+; Puts one character back, so that the next read produces it again. The character itself
+; is in A, because whether it was a newline decides whether a line goes back too.
+;
+; The tokenizer holds one character of lookahead, and at an #Include that character belongs
+; to the file about to be put aside. Undoing the read is how it stays with that file: when
+; the file is opened again the character is simply still there, and nothing has to be
+; carried across the include or handed back at some moment chosen by the reader.
+;
+; CARRYING IT ACROSS WAS THE OBVIOUS THING AND IT WAS WRONG. A file runs out in the middle
+; of whatever the tokenizer happens to be doing, so handing the character back then injects
+; it into the middle of a word: `start:` came back as `s` and then `tart:`, which assembles
+; into a file that looks entirely reasonable.
+srcStepBack:
+  INIB 0x0A
+  XOR
+  BNQ srcStepBackAt
+  SETD.0 SrcLine
+  SETD.2 SrcOne
+  CALL numTake              ; A newline not yet read has not started a line either.
+
+srcStepBackAt:
+  SETD.0 SrcAt
+  SETD.2 SrcOne
+  CALL numTake
+  SETD.0 SrcPointer
+  SETD.2 SrcOne
+  CALL numTake
+  RET
+
+; Where the slot for the current depth is, into SrcSlot.
+srcSlot:
+  SETD.0 SrcStack
+  SETD.1 SrcSlot
+  STD.0.1                   ; WHERE the stack is, not what is in it.
+  SETD.0 SrcSlotLeft
+  SETD.2 SrcDepth
+  LDA.2
+  STA.0
+srcSlotLoop:
+  SETD.0 SrcSlotLeft
+  LDA.0
+  BRA srcSlotDone
+  DECA
+  STA.0
+  SETD.0 SrcSlot
+  SETD.2 SrcStateBytes
+  CALL numAdd
+  BRI srcSlotLoop
+srcSlotDone:
+  RET
+
+; The whole reader state, from DP0 to DP1.
+srcCopyState:
+  SETD.2 SrcCopyFrom
+  STD.0.2
+  SETD.2 SrcCopyTo
+  STD.1.2
+  SETD.0 SrcCopyLeft
+  SETD.2 SrcStateBytes
+  CALL numSet
+srcCopyLoop:
+  SETD.1 SrcCopyFrom
+  LDD.0.1
+  LDA.0
+  SETD.1 SrcCopyTo
+  LDD.0.1
+  STA.0
+  SETD.0 SrcCopyFrom
+  CALL numStep
+  SETD.0 SrcCopyTo
+  CALL numStep
+  SETD.0 SrcCopyLeft
+  SETD.2 SrcOne
+  CALL numTake
+  SETD.0 SrcCopyLeft
+  LDA.0
+  INCD.0
+  LDB.0
+  OR
+  BNQ srcCopyLoop
+  RET
+
+; ---- Which files have been in ----
+
+; Q is zero if IncWanted has already been included in this pass.
+srcSeenAlready:
+  RSTA
+  SETD.0 IncLeft
+  STA.0
+srcSeenLoop:
+  SETD.0 IncLeft
+  LDA.0
+  SETD.2 IncCount
+  LDB.2
+  CCF
+  SUB
+  BRQ srcSeenNo
+  CALL srcSeenSlot
+  SETD.1 IncSlot
+  LDD.0.1
+  SETD.1 IncWanted
+  CALL sameText
+  BRQ srcSeenYes
+  SETD.0 IncLeft
+  LDA.0
+  INCA
+  STA.0
+  BRI srcSeenLoop
+srcSeenYes:
+  RSTA
+  RSTB
+  CCF
+  ADD
+  RET
+srcSeenNo:
+  RSTA
+  INIB 0d1
+  CCF
+  ADD
+  RET
+
+; Writes IncWanted down as having been included.
+srcRemember:
+  SETD.0 IncLeft
+  SETD.2 IncCount
+  LDA.2
+  STA.0
+  CALL srcSeenSlot
+  SETD.0 IncWanted
+  SETD.1 IncSlot
+  LDD.1.1
+  CALL srcKeepName
+  SETD.0 IncCount
+  LDA.0
+  INCA
+  STA.0
+  RET
+
+; Where name number IncLeft sits, into IncSlot. Fixed fields of 23 bytes: there are few of
+; these and they are short, so an arena would cost more code than it saved.
+srcSeenSlot:
+  SETD.0 IncNames
+  SETD.1 IncSlot
+  STD.0.1
+  SETD.0 IncSlotLeft
+  SETD.2 IncLeft
+  LDA.2
+  STA.0
+srcSeenSlotLoop:
+  SETD.0 IncSlotLeft
+  LDA.0
+  BRA srcSeenSlotDone
+  DECA
+  STA.0
+  INIA 0d23
+  SETD.0 IncSlot
+  CALL numAddByte
+  BRI srcSeenSlotLoop
+srcSeenSlotDone:
   RET
 
 ; Fetches the block SrcIndex names, and steps SrcIndex past it. Q is zero if there was one.
@@ -199,6 +480,9 @@ srcKeepEnd:
 
 #Data
 
+; ---- The current file, as one block so that it can be put aside in one piece ----
+;
+SrcState:
 SrcName:
   #Reserve 0d23
 SrcBlocks:
@@ -215,11 +499,60 @@ SrcPointer:
   0x00 0x00
 SrcEnded:
   0x00
-SrcChar:
-  0x00
-SrcLeft:
-  0x00
 
 ; One block, which is the whole of what a source file costs in memory however big it is.
 SrcBuffer:
   #Reserve 0d256
+
+; 292 bytes: a name of 23, six numbers of two, one single byte, and the buffer. NOTHING MAY
+; BE ADDED IN THE MIDDLE OF THE BLOCK ABOVE without changing this to match.
+SrcStateBytes:
+  0x01 0x24
+SrcDepthLimit:
+  0d6
+SrcOne:
+  0x00 0x01
+
+SrcChar:
+  0x00
+SrcLeft:
+  0x00
+SrcDepth:
+  0x00
+SrcSlot:
+  0x00 0x00
+SrcSlotLeft:
+  0x00
+SrcCopyFrom:
+  0x00 0x00
+SrcCopyTo:
+  0x00 0x00
+SrcCopyLeft:
+  0x00 0x00
+
+; The file the assembly started from, so that each pass can open it again.
+SrcTopName:
+  #Reserve 0d23
+
+IncWanted:
+  #Reserve 0d23
+IncCount:
+  0x00
+IncLeft:
+  0x00
+IncSlot:
+  0x00 0x00
+IncSlotLeft:
+  0x00
+
+TooDeepText:
+"included files are nested deeper than this assembler will follow
+"
+
+; Six levels of nesting, at 293 bytes each. CosmOS itself nests three deep.
+SrcStack:
+  #Reserve 0d1758
+
+; Sixteen names of 23 bytes, which is more separate files than anything here includes.
+IncNames:
+  #Reserve 0d368

Programs/CosmOS/Assembler/token.asm

@@ -25,6 +25,18 @@
 ; the line it STARTED on - captured before the token is read, because a token ending in a
 ; newline has already moved the reader on to the next line by the time it is finished.
 tokNext:
+  SETD.0 TokHeld
+  LDA.0
+  BRA tokFresh
+  RSTA
+  STA.0
+  RSTA
+  RSTB
+  CCF
+  ADD                       ; The one that was handed back, exactly as it was.
+  RET
+
+tokFresh:
   RSTA
   SETD.0 TokString
   STA.0
@@ -131,6 +143,36 @@ tokEnded:
   ADD                       ; Q is not zero: the source is finished.
   RET
 
+; Puts the held character back into the file it came from, leaving nothing in hand.
+;
+; This is what #Include calls before the reader puts the current file aside. The character
+; the tokenizer is holding was read from that file and has not been used, so it goes back
+; into it; there is then nothing to carry across the include and nothing to hand back at a
+; moment that might land in the middle of a word.
+tokUnread:
+  SETD.0 TokPending
+  LDA.0
+  BRA tokUnreadDone
+  CALL srcStepBack
+  RSTA
+  SETD.0 TokPending
+  STA.0
+tokUnreadDone:
+  RET
+
+; Hands the token just read back, so that the next tokNext produces it again.
+;
+; ONE TOKEN, and only where nothing has changed it since. The Vector Segment needs it: a
+; name there may be followed by a number, by a handler, or by the next line's name, and
+; which it is cannot be known without looking. Do NOT use it after anything that alters
+; TokText - a label definition with its colon written over would come back as a use of the
+; name rather than as a definition of it.
+tokBack:
+  INIA 0d1
+  SETD.0 TokHeld
+  STA.0
+  RET
+
 ; The next character, into TokChar. Q is zero if there was one. Takes the held one first.
 tokGet:
   SETD.0 TokPending
@@ -242,6 +284,8 @@ TokChar:
   0x00
 TokPending:
   0x00
+TokHeld:
+  0x00
 
 ; As long as a token may be, and one more for the zero. The other assembler stops at the
 ; same 255, and the limit is worth matching rather than choosing again.

Programs/CosmOS/Assembler/tokenTest.asm

@@ -54,11 +54,9 @@ tokenLoop:
 
   SETD.0 ClsLength
   LDA.0
-  RSTB
+  INCD.0
-  PSHA
+  LDB.0
-  POPB
+  SWI osPrintNumber         ; Sixteen bits: a string of 255 characters is 256 bytes long.
-  RSTA
-  SWI osPrintNumber
 
   SETD.0 OpenMark
   SWI osPrintString

Programs/CosmOS/Assembler/vectors.asm

@@ -0,0 +1,214 @@
+; The names in the Vector Segment, and what numbers they have.
+;
+; A vector name is not a label and the two are kept deliberately apart, so a program may
+; call a routine `announce` and name a vector `announce` without either shadowing the
+; other. They are looked up in different places because they mean different things: a label
+; is an address and a vector is a number.
+;
+; FIXED FIELDS HERE, unlike the label table's arena. There are at most a couple of hundred
+; of these against several hundred labels, and the names are short, so packing them would
+; cost more code than it saved. Twenty four bytes an entry: a name of up to twenty two with
+; its zero, and the number.
+;
+; Numbers come from two places. A pinned one is written down in the source, and that is how
+; anything two separately assembled programs must agree about is fixed - the system's
+; services are all pinned. Everything else is numbered automatically from 64 up, out of a
+; range nothing outside one program can name, so what number it gets cannot matter.
+;
+; Written by Anachronaut
+
+#Program
+
+vecReset:
+  SETD.0 VecCount
+  CALL numZero
+  INIA 0d64
+  SETD.0 VecNextAuto
+  STA.0
+  RET
+
+; Declares the name at DP0 with the number in A. Q is zero if it went in.
+vecDeclare:
+  SETD.2 VecPutNumber
+  STA.2
+  SETD.2 VecSubject
+  STD.0.2
+
+  CALL vecFind
+  BNQ vecDeclareFresh
+  SETD.0 VecTwice
+  CALL clsComplain
+  BRI vecDeclareNo
+
+vecDeclareFresh:
+  SETD.0 VecCount
+  SETD.2 VecLimit
+  CALL numCompare
+  BNC vecDeclareFull
+
+  SETD.0 VecWhich
+  SETD.2 VecCount
+  CALL numSet
+  CALL vecSlotAt
+
+  SETD.1 VecSubject
+  LDD.0.1
+  SETD.1 VecSlot
+  LDD.1.1
+  CALL srcKeepName
+  SETD.1 VecSlot
+  LDD.0.1
+  INIA 0d23
+  SETD.0 VecSlot
+  CALL numAddByte
+  SETD.1 VecSlot
+  LDD.0.1
+  SETD.2 VecPutNumber
+  LDA.2
+  STA.0
+
+  SETD.0 VecCount
+  CALL numStep
+  RSTA
+  RSTB
+  CCF
+  ADD
+  RET
+
+vecDeclareFull:
+  SETD.0 VecFull
+  CALL clsComplain
+vecDeclareNo:
+  RSTA
+  INIB 0d1
+  CCF
+  ADD
+  RET
+
+; The next number nothing has taken, into VecPutNumber. These start at 64, above everything
+; that may be pinned, so a name a program made up for itself can never land on a system
+; service.
+;
+; Into memory rather than into A, because a CALL puts A back as it found it.
+vecTakeAuto:
+  SETD.0 VecNextAuto
+  LDA.0
+  SETD.0 VecPutNumber
+  STA.0
+  SETD.0 VecNextAuto
+  LDA.0
+  INCA
+  STA.0
+  RET
+
+; Looks up the name at DP0. Q is zero if it is there, and then VecNumber is its number.
+vecFind:
+  SETD.2 VecSought
+  STD.0.2
+  SETD.0 VecWhich
+  CALL numZero
+
+vecFindLoop:
+  SETD.0 VecWhich
+  SETD.2 VecCount
+  CALL numCompare
+  BNC vecFindMissing
+
+  CALL vecSlotAt
+  SETD.1 VecSlot
+  LDD.0.1
+  SETD.1 VecSought
+  LDD.1.1
+  CALL sameText
+  BRQ vecFindGot
+
+  SETD.0 VecWhich
+  CALL numStep
+  BRI vecFindLoop
+
+vecFindGot:
+  SETD.1 VecSlot
+  LDD.0.1
+  INIA 0d23
+  SETD.0 VecSlot
+  CALL numAddByte
+  SETD.1 VecSlot
+  LDD.0.1
+  LDA.0
+  SETD.0 VecNumber
+  STA.0
+  RSTA
+  RSTB
+  CCF
+  ADD
+  RET
+
+vecFindMissing:
+  RSTA
+  INIB 0d1
+  CCF
+  ADD
+  RET
+
+; Where entry number VecWhich sits, into VecSlot. Twenty four bytes an entry.
+vecSlotAt:
+  SETD.0 VecNames
+  SETD.1 VecSlot
+  STD.0.1
+  SETD.0 VecSlotLeft
+  SETD.2 VecWhich
+  CALL numSet
+vecSlotLoop:
+  SETD.0 VecSlotLeft
+  LDA.0
+  INCD.0
+  LDB.0
+  OR
+  BRQ vecSlotDone
+  INIA 0d24
+  SETD.0 VecSlot
+  CALL numAddByte
+  SETD.0 VecSlotLeft
+  SETD.2 VecOne
+  CALL numTake
+  BRI vecSlotLoop
+vecSlotDone:
+  RET
+
+#Data
+
+VecCount:
+  0x00 0x00
+VecWhich:
+  0x00 0x00
+VecSlot:
+  0x00 0x00
+VecSlotLeft:
+  0x00 0x00
+VecSought:
+  0x00 0x00
+VecSubject:
+  0x00 0x00
+VecNumber:
+  0x00
+VecPutNumber:
+  0x00
+VecNextAuto:
+  0x00
+VecOne:
+  0x00 0x01
+
+; Sixty four names, which is every number a program may name for itself.
+VecLimit:
+  0x00 0x40
+
+VecTwice:
+"that vector name is declared twice"
+VecFull:
+"too many vector names"
+
+VecName:
+  #Reserve 0d23
+
+VecNames:
+  #Reserve 0d1536

Programs/makefile

@@ -84,17 +84,23 @@ cosmos: $(COSMOS) $(APPS) $(NATIVE_ASM)
 
 # Made from scratch every time, so that what is on it is what is in Apps/ now and not
 # also whatever used to be.
-$(COSMOS_DISK): $(APPS) $(NATIVE_ASM) hello.asm testPrograms/stringKeyword.asm
+$(COSMOS_DISK): $(APPS) $(NATIVE_ASM) hello.asm testPrograms/stringKeyword.asm \
+                CosmOS/Apps/Say.asm CosmOS/Source/services.asm
 	@mkdir -p $(@D)
 	rm -f $@
 	$(DISKTOOL) format $@ 2048 4
 	@for app in $(APPS); do $(DISKTOOL) put $@ $$app; done
 	$(DISKTOOL) put $@ $(NATIVE_ASM)
 	@# SOURCE goes on as well, because an assembler with nothing to assemble is a
-	@# demonstration of nothing. Both of these are single files with no #Include, which
+	@# demonstration of nothing. hello.asm and strings.asm are boot images built from one
-	@# is what the native assembler handles so far.
+	@# file; Say.asm is an application, which needs the include and the service names.
+	@#
+	@# Assembling Say.asm writes Say.sbx over the one the host tool put there, so the
+	@# machine ends up running a program it built itself.
 	$(DISKTOOL) put $@ hello.asm
 	$(DISKTOOL) put $@ testPrograms/stringKeyword.asm strings.asm
+	$(DISKTOOL) put $@ CosmOS/Apps/Say.asm
+	$(DISKTOOL) put $@ CosmOS/Source/services.asm
 
 # The system as well as the disk. Building only the image leaves whatever cosmos.bin was
 # there before, or none at all, and then the disk is booted with a system that does not

README.md

@@ -16,7 +16,7 @@ SplitBit is a custom 8 bit system designed for hobbyist projects and experimenta
 - Loadable Programs: A program that was not booted from carries a header saying where it belongs, and Programs/loader.asm reads one off a disk, puts it there, and runs it.
 - An Operating System: CosmOS boots the machine, mounts a disk, lists what is on it, loads a program and runs it, and takes the machine back when it finishes. It comes with a library of programs to run, including a game and a line editor that writes files a person typed.
 - System Services: A loaded program reaches the console and the disk through numbered software interrupts rather than carrying a copy of the code that drives them. The numbers are written down in one file that both sides include, so neither ever types one. It took the editor from 4941 bytes to 1983 without changing a line of what it does.
-- A Native Assembler: SplitBit assembles SplitBit. Programs/CosmOS/Assembler/ is an assembler written in SplitBit assembly that runs under CosmOS, reads source off a SplitBit disk, and writes a binary back to it with no host involved. Its output has to be byte for byte identical to what the C assembler produces from the same source, which is what Tests/native.sh checks.
+- A Native Assembler: SplitBit assembles SplitBit. Programs/CosmOS/Assembler/ is an assembler written in SplitBit assembly that runs under CosmOS, reads source off a SplitBit disk, and writes a binary back to it with no host involved. It builds boot images and loadable applications, following #Include, #Base, #Reserve and #Align, so a program assembled on the machine can then be loaded and run by it. Its output has to be byte for byte identical to what the C assembler produces from the same source, which is what Tests/native.sh checks.
 - Streaming Reads: A file bigger than the machine's memory is read a block at a time, through services that keep nothing open between calls. CosmOS's own source is 104K against 64K of Data Memory, so this is what a self-hosted assembler will stand on.
 - Storage: A block device with 256 byte blocks and 16 megabytes of them, backed by an image file on the host. It knows blocks and not files, because a filesystem is meant to be software SplitBit runs.
 - Memory Controller: Reads and writes Program Memory, moves blocks between memory banks, reaches memory that devices bring with them, and guards a range against being written by accident. It is how a SplitBit machine loads a program.

SplitBit Assembler Manual.md

@@ -501,8 +501,25 @@ Two passes are enough because **every length is known without resolving anything
 
 One thing is genuinely easier here than on a host. The host assembler searches a list of include directories, because a host has directories; **SBFS is flat**, so an include is a file name and there is nowhere else to look.
 
+### Building Applications:
+
+`#Include` splices another file in where it stands, so the reader is a stack of readers: the current file's whole state goes aside, the new one opens, and the end of it pops the old one back. A file is included **once** — including it twice is not an error, it just does nothing, which is what lets two libraries depend on a third.
+
+`#Base` says where a segment is loaded, and a program that says so gets the SBEX loadable header instead of the SPBT boot one, with a `.sbx` name rather than a `.bin`. `#Reserve` and `#Align` lay down runs of zeroes; how many an `#Align` comes to depends on where the cursor has reached, which is why both passes keep a cursor rather than the second one keeping only a write pointer.
+
+Names in `#Vectors` are read and numbered, pinned where the source pins them, so `SWI osPrintString` resolves. **What a name after `SWI` means is settled by what it follows**, not by anything about the name — the Vector Segment may live in a file included further down and may not have been read yet.
+
+That is everything an application needs:
+
+```
+> load Asm.sbx
+> run Say.asm
+wrote Say.sbx: program 46, data 93, labels 7
+> load Say.sbx
+> run built by the machine itself
+it says: built by the machine itself
+```
+
 ### What It Does Not Do Yet:
 
-`#Include`, `#Base`, `#Align`, `#Reserve` and `#Vectors` are **refused by name** rather than ignored. An assembler that quietly skipped a directive would produce a file that looked right and was the wrong length, which is the worst thing it could do; being told "this assembler does not understand that directive yet" costs nothing and hides nothing.
+A `#Vectors` line that names a **handler** rather than only declaring a name. That needs a Vector Segment in the output file and the version two header that carries it, so a program bringing its own interrupt handlers cannot be built on the machine yet. It is refused by name rather than ignored, as everything unfinished here is: an assembler that quietly skipped a directive would produce a file that looked right and was the wrong length, which is the worst thing it could do.
-
-So what it assembles today is a single file with no includes, which is `Programs/hello.asm` — the oldest program in the repository, and now the first one the machine assembles for itself.

Tests/native.sh

@@ -10,6 +10,10 @@
 # Then it runs what the machine built, because a file that matches and does not work would
 # mean both assemblers were wrong together.
 #
+# It checks a boot image and three loadable programs. The loadable ones are the harder case
+# and the interesting one: they include another file, they are based somewhere other than
+# zero, and every service they call is a name declared in that included file.
+#
 # Written by Anachronaut
 
 set -u
@@ -55,8 +59,21 @@ mkdir -p "$WORK"
 "$TOOL" put "$WORK/native.img" "$ROOT/Programs/hello.asm" hello.asm >/dev/null
 "$TOOL" put "$WORK/native.img" "$WORK/Asm.sbx" Asm.sbx >/dev/null
 
-printf 'load Asm.sbx\nrun hello.asm\nexit\n' \
+# The applications, and the file of service names they all include. These are the reason
-    | "$EMU" --fast --cycles 50000000 -D "$WORK/native.img" "$WORK/cosmos.bin" \
+# the second milestone exists: an assembler that cannot follow an #Include cannot build
+# anything that asks the system for anything.
+APPS="Say greet Files"
+"$TOOL" put "$WORK/native.img" "$ROOT/Programs/CosmOS/Source/services.asm" services.asm >/dev/null
+for app in $APPS; do
+    "$TOOL" put "$WORK/native.img" "$ROOT/Programs/CosmOS/Apps/$app.asm" "$app.asm" >/dev/null
+done
+
+{
+    echo "load Asm.sbx"
+    echo "run hello.asm"
+    for app in $APPS; do echo "run $app.asm"; done
+    echo "exit"
+} | "$EMU" --fast --cycles 600000000 -D "$WORK/native.img" "$WORK/cosmos.bin" \
     > "$WORK/session.txt" 2>&1
 
 # ---- It got as far as writing something ----
@@ -75,9 +92,25 @@ check "byte for byte" cmp -s "$WORK/native.bin" "$WORK/reference.bin"
 check "and it runs" grep -q "^Hello, World!$" "$WORK/ran.txt"
 
 # ---- The report it printed says what it did ----
-REPORT="$(grep -o 'program [0-9]*, data [0-9]*, labels [0-9]*' "$WORK/session.txt" || true)"
+REPORT="$(grep -o 'program [0-9]*, data [0-9]*, labels [0-9]*' "$WORK/session.txt" | head -1 || true)"
 check "it counted right" grep -q "program 17, data 14, labels 2" "$WORK/session.txt"
 
+# ---- And the applications, which need an include, a base and the service names ----
+#
+# A loadable program is the harder case and the one that matters: #Include splices another
+# file in, #Base moves every label to where the program will really live, and every SWI in
+# them names a vector declared in a file the source never mentions by number.
+for app in $APPS; do
+    "$ASM" -I "$ROOT/Programs/CosmOS/Source" -o "$WORK/ref-$app.sbx" \
+        "$ROOT/Programs/CosmOS/Apps/$app.asm" >/dev/null
+    rm -f "$WORK/got-$app.sbx"
+    "$TOOL" get "$WORK/native.img" "$app.sbx" "$WORK/got-$app.sbx" >/dev/null 2>&1
+    if [ -f "$WORK/got-$app.sbx" ]; then
+        REPORT="$(wc -c < "$WORK/got-$app.sbx" | tr -d ' ') bytes"
+    fi
+    check "$app byte for byte" cmp -s "$WORK/got-$app.sbx" "$WORK/ref-$app.sbx"
+done
+
 echo
 if [ "$FAIL" -eq 0 ]; then
     echo "All $PASS native assembler checks passed."