From 54f5cfe8a40bbc49e7c6c4826b4cab0d77dff363 Mon Sep 17 00:00:00 2001 From: Anachronaut Date: Tue, 25 Aug 2026 20:21:43 -0400 Subject: [PATCH] Add Copy and Compare, which stream in both directions at once Copy joins the read and write streaming services: source and destination are both larger than Data Memory while the program owns one block. Compare reads two files through separate blocks and ignores the bytes past a short final block, which belong to neither file. Between them they exercise empty, exact-block, part-block and 84,000 byte files, and the host extracts the copy afterwards so that two native programs agreeing with each other is not the only oracle. Written by ChatGPT, as their headers record, along with the agree.sh section and the manifest entry that drive them. THIS SHOULD HAVE COME FIRST. The commit before it staged whole files rather than the hunks it meant, so its manifest already names these two programs while their source was still untracked - that commit will not build on its own. Left in place rather than rewritten, since the pair is right and only their order is wrong. NOTES.md is their review of the streaming work. The first item in it is fixed by the commit before this one; the rest are still open. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW --- NOTES.md | 106 ++++++++++ Programs/CosmOS/Apps/Compare.asm | 277 +++++++++++++++++++++++++++ Programs/CosmOS/Apps/Copy.asm | 269 ++++++++++++++++++++++++++ Programs/CosmOS/README.md | 2 + Tests/agree.sh | 33 ++++ Tests/expected/cosmosCopyCompare.out | 30 +++ Tests/input/cosmosCopyCompare.in | 14 ++ 7 files changed, 731 insertions(+) create mode 100644 NOTES.md create mode 100644 Programs/CosmOS/Apps/Compare.asm create mode 100644 Programs/CosmOS/Apps/Copy.asm create mode 100644 Tests/expected/cosmosCopyCompare.out create mode 100644 Tests/input/cosmosCopyCompare.in diff --git a/NOTES.md b/NOTES.md new file mode 100644 index 0000000..ec79fd5 --- /dev/null +++ b/NOTES.md @@ -0,0 +1,106 @@ +# Delete me after these things are fixed + +Temporary notes from the SBFS v2 and CosmOS review, in priority order. + +## Streaming writes: reject a final size larger than the reservation + +`osFileStart` reserves an extent and `osFileWrite` correctly refuses a block index outside +that extent. `sbfsStreamDone`, however, appears to accept a final block/tail count larger +than the extent reserved at the start. It writes the larger size into the descriptor and +then subtracts that size from the reservation when updating the free-block count. + +A caller could reserve one block, finish while claiming two, and make the file descriptor +claim a neighboring block that was never allocated and may belong to another file. + +Before deleting the old destination or modifying the temporary entry, compare the final +extent (`whole blocks + a possible tail block`) with the reserved extent. Refuse the commit +if the final extent is larger. Add tests for a final size that is smaller than, equal to, +and larger than the reservation, including tail/no-tail boundaries. + +## Protect the temporary-file namespace + +Whole-file saves and streamed writes use the fixed names `sbfs.part` and `sbfs.out`. +Starting another operation deletes an existing entry with the corresponding name as stale +temporary output, but these are otherwise legal user filenames. A real user file with one +of those names can therefore be deleted. + +Possible resolutions: + +- Reserve these names and refuse ordinary creation under them. +- Use a spare descriptor flag to identify temporary entries, and clean up by identity + rather than by name. +- Move temporary artifacts into a defined `/tmp` policy where appropriate. A stream's + temporary currently needs to be in the destination directory because rename changes a + name but not a parent, so `/tmp` alone may require adding a move operation. + +Whatever rule is chosen should be enforced in both SplitDisk and the native SBFS code and +documented as part of the format or CosmOS policy. + +## Correct stale CosmOS memory-map documentation + +`Programs/CosmOS/README.md` currently gives overlapping Data Memory ranges: + +```text +CosmOS: 0x0000 through 0x3FFF +Application: 0x2000 and above +``` + +The intended 8 KiB system allocation appears to be `0x0000` through `0x1FFF`, with +applications at `0x2000` and above. The minimal application in the same section still uses +the old bases (`0x2000` Program and `0x1000` Data), while current applications use `0x4000` +and `0x2000`. The opening comments in `cosmos.asm` and several monitor examples also retain +old addresses. + +The README says `make test` checks these boundaries, but the overlapping table passed. +Strengthen `Tests/docs.sh` so the documented system endpoints and application bases are +parsed and checked for agreement/non-overlap. + +## Establish one portable path-length limit + +SplitDisk carries paths up to 511 characters, while the native path machinery appears to +use a 255-byte limit. Paths are not stored on disk, so this does not change the format, but +the host can construct a tree addressable by a path that CosmOS cannot express in one +operation. + +Declare a portable CosmOS/SBFS path limit and have SplitDisk enforce it when modifying an +image. The native 255-byte limit is reasonable; the important property is agreement and +documentation. Component names remain limited to 22 bytes independently. + +## Bound the number of directory entries + +The parent field is 16 bits and stores `descriptor index + 1`, with zero reserved for the +root. Descriptor index 65535 therefore cannot be represented as a parent because adding +one wraps to zero. SplitDisk currently accepts directory sizes large enough to exceed the +representable parent domain. + +Define and validate a maximum directory-block/entry count such that every directory entry +can be named as a parent. Apply the check while formatting and mounting/reading malformed +images in both implementations. + +## Document the save/stream crash guarantee precisely + +The temporary-first ordering protects the old file from expected failures such as no +contiguous run or a refused block write. It is not power-loss atomic: a reset between +deleting the old entry and renaming the temporary can lose the destination and leave the +temporary behind. + +That is an appropriate tradeoff for SBFS; it does not call for a journal. Document the +guarantee as protection against normal operation failures rather than full crash +atomicity. A future boot cleanup or small consistency checker could reclaim unmistakable +temporary entries, especially if they receive a descriptor flag. + +## Design strengths worth preserving + +- Parent-as-index-plus-one makes every version one entry a valid root child without + conversion. +- Directories consume one descriptor and no data blocks, leaving the descriptor array as + the complete allocation map. +- Files remain contiguous and the block allocator remains ignorant of hierarchy. +- Path resolution below the service boundary gave existing applications directories + without changing their interfaces. +- The current directory is a two-byte identity rather than a stored string, and the shell + restores it after applications run. +- The independent host/native implementations and byte-identical disk agreement tests are + unusually strong validation of the written format. +- `osFileStart`/`osFileWrite`/`osFileFetch`/`osFileDone` provide the bounded-memory output + abstraction needed by assemblers, compilers, linkers, and future sequential pipelines. diff --git a/Programs/CosmOS/Apps/Compare.asm b/Programs/CosmOS/Apps/Compare.asm new file mode 100644 index 0000000..d4eb647 --- /dev/null +++ b/Programs/CosmOS/Apps/Compare.asm @@ -0,0 +1,277 @@ +; Compare two files without asking either one to fit in Data Memory. +; +; Each occupied block is read into its own 256-byte buffer. The byte count returned for +; the block is compared before its contents, and only those bytes are examined: bytes +; after the end of a short final block belong to neither file and are allowed to differ. +; +; Written by ChatGPT for Anachronaut's SplitBit + +#Include services.asm + +#Program + #Base 0x4000 + +start: + SETD.0 Arguments + INIB 0xFF + SWI osArgument + SETD.0 Arguments + LDA.0 + BRA usage + CALL textSplit + SETD.0 TextRest + LDD.0.0 + LDA.0 + BRA usage + + SETD.0 Arguments + SWI osFileInfo + BNQ firstFailed + SETD.0 FirstBlocks + STD.3.0 + + SETD.0 TextRest + LDD.0.0 + SWI osFileInfo + BNQ secondFailed + SETD.0 SecondBlocks + STD.3.0 + + ; A different number of occupied blocks is immediately a different file. + SETD.0 FirstBlocks + LDA.0 + SETD.1 SecondBlocks + LDB.1 + CCF + SUB + BNQ different + INCD.0 + INCD.1 + LDA.0 + LDB.1 + CCF + SUB + BNQ different + + SETD.0 FirstBlocks + SETD.1 RemainingBlocks + CALL copyWord + RSTA + SETD.0 Index + STA.0 + INCD.0 + STA.0 + +nextBlock: + SETD.0 RemainingBlocks + LDA.0 + INCD.0 + LDB.0 + OR + BRQ alike + + SETD.0 Arguments + SETD.1 FirstBlock + CALL readAtIndex + BNQ firstReadFailed + SETD.0 FirstCount + STD.3.0 + + SETD.0 TextRest + LDD.0.0 + SETD.1 SecondBlock + CALL readAtIndex + BNQ secondReadFailed + SETD.0 SecondCount + STD.3.0 + + ; Equal occupied-block counts do not imply equal tails, so compare the valid byte count + ; returned for this block as well. + SETD.0 FirstCount + LDA.0 + SETD.1 SecondCount + LDB.1 + CCF + SUB + BNQ different + INCD.0 + INCD.1 + LDA.0 + LDB.1 + CCF + SUB + BNQ different + + SETD.0 FirstCount + SETD.1 BytesLeft + CALL copyWord + SETD.0 FirstBlock + SETD.1 SecondBlock + +compareByte: + SETD.2 BytesLeft + LDA.2 + INCD.2 + LDB.2 + OR + BRQ blockSame + LDA.0 + LDB.1 + CCF + SUB + BNQ different + INCD.0 + INCD.1 + SETD.2 BytesLeft + CALL takeByte + BRI compareByte + +blockSame: + CALL stepIndex + CALL takeBlock + BRI nextBlock + +; DP0 is the path and DP1 the destination buffer. Q and DP3 are the service answers. +readAtIndex: + SETD.2 Index + LDA.2 + INCD.2 + LDB.2 + SWI osFileBlock + RET + +; DP0 names the source word and DP1 the destination word. +copyWord: + LDA.0 + STA.1 + INCD.0 + INCD.1 + LDA.0 + STA.1 + RET + +stepIndex: + SETD.0 Index + INCD.0 + LDA.0 + INCA + STA.0 + BNC stepDone + DECD.0 + LDA.0 + INCA + STA.0 +stepDone: + RET + +takeBlock: + SETD.0 RemainingBlocks + INCD.0 + LDA.0 + BRA blockBorrow + DECA + STA.0 + RET +blockBorrow: + INIA 0xFF + STA.0 + DECD.0 + LDA.0 + DECA + STA.0 + RET + +; Decrement the big-endian sixteen-bit BytesLeft. A full block arrives as 0x0100, so an +; eight-bit counter would compare none of it and call many different files equal. +takeByte: + INCD.2 + LDA.2 + BRA byteBorrow + DECA + STA.2 + RET +byteBorrow: + INIA 0xFF + STA.2 + DECD.2 + LDA.2 + DECA + STA.2 + RET + +alike: + SETD.0 SameText + SWI osPrintString + SWI osExit +different: + SETD.0 DifferentText + SWI osPrintString + SWI osExit +usage: + SETD.0 Usage + SWI osPrintString + SWI osExit +firstFailed: + SETD.0 FirstError + SWI osPrintString + SWI osExit +secondFailed: + SETD.0 SecondError + SWI osPrintString + SWI osExit +firstReadFailed: + SETD.0 FirstReadError + SWI osPrintString + SWI osExit +secondReadFailed: + SETD.0 SecondReadError + SWI osPrintString + SWI osExit + +#Data + #Base 0x2000 + +Arguments: + #Reserve 0d256 +FirstBlocks: + 0x00 0x00 +SecondBlocks: + 0x00 0x00 +RemainingBlocks: + 0x00 0x00 +Index: + 0x00 0x00 +FirstCount: + 0x00 0x00 +SecondCount: + 0x00 0x00 +BytesLeft: + 0x00 0x00 +FirstBlock: + #Reserve 0d256 +SecondBlock: + #Reserve 0d256 + +Usage: +"compare: give me two files +" +FirstError: +"compare: cannot find the first file +" +SecondError: +"compare: cannot find the second file +" +FirstReadError: +"compare: cannot read the first file +" +SecondReadError: +"compare: cannot read the second file +" +SameText: +"the same +" +DifferentText: +"different +" + +#Include text.asm diff --git a/Programs/CosmOS/Apps/Copy.asm b/Programs/CosmOS/Apps/Copy.asm new file mode 100644 index 0000000..fe82fad --- /dev/null +++ b/Programs/CosmOS/Apps/Copy.asm @@ -0,0 +1,269 @@ +; Copy one file to another without asking either file to fit in Data Memory. +; +; The two paths are the two words in the argument. Filesystem names can contain spaces, +; but the CosmOS command line has no quoting yet, so this deliberately has the same rule +; as the shell: a space separates things. +; +; osFileInfo reports how many disk blocks the source occupies, while osFileStart wants the +; shape stored in an SBFS entry: whole blocks and a possible tail. Reading the last block +; first recovers that distinction. The ordinary copy then walks forward one block at a +; time, through one 256-byte buffer. +; +; Written by ChatGPT for Anachronaut's SplitBit + +#Include services.asm + +#Program + #Base 0x4000 + +start: + SETD.0 Arguments + INIB 0xFF + SWI osArgument + SETD.0 Arguments + LDA.0 + BRA usage + CALL textSplit + SETD.0 TextRest + LDD.0.0 + LDA.0 + BRA usage + + ; How many occupied blocks the source has. Keep it as both the loop bound and the + ; distinction between an empty file and one whose last block must be inspected. + SETD.0 Arguments + SWI osFileInfo + BNQ sourceFailed + SETD.0 SourceBlocks + STD.3.0 + PSHD.3 + POPB + POPA + OR + BRQ emptySource + + ; Read the final occupied block. DP3 says how many bytes of it belong to the file: + ; 0x0100 for a whole block, or 0x0001 through 0x00FF for a tail. + SETD.0 SourceBlocks + LDD.3.0 + DECD.3 + SETD.0 Index + STD.3.0 + SETD.0 Arguments + SETD.1 Block + SETD.2 Index + LDA.2 + INCD.2 + LDB.2 + SWI osFileBlock + BNQ readFailed + SETD.0 LastCount + STD.3.0 + + ; A full final block means every occupied block is whole. A short final block means + ; there is one fewer whole block and its low-byte count is the tail. + SETD.0 LastCount + LDA.0 + BNA wholeEnding + SETD.0 Index + SETD.1 OutputWhole + CALL copyWord + SETD.0 LastCount + INCD.0 + LDA.0 + SETD.1 OutputTail + STA.1 + BRI sizeKnown + +wholeEnding: + SETD.0 SourceBlocks + SETD.1 OutputWhole + CALL copyWord + RSTA + SETD.0 OutputTail + STA.0 + +sizeKnown: + BRI startOutput + +emptySource: + RSTA + SETD.0 OutputWhole + STA.0 + INCD.0 + STA.0 + SETD.0 OutputTail + STA.0 + +startOutput: + SETD.0 TextRest + LDD.0.0 + SETD.2 OutputWhole + LDD.3.2 + SETD.2 OutputTail + LDA.2 + SWI osFileStart + BNQ startFailed + + ; Empty files have no blocks to transfer, but still need committing so that an existing + ; destination becomes an empty file. + SETD.0 SourceBlocks + SETD.1 RemainingBlocks + CALL copyWord + RSTA + SETD.0 Index + STA.0 + INCD.0 + STA.0 + +copyNext: + SETD.0 RemainingBlocks + LDA.0 + INCD.0 + LDB.0 + OR + BRQ copyDone + + SETD.0 Arguments + SETD.1 Block + SETD.2 Index + LDA.2 + INCD.2 + LDB.2 + SWI osFileBlock + BNQ readFailed + + SETD.1 Block + SETD.2 Index + LDA.2 + INCD.2 + LDB.2 + SWI osFileWrite + BNQ writeFailed + + CALL stepIndex + CALL takeBlock + BRI copyNext + +copyDone: + SETD.2 OutputWhole + LDD.3.2 + SETD.2 OutputTail + LDA.2 + SWI osFileDone + BNQ doneFailed + SETD.0 Copied + SWI osPrintString + SWI osExit + +; DP0 names the source word and DP1 the destination word. +copyWord: + LDA.0 + STA.1 + INCD.0 + INCD.1 + LDA.0 + STA.1 + RET + +; Increment the big-endian sixteen-bit Index. +stepIndex: + SETD.0 Index + INCD.0 + LDA.0 + INCA + STA.0 + BNC stepDone + DECD.0 + LDA.0 + INCA + STA.0 +stepDone: + RET + +; Decrement the big-endian sixteen-bit RemainingBlocks. +takeBlock: + SETD.0 RemainingBlocks + INCD.0 + LDA.0 + BRA takeBorrow + DECA + STA.0 + RET +takeBorrow: + INIA 0xFF + STA.0 + DECD.0 + LDA.0 + DECA + STA.0 + RET + +usage: + SETD.0 Usage + SWI osPrintString + SWI osExit +sourceFailed: + SETD.0 SourceError + SWI osPrintString + SWI osExit +readFailed: + SETD.0 ReadError + SWI osPrintString + SWI osExit +startFailed: + SETD.0 StartError + SWI osPrintString + SWI osExit +writeFailed: + SETD.0 WriteError + SWI osPrintString + SWI osExit +doneFailed: + SETD.0 DoneError + SWI osPrintString + SWI osExit + +#Data + #Base 0x2000 + +Arguments: + #Reserve 0d256 +SourceBlocks: + 0x00 0x00 +RemainingBlocks: + 0x00 0x00 +Index: + 0x00 0x00 +LastCount: + 0x00 0x00 +OutputWhole: + 0x00 0x00 +OutputTail: + 0x00 +Block: + #Reserve 0d256 + +Usage: +"copy: give me a source and destination +" +SourceError: +"copy: cannot find the source +" +ReadError: +"copy: cannot read the source +" +StartError: +"copy: cannot create the destination +" +WriteError: +"copy: cannot write the destination +" +DoneError: +"copy: cannot finish the destination +" +Copied: +"copied +" + +#Include text.asm diff --git a/Programs/CosmOS/README.md b/Programs/CosmOS/README.md index 8724f5c..fac694e 100644 --- a/Programs/CosmOS/README.md +++ b/Programs/CosmOS/README.md @@ -327,6 +327,8 @@ from every assembly file in it. Several are old programs written for the bare ma | Stream | Reads an 84,000 byte file through a buffer of 256, which is what says a file bigger than Data Memory can be read at all. | | Type | Prints a named text file a block at a time, including one too large to fit in Data Memory. | | Pour | Writes a file a block at a time, never holding more than one block of it. Each block is filled with a byte naming itself, so a block written to the wrong place shows up as content rather than as a length. | +| Copy | Copies one path to another a block at a time, including an empty file or one larger than Data Memory. | +| Compare | Compares two files a block at a time, stopping at their real tails rather than comparing unused bytes in the final disk blocks. | | Wander | Goes to the directory it is given and reads a file there by a bare name. The only thing that moves the machine from inside a program, and so the only thing that can check the shell puts the working directory back afterwards. | | More | A forward-only pager. Space advances a screen, Return one line, and q stops. | diff --git a/Tests/agree.sh b/Tests/agree.sh index 18e58d5..eab5126 100755 --- a/Tests/agree.sh +++ b/Tests/agree.sh @@ -142,6 +142,39 @@ else report FAIL "and every block landed" "a block is not where it was written" fi +# ---- A large file copied and compared on the machine ---- +# +# Copy joins the read and write streaming services: the source and destination are both +# larger than Data Memory, while the application owns one block. Compare then reads both +# through separate blocks and must ignore bytes past the tail. The host extracts the copy +# afterwards, so two native programs agreeing with each other is not the only oracle. +"$ASM" -I "$ROOT/Programs/CosmOS/Source" \ + "$ROOT/Programs/CosmOS/Apps/Copy.asm" -o Copy.sbx >/dev/null 2>&1 +"$ASM" -I "$ROOT/Programs/CosmOS/Source" \ + "$ROOT/Programs/CosmOS/Apps/Compare.asm" -o Compare.sbx >/dev/null 2>&1 +"$TOOL" format copied.img 1024 4 >/dev/null +"$TOOL" mkdir copied.img /Apps >/dev/null +"$TOOL" mkdir copied.img /Input >/dev/null +"$TOOL" mkdir copied.img /Output >/dev/null +"$TOOL" put copied.img Copy.sbx /Apps/Copy.sbx >/dev/null +"$TOOL" put copied.img Compare.sbx /Apps/Compare.sbx >/dev/null +awk 'BEGIN { for (i = 0; i < 84000; i++) printf "%c", 65 + (i % 26) }' > toCopy.dat +"$TOOL" put copied.img toCopy.dat /Input/source.dat >/dev/null +printf 'Copy /Input/source.dat /Output/copy.dat\nCompare /Input/source.dat /Output/copy.dat\nexit\n' \ + | "$EMU" cosmos.bin --fast --cycles 1000000000 --disk copied.img > copied.txt 2>&1 +"$TOOL" get copied.img /Output/copy.dat copiedBack.dat >/dev/null 2>&1 + +if cmp -s toCopy.dat copiedBack.dat; then + report ok "large native copy" "84000 bytes, host-identical" +else + report FAIL "large native copy" "the host read back different bytes" +fi +if grep -q '^> the same$' copied.txt; then + report ok "native compare" "the streamed files agree" +else + report FAIL "native compare" "Compare did not call the copied files equal" +fi + # ---- And each can read what the other wrote ---- # # Matching bytes and being readable are not the same claim. A field both of them write diff --git a/Tests/expected/cosmosCopyCompare.out b/Tests/expected/cosmosCopyCompare.out new file mode 100644 index 0000000..b952297 --- /dev/null +++ b/Tests/expected/cosmosCopyCompare.out @@ -0,0 +1,30 @@ +CosmOS +> copied +finished +> the same +finished +> copied +finished +> the same +finished +> copied +finished +> the same +finished +> copied +finished +> the same +finished +> different +finished +> copy: cannot find the source +finished +> compare: cannot find the second file +finished +> copy: give me a source and destination +finished +> compare: give me two files +finished +> halted +Execution halted. +[exit 0] diff --git a/Tests/input/cosmosCopyCompare.in b/Tests/input/cosmosCopyCompare.in new file mode 100644 index 0000000..c12bcac --- /dev/null +++ b/Tests/input/cosmosCopyCompare.in @@ -0,0 +1,14 @@ +Copy /Input/empty.dat /Output/empty.dat +Compare /Input/empty.dat /Output/empty.dat +Copy /Input/exact.dat /Output/exact.dat +Compare /Input/exact.dat /Output/exact.dat +Copy /Input/tail.dat /Output/tail.dat +Compare /Input/tail.dat /Output/tail.dat +Copy /Input/large.dat /Output/large.dat +Compare /Input/large.dat /Output/large.dat +Compare /Input/large.dat /Input/different.dat +Copy /Input/missing.dat /Output/missing.dat +Compare /Input/large.dat /Input/missing.dat +Copy +Compare /Input/large.dat +exit