Streaming: read a file bigger than the machine's memory
osFileRead hands over a whole file, which settles anything under 64K and settles nothing above it. CosmOS's own source is above it - the sources together are 104K against 64K of Data Memory - so a machine that is going to assemble itself needs another way to ask. osFileInfo (0d26) says how many blocks a file occupies. osFileBlock (0d27) hands over one of them and says how many of its bytes belong to the file. Between them a program reads a file of any size through a buffer of 256. Blocks rather than bytes from osFileInfo is forced, not chosen: a file on a sixteen megabyte disk is up to twenty four bits long and a pointer holds sixteen. osFileBlock's count answers in DP3 for the same kind of reason - a whole block is 256 bytes, which does not fit in a register, and a count that reported it as zero would make every reader special-case the end. Nothing is kept open. Every call names the file, so there is no handle to leak and nothing left behind by a program that stops halfway. Taken at its word that means searching the directory once per block, so the system remembers where the last file it was asked about lives; every path that can change what a name means calls fileForget, including the shell's own delete and rename, which do not go through the services. Correctness never depends on the cache - a cache thrown away is indistinguishable from one never filled. Measured on a 329 block file: 7% of the run saved when the file is the first directory entry, 11% when it is the sixteenth. These two say WHY when the answer is no, which the others do not. Elsewhere the only useful response to a failure is to give up, so one value suffices. These are asked questions, and running off the end is how a reader learns it has finished, so it gets an answer of its own: 1 no disk, 2 no such file, 3 past the end, 4 the disk refused. Apps/Stream.asm reads an 84,000 byte file through 256 bytes. The check that matters is the second one: a small file read BOTH ways - whole with osFileRead and streamed - with the two checksums compared, so streaming is measured against the path already known to work rather than against a number someone wrote down. The checksum is Fletcher's rather than a sum, because a sum is the same whatever order the bytes arrived in and the order is exactly what streaming has to get right. Both checksums were also confirmed against the same arithmetic run on the host. The rest of the test is the cache: two files read alternately catch a memory that missed the name changing, and a rename catches one that missed the file moving - and that one would otherwise pass, since the blocks are still there holding the same bytes. The test file is generated rather than taken from the repository. The CosmOS sources would be a truer picture and would move the recorded checksum every time a line of CosmOS was edited, putting a real difference in a crowd of meaningless ones - the same trap the cycle counts used to set. cosmosBreak's recorded output moves by two bytes in two pointers: SbfsIndex added two bytes to the filesystem's data and Break prints the system addresses the registers happened to hold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2JrLzFvuFX9fgi1LDRjrW
This commit is contained in:
Anachronaut
co-authored by
Claude Opus 5
Claude Opus 5
parent
3b800a69e8
commit
3d2ab34229
@@ -581,6 +581,8 @@ doDelete:
|
||||
LDA.0
|
||||
BRA deleteWhat
|
||||
|
||||
; What a name means on the disk is about to change, so the remembered file goes.
|
||||
CALL fileForget
|
||||
CALL sbfsDelete
|
||||
BNQ deleteFailed
|
||||
SETD.0 Deleted
|
||||
@@ -620,6 +622,8 @@ doRename:
|
||||
|
||||
SETD.2 RenameFrom
|
||||
LDD.0.2
|
||||
; What a name means on the disk is about to change, so the remembered file goes.
|
||||
CALL fileForget
|
||||
CALL sbfsRename
|
||||
BNQ renameFailed
|
||||
SETD.0 Renamed
|
||||
@@ -944,6 +948,8 @@ handleFileSave:
|
||||
SETD.2 SbfsFileTail
|
||||
STB.2
|
||||
|
||||
; What a name means on the disk is about to change, so the remembered file goes.
|
||||
CALL fileForget
|
||||
CALL sbfsSaveFile
|
||||
MVQA
|
||||
MVSD.2
|
||||
@@ -964,6 +970,8 @@ handleFileDelete:
|
||||
SETD.2 DiskReady
|
||||
LDA.2
|
||||
BRA serviceNoDisk
|
||||
; What a name means on the disk is about to change, so the remembered file goes.
|
||||
CALL fileForget
|
||||
CALL sbfsDelete
|
||||
MVQA
|
||||
MVSD.2
|
||||
@@ -976,6 +984,8 @@ handleFileRename:
|
||||
SETD.2 DiskReady
|
||||
LDA.2
|
||||
BRA serviceNoDisk
|
||||
; What a name means on the disk is about to change, so the remembered file goes.
|
||||
CALL fileForget
|
||||
CALL sbfsRename
|
||||
MVQA
|
||||
MVSD.2
|
||||
@@ -990,6 +1000,225 @@ serviceNoDisk:
|
||||
STA.2
|
||||
RETI
|
||||
|
||||
; ---- Reading a file that will not fit ----
|
||||
;
|
||||
; A file bigger than Data Memory cannot be handed over whole, and CosmOS's own source is
|
||||
; now that file, so these two are how anything reads one: ask how many blocks, then ask for
|
||||
; each block in turn. There is no open and no close. A program that stops halfway through
|
||||
; leaves nothing behind, because there was never anything to leave.
|
||||
|
||||
; Finds a file, or remembers that it already did. DP0 names it. Q is zero if it is there,
|
||||
; and then SbfsFileStart, SbfsFileBlocks and SbfsFileTail describe it exactly the way
|
||||
; sbfsFind leaves them - whether the search happened or not, which is the whole point.
|
||||
fileLookup:
|
||||
SETD.2 FileCacheValid
|
||||
LDA.2
|
||||
BRA fileLookupSearch
|
||||
|
||||
SETD.1 FileCacheName
|
||||
CALL textSame
|
||||
BNQ fileLookupSearch
|
||||
|
||||
; The same file as last time. The description still has to be put back, because anything
|
||||
; that went to the disk in between - a directory listing, a program being loaded - left
|
||||
; its own answer in those three.
|
||||
SETD.0 SbfsFileStart
|
||||
SETD.2 FileCacheStart
|
||||
CALL sbfsSetWord
|
||||
SETD.0 SbfsFileBlocks
|
||||
SETD.2 FileCacheBlocks
|
||||
CALL sbfsSetWord
|
||||
SETD.0 FileCacheTail
|
||||
LDA.0
|
||||
SETD.0 SbfsFileTail
|
||||
STA.0
|
||||
RSTA
|
||||
RSTB
|
||||
CCF
|
||||
ADD ; Q is zero: found.
|
||||
RET
|
||||
|
||||
fileLookupSearch:
|
||||
CALL sbfsFind
|
||||
BNQ fileLookupMissing
|
||||
|
||||
; Remember it. DP0 still names the file: a CALL puts the pointers back, which is the one
|
||||
; place that convention is a convenience rather than an obstacle.
|
||||
SETD.1 FileCacheName
|
||||
CALL sbfsKeepName
|
||||
SETD.0 FileCacheStart
|
||||
SETD.2 SbfsFileStart
|
||||
CALL sbfsSetWord
|
||||
SETD.0 FileCacheBlocks
|
||||
SETD.2 SbfsFileBlocks
|
||||
CALL sbfsSetWord
|
||||
SETD.0 SbfsFileTail
|
||||
LDA.0
|
||||
SETD.0 FileCacheTail
|
||||
STA.0
|
||||
|
||||
; Marked good last, so that a cache half filled is never a cache believed.
|
||||
INIA 0d1
|
||||
SETD.0 FileCacheValid
|
||||
STA.0
|
||||
|
||||
RSTA
|
||||
RSTB
|
||||
CCF
|
||||
ADD ; Q is zero: found.
|
||||
RET
|
||||
|
||||
fileLookupMissing:
|
||||
RET ; Q is not zero, and sbfsFind is what made it so.
|
||||
|
||||
; Throws the remembered file away. Everything that can change what a name means on the disk
|
||||
; calls this before it does: a file saved over may have moved, a deleted one is gone, and a
|
||||
; renamed one answers to something else. A remembered start block that survived any of
|
||||
; those is a pointer at whatever took its place.
|
||||
;
|
||||
; Q is deliberately untouched, so this can be dropped into a handler without disturbing the
|
||||
; answer that handler is in the middle of working out.
|
||||
fileForget:
|
||||
RSTA
|
||||
SETD.0 FileCacheValid
|
||||
STA.0
|
||||
RET
|
||||
|
||||
; DP0 names it. Q is zero if it is there, and DP3 comes back holding how many blocks it
|
||||
; occupies, counting a part one on the end.
|
||||
;
|
||||
; Blocks, not bytes, and that is forced rather than chosen: a file on a sixteen megabyte
|
||||
; disk is up to twenty four bits long, which does not fit in a pointer. Blocks do, and the
|
||||
; bytes in the last one come back from osFileBlock when the reader gets there.
|
||||
handleFileInfo:
|
||||
SETD.2 DiskReady
|
||||
LDA.2
|
||||
BRA fileInfoNoDisk
|
||||
|
||||
CALL fileLookup
|
||||
BNQ fileInfoMissing
|
||||
|
||||
CALL sbfsFileExtent
|
||||
SETD.2 SbfsWantBlocks
|
||||
LDA.2
|
||||
INCD.2
|
||||
LDB.2
|
||||
|
||||
MVSD.2
|
||||
DPUP.2 0d05 ; The saved DP3, high byte first.
|
||||
STA.2
|
||||
INCD.2
|
||||
STB.2
|
||||
|
||||
MVSD.2
|
||||
DPUP.2 0d02
|
||||
RSTA
|
||||
STA.2 ; And the saved Q: it is there.
|
||||
RETI
|
||||
|
||||
fileInfoNoDisk:
|
||||
MVSD.2
|
||||
DPUP.2 0d02
|
||||
INIA 0d1
|
||||
STA.2
|
||||
RETI
|
||||
|
||||
fileInfoMissing:
|
||||
MVSD.2
|
||||
DPUP.2 0d02
|
||||
INIA 0d2
|
||||
STA.2
|
||||
RETI
|
||||
|
||||
; DP0 names it, DP1 says where to put it, and A and B together are which block, counting
|
||||
; from zero. Q is zero if it read, and DP3 comes back holding how many of the block's bytes
|
||||
; belong to the file.
|
||||
handleFileBlock:
|
||||
; Which block, before anything else, because finding out whether there is a disk needs A
|
||||
; and there is nowhere else the number is written down.
|
||||
SETD.2 SbfsIndex
|
||||
STA.2
|
||||
INCD.2
|
||||
STB.2
|
||||
|
||||
SETD.2 DiskReady
|
||||
LDA.2
|
||||
BRA fileBlockNoDisk
|
||||
|
||||
CALL fileLookup
|
||||
BNQ fileBlockMissing
|
||||
|
||||
; Running off the end is how a reader finds out it has finished, so it gets an answer of
|
||||
; its own rather than being told the disk failed.
|
||||
CALL sbfsFileExtent
|
||||
SETD.0 SbfsIndex
|
||||
SETD.2 SbfsWantBlocks
|
||||
CALL sbfsCompareWord
|
||||
BNC fileBlockPastEnd ; The Carry is set only when the index is the smaller.
|
||||
|
||||
CALL sbfsReadOne ; DP1 still says where. A CALL puts the pointers back.
|
||||
BNQ fileBlockFailed
|
||||
|
||||
; How much of it is the file's. Every block but a short last one is a whole 256, and 256
|
||||
; is why this answers in a pointer instead of a register.
|
||||
SETD.2 SbfsFileTail
|
||||
LDA.2
|
||||
BRA fileBlockWhole ; Nothing partial on the end, so they are all whole.
|
||||
SETD.0 SbfsIndex
|
||||
SETD.2 SbfsFileBlocks
|
||||
CALL sbfsCompareWord
|
||||
BNQ fileBlockWhole ; Not the last one.
|
||||
|
||||
SETD.2 SbfsFileTail
|
||||
LDB.2
|
||||
RSTA
|
||||
BRI fileBlockAnswer
|
||||
|
||||
fileBlockWhole:
|
||||
INIA 0x01
|
||||
RSTB
|
||||
|
||||
fileBlockAnswer:
|
||||
MVSD.2
|
||||
DPUP.2 0d05
|
||||
STA.2
|
||||
INCD.2
|
||||
STB.2
|
||||
|
||||
MVSD.2
|
||||
DPUP.2 0d02
|
||||
RSTA
|
||||
STA.2
|
||||
RETI
|
||||
|
||||
fileBlockNoDisk:
|
||||
MVSD.2
|
||||
DPUP.2 0d02
|
||||
INIA 0d1
|
||||
STA.2
|
||||
RETI
|
||||
|
||||
fileBlockMissing:
|
||||
MVSD.2
|
||||
DPUP.2 0d02
|
||||
INIA 0d2
|
||||
STA.2
|
||||
RETI
|
||||
|
||||
fileBlockPastEnd:
|
||||
MVSD.2
|
||||
DPUP.2 0d02
|
||||
INIA 0d3
|
||||
STA.2
|
||||
RETI
|
||||
|
||||
fileBlockFailed:
|
||||
MVSD.2
|
||||
DPUP.2 0d02
|
||||
INIA 0d4
|
||||
STA.2
|
||||
RETI
|
||||
|
||||
; A breakpoint. Shows every register as the interrupted program had them, waits for a key,
|
||||
; and returns as though nothing happened.
|
||||
;
|
||||
@@ -2465,6 +2694,37 @@ RunArgument:
|
||||
PrintNumber:
|
||||
0x00 0x00
|
||||
|
||||
; ---- Where the last file anybody asked about lives ----
|
||||
;
|
||||
; osFileBlock is handed a name every time it is called, because a stateless service has no
|
||||
; handle to leak and nothing left open by a program that stops in the middle. Taken at its
|
||||
; word that means searching the directory once per block, so reading a four hundred block
|
||||
; file walks the directory four hundred times over to be told the same thing.
|
||||
;
|
||||
; So the last answer is kept. A call naming the same file as the one before it skips the
|
||||
; search and puts these back where sbfs keeps them. MEASURED, on the 329 block file the
|
||||
; streaming test reads: 7% of the whole run saved when the file is the first entry in the
|
||||
; directory, 11% when it is the sixteenth. Modest, and worth having for the shape rather
|
||||
; than the size - what it really removes is a cost that grows with how full the disk is,
|
||||
; on the one operation that is repeated once per block. NOTHING IS EVER TRUSTED THAT WAS NOT
|
||||
; PUT HERE BY A SEARCH: this is a copy of an answer, not a second place where the truth
|
||||
; about a file is written, and every path that could make it wrong calls fileForget. That
|
||||
; is what makes it a speed rather than a promise - a cache that has been thrown away is
|
||||
; indistinguishable from one that was never filled.
|
||||
;
|
||||
; The name is twenty three bytes for a name of twenty two, so that a name filling the
|
||||
; field still has a zero after it and can be compared as a string.
|
||||
FileCacheValid:
|
||||
0x00
|
||||
FileCacheName:
|
||||
#Reserve 0d23
|
||||
FileCacheStart:
|
||||
0x00 0x00
|
||||
FileCacheBlocks:
|
||||
0x00 0x00
|
||||
FileCacheTail:
|
||||
0x00
|
||||
|
||||
; ---- The vectors a loaded program brought with it ----
|
||||
;
|
||||
; Six bytes each: where it goes, what goes there, and what was there before. The last two
|
||||
@@ -2633,6 +2893,8 @@ CommandLine:
|
||||
osFileSave handleFileSave
|
||||
osFileDelete handleFileDelete
|
||||
osFileRename handleFileRename
|
||||
osFileInfo handleFileInfo
|
||||
osFileBlock handleFileBlock
|
||||
osPrintNumber handlePrintNumber
|
||||
osBreak handleBreak
|
||||
Device 0x20 diskDone
|
||||
|
||||
Reference in New Issue
Block a user