Storage
A data host lives in one directory on your own server, named for the host, under /srv/datahoster/. It holds two things. First, the definitions you author on the local client, its collections, users and endpoints, synced here wholesale: a duplicate of what is on the local client. Second, a datahoster-data/ directory, where the records themselves are stored. The data directory keeps only data, one directory per collection, each holding its records, its deleted ids, and a directory of columns. Schema and configuration are not repeated here; they live in the synced definitions. Values are written in row order, and row number i refers to the same record in every column, so a record is a position shared across the columns. Where this sits among the server's other directories is the Server page.
/srv/datahoster/
api.example.com/ one directory per data host
datahoster-collections/ synced from the local client
datahoster-users/ synced from the local client
datahoster-endpoints/ synced from the local client
datahoster-data/ the stored records
products/ one directory per collection
datahoster-data-records/ the record metadata
datahoster-data-record-00000000.dat
datahoster-data-deleted/ the deleted ids
datahoster-data-deleted-00000000.dat
datahoster-data-import.dat import change tokens
datahoster-data-columns/ one directory per column
vendor/
datahoster-data-column-text-00000000.dat
datahoster-data-column-text-00000000-00000000.jsonl
price/
datahoster-data-column-float64-00000000.dat
in_stock/
datahoster-data-column-uint8-00000000.datproducts host: its definitions synced from the local client, and its records under datahoster-data.Contents
One directory per host under /srv/datahoster/: the local client's definitions synced in, then the stored data under datahoster-data/.
| Name | Type | Description |
|---|---|---|
api.example.com/ | directory | A data host, one directory per hostname the server serves. |
datahoster-collections/ | directory | A copy of the host's collections, synced wholesale from the local client. |
datahoster-users/ | directory | A copy of the host's users, synced wholesale from the local client. |
datahoster-endpoints/ | directory | A copy of the host's endpoints, synced wholesale from the local client. |
datahoster-data/ | directory | The stored records, one directory per collection. |
products/ | directory | A collection's data, named for the collection. |
datahoster-data-records/ | directory | The record metadata, one file per chunk. |
datahoster-data-record-00000000.dat | file | A metadata chunk: one fixed-width row per version. |
datahoster-data-deleted/ | directory | The ids of deleted records, one file per chunk. |
datahoster-data-deleted-00000000.dat | file | A chunk of deleted records: each an id and the append index at deletion. |
datahoster-data-import.dat | file | The change token per imported record: its id and its source file's time. |
datahoster-data-columns/ | directory | One directory per column. |
price/ | directory | A column, named for the column; holds its chunk files. |
datahoster-data-column-<type>-00000000.dat | file | A fixed-width column chunk: one value per row at a fixed byte width. |
datahoster-data-column-text-00000000-00000000.jsonl | file | A text column's strings, one JSON string per line. |
api.example.com/
One directory per data host, named exactly for the hostname it serves. Everything that host stores, its definitions and their data, lives inside it under /srv/datahoster/. This is what the local client's sync writes and what the API routes a request to by its Host header. The Server page has where it sits among the server's other directories.
datahoster-data/
Where the records themselves are stored, one directory per collection. It holds only data, no schema and no configuration, since those live in the synced collection definitions. Row number i refers to the same record in every column, so a record is a position shared across the columns.
products/
One directory per collection, named for the collection. It holds the datahoster-data-records and datahoster-data-deleted directories, and the datahoster-data-columns directory of columns. Its schema is not stored here; it is the collection definition synced from the local client.
datahoster-data-records/
The metadata every record carries, one fixed-width row per version, in the same row order as the columns. It holds one or more datahoster-data-record-00000000.dat chunks, a new file starting each time the current one fills. Every row carries:
| Value | Holds |
|---|---|
id | The record id, a 128-bit number (int128). A public insert gets a random one the server assigns; an imported record carries one the local client derived from its content, so a re-import updates it in place. |
at | When the row was written, in whole milliseconds since 1970, stored as a float64 like any number column (integer milliseconds are exact well past the year 285,000). |
by | The user who wrote it, a username up to 32 characters, empty for a public write. An imported record carries the built-in datahoster-import. |
datahoster-data-deleted/
Lists deleted records, one fixed-width row each, held in one or more datahoster-data-deleted-00000000.dat chunks. A row is the record's 128-bit id and the append index the collection stood at when it was deleted. A record's rows stay in place in every column; the row here marks it gone as of that index.
Storing the index is what lets an id come back. On load, the current version of an id is its latest record row; it is live unless a delete was recorded at or after that row. So deleting a record and later inserting it again under the same id (as an import does when a removed record returns) revives it: the new row sits past the delete index, so it wins.
datahoster-data-import.dat
The change token for every imported record: fixed-width rows of a 128-bit id and the float64 modification time of the file it came from. It lets the server answer the import pre-check — which records' files have changed since last time — so the local client resends only those. It is not versioned record data but sync bookkeeping, so a token is simply rewritten as its file's time moves; the file is append-only and the last row for an id wins on load. When a record is reconciled away (removed from the import files), it is tombstoned like any delete and its token is dropped with a row whose time is negative, a sentinel a reload reads as "removed". Records collected by a public insert have no token here.
datahoster-data-columns/
One directory per column, each named for the column. A column's directory holds its chunk files at fixed names: a fixed-width .dat chunk, and for a text column a .jsonl strings file beside it. Every column has the same number of rows, so the value for row i is at the same position in each.
price/
A single column, named for the column (price here). It holds the column's chunk files: one or more fixed-width .dat chunks, plus a .jsonl strings file for a text column. The files carry fixed names (the stored type and the chunk number), so the column's own name lives only in this directory.
datahoster-data-column-<type>-00000000.dat
A fixed-width column chunk: one value per row, packed at a fixed width, so row i's value is at byte offset i × width. The <type> in the name is the stored type and the number is the chunk. A column's type maps to a stored type:
| Column type | Stored type |
|---|---|
number | float64 |
integer | int32 |
boolean | uint8 (1 for true, 0 for false) |
int128 | int128 (a 128-bit id, 16 bytes) |
char8 … char1024 | the same name (char8, char16, char32, char64, char128, char256, char512, char1024): its fixed byte count, straight in the .dat |
text | text plus a strings file (see the strings file) |
datahoster-data-column-text-00000000-00000000.jsonl
A text column keeps two kinds of file in its directory. The datahoster-data-column-text-00000000.dat chunk holds one fixed-width entry per row: which strings file the value is in, its byte offset, and its byte length. The strings themselves live in the .jsonl files beside it, one JSON string per line. To read a value, take its entry and read that many bytes from that offset in the named strings file.
A strings file is named datahoster-data-column-text-NNNNNNNN-MMMMMMMM.jsonl: NNNNNNNN is the .dat chunk it belongs to (the same number as the fixed-width chunk beside it), and MMMMMMMM counts the strings files within that chunk, from 00000000 up. When a strings file reaches its byte limit (datahoster-collection-maxTextFileSize, 64,000,000 by default) the next string opens a new file with the next MMMMMMMM, so one .dat chunk can point into several strings files. Each string is written whole, so a strings file can pass the limit by the length of its final string.
The offset a value records is where its bytes actually sit in the strings file. A new value is always appended at the end of the chunk's current strings file, so after a restart the writer resumes from the size of that file on disc, and every offset it records stays true.
Writing
Inserting or updating a record appends one value to the end of every column and one row to the record metadata, so all columns keep the same length and the same order. The current version of a record is its most recent row; the same id can appear in earlier rows, which are its past versions.
Deleting a record adds its id, with the current append index, to the datahoster-data-deleted directory. Its rows stay in place, marked gone as of that index; inserting the same id again later revives it.
Adding a column
Adding a column writes its directory and chunk files and fills them with the column's default value for every existing row, so it lines up with the other columns. Existing columns stay as they are. The column is added to the collection definition on the local client, and the sync carries the change here.
Chunk sizes
A fixed-width column chunk starts a new file after datahoster-collection-recordsPerChunkFile values, 1,000,000 by default; the record-metadata and deleted-id files roll at the same count. A text .jsonl strings file grows until it passes datahoster-collection-maxTextFileSize bytes, 64,000,000 by default, then the next string opens a new file. Each string is written whole, so a strings file can pass the limit by the length of its final string. Both limits are set per collection.