Memory

Objects

The data structures this page describes, from the top down.

Database

A host's data is one Database: a map from each collection name to the Collection that holds it. The server keeps one Database per hosted domain, so a request routed by host lands on exactly one of them.

Database {
  collections: Map<string, Collection>
}

Collection

A Collection is the columns of one table, plus the index that finds the current version of each row. Every field is a column, and so is each part of the record metadata: id, at and by are columns like any other, so every part of a record is stored the same way.

Collection {
  length:        number
  columns:       Map<string, Column>
  current:       Map<int128, number>
  chunkRecords:  number
  bufferRecords: number
}

Reading the collection is a walk over current: for each live id, read its row from every column. An insert or update appends one row to every column and points current at it. A delete drops the id from current; its rows stay in place as history, and its id joins the collection's deleted list on disc with the append index at deletion, so inserting the same id again later revives it.

Column

A Column holds one field's value for every row, in row order, as typed arrays. It is a list of full chunks, one buffer per numbered file on disc, followed by a growing tail for the newest rows.

Column {
  type:   string
  width:  number
  length: number
  chunks: ArrayBuffer[]
  tail:   ArrayBuffer[]
}

Chunks and the tail

The last chunk grows as rows arrive. Preallocating a whole chunk, a million values, for every column would waste memory, and growing one chunk by reallocation would copy it in full on every extension. So the tail is built from small buffers: allocate one buffer of bufferRecords values, fill it, allocate the next, and so on. When the tail holds a full chunk's worth of values, its buffers are copied once into a single chunk buffer, appended to chunks, and written to the next numbered file. The tail then begins again empty for the next chunk.

The result is one layout in two places. Every completed chunk is a single buffer of chunkRecords values, matching one file byte for byte. Only the final, open chunk is split into buffers, and only in memory.

Both are set per collection. The tail fills buffer by buffer until it reaches a full chunk; the last buffer of a chunk may be only partly used, since chunkRecords need not be a whole number of buffers.

Types

A column uses the typed array that matches its stored type, so the bytes in memory are the bytes on disc. Text is the one exception: memory holds the locator, and the strings stay on disc.

The automatic columns are held the same binary way: id as an int128 (two BigUint64 lanes), at as a number (a Float64Array of whole milliseconds), and by as a fixed-width username. Because a text column's bodies are not in memory, a text column can be returned but cannot be filtered, sorted, or used in an endpoint's logic. A char column is fixed-width and in memory, so it can; every in-memory column can.

Reading a value

To read row i of a column, find its chunk and its slot: chunk = i / chunkRecords and slot = i % chunkRecords. A completed chunk is one buffer, indexed at the slot. In the open chunk the slot sits in tail buffer slot / bufferRecords at slot % bufferRecords. Wrap the buffer in the column's typed array and read that index. For a text column the slot holds the locator, and the string is read from its strings file on disc.