Import

Import records are the data you ship with a collection. They sit in a datahoster-collection-import folder beside the collection's columns, one or more JSON files, and the local client loads them into the collection every time it syncs. It inserts records that are new, updates ones whose data changed, and deletes ones you removed, so your files are the source of truth for the records they own. This is how a website populates a collection it will read back, a search index being the common case.

<datahoster-config-contentRoot>/
  api.example.com/
    datahoster-collections/
      frameworks/
        datahoster-collection.json
        datahoster-collection-columns/
        datahoster-collection-import/   records to load in
          javascript.json              an array: several records
          flutter.json                 an object: one record
          swiftui.json                 an object: one record

A collection's import files, loaded on every sync.

Imported records are written by a built-in user, datahoster-import, so they carry it in their by field and are always told apart from records an insert endpoint collected. No user you define may take a name starting with datahoster, so that attribution cannot be forged.

Every file in the datahoster-collection-import folder holds records for this collection. A file's name is yours to choose; group records however suits you.

Name	Type	Description
`javascript.json`	file	An array of records. Each object in the array is one record.
`flutter.json`	file	A single record, written as one object rather than a one-element array.

Record format

A file is read as one record when its top level is an object, and as many when its top level is an array. Each record is an object whose keys are column names and whose values are the values for those columns. A column with a default may be left out and takes its default; a column with no default must be present. A key that is not a column is an error.

[
  {
    "id": "react",
    "name": "React",
    "language": "JavaScript",
    "released": 2013,
    "stars": 228000
  },
  {
    "id": "vue",
    "name": "Vue",
    "language": "JavaScript",
    "released": "hex:7de",
    "stars": 208000
  }
]

An array file: two records for the frameworks collection.

{
  "id": "flutter",
  "name": "Flutter",
  "language": "Dart",
  "released": 2018,
  "stars": 170000
}

An object file: one record on its own.

Two keys are special. id is optional and sets the record's identity. The automatic at and by fields may not appear: at is always generated, and by is always datahoster-import.

Value forms

A value's plain form is its natural JSON: a string for text and char columns, a number for numbers, true or false for a boolean. The numeric types also accept a string, so you can give an exact value a JSON number cannot hold or write the raw bytes directly.

Type	Accepts
`text`	A string, stored as it is.
`char8` … `char1024`	A string, up to that many bytes of UTF-8.
`boolean`	`true` or `false`. The strings `"true"`, `"yes"`, `"1"` and their opposites are accepted too.
`integer`	A JSON number, or a string: a decimal such as `"2016"`, or `"hex:7de"`, `"binary:11111100000"`, `"base64:B+I="`. A value past JavaScript's safe range must be a string.
`int128`	The same forms as `integer`, for a 128-bit unsigned value (0 to 2¹²⁸ minus 1). Large values go as a decimal or `hex:` string.
`number`	A JSON number, a numeric string such as `"3.14"`, or the raw 8 bytes of the double as `"hex:"` (16 hex digits), `"binary:"` (64 bits) or `"base64:"` (8 bytes), little-endian.

For an integer, hex:, binary: and base64: describe the value: hex: and binary: are its digits in base 16 and base 2, and base64: is its bytes big-endian. For a number, those same prefixes carry the exact 8 bytes of the IEEE 754 double instead, so a value round-trips without a decimal rewrite.

Record identity

Every record needs a stable id so a later sync updates it in place rather than inserting a copy. By default the id is a 128-bit hash of the file name and the record's position in the file. That is enough for a list you keep in order, but inserting a record in the middle shifts the positions after it, so those records are seen as changed.

To pin an identity, give a record an id: any local value, a slug like "react", a path, or a number. Its server id becomes a hash of that value, so it stays fixed however you edit or reorder the files. A local id doubles as a foreign key: another collection referring to this record uses the same local value, and both resolve to the same 128-bit id on the server.

How records load

Import is a phase of --sync, run after the collection definitions are pushed and confirmed online, so the records are always checked against the schema they will land in. If any file has a bad record the phase stops before sending anything, naming the file and the field, so a partial import never happens.

To avoid resending records that have not changed, the local client first tells the server each record's id and the modification time of the file it came from; the server replies with just the ids whose file time it has not seen, and the local client sends only those. The server then compares each record it receives with the stored one and appends a new version only when the data truly differs, so touching a file never churns records whose values are the same. Pass --full to skip the check and send every record, for a first load or a forced refresh.

That same step is where a removed record is deleted. The set of ids the local client sends is the whole desired state for the records this collection's import files own, so the server marks as deleted, in the usual way, any imported record whose id is no longer among them. Only records the import brought in are affected; a record an insert endpoint collected is never touched, so a collection can safely mix the two. This runs for every collection, so removing a record from a file deletes it, and removing every import file deletes all the records the import owned. A record you bring back later, under the same id, simply reappears.

Errors

Each message names the file and, in brackets, the record's index in that file.