Client-Server Synchronisation

Private API

Every call is authenticated with a bearer token — the datahoster-config-apiToken from your local client config, sent as Authorization: Bearer <token>. There is nothing else to the surface: no dashboard, login, session or cookie. Without a valid token — a missing or wrong token, or an unknown route — every reply is an identical plain 404, so a probe cannot tell the API is there at all (see Server). With a valid token the HTTP status line carries the outcome: 200 ok, 400 a bad path or bad host, 409 an append-only violation, 413 a too-large body, and 426 a request on a different API version's path. The routes are versioned under /api/v3/; the local client pins the prefix to its own shared/version.js version, so a server on a different version answers 426 (or the wall) rather than the local client silently following it onto a contract it does not speak.

GET /api/version

Reports the server's wire-contract version. It is deliberately unversioned (no /api/vN prefix), so a local client whose version has drifted can still ask what the server speaks. When a versioned request comes back 426 (or the wall), the local client reads this endpoint to say which side to upgrade — turning an otherwise opaque failure into a clear message. The local client itself pins its path prefix to its own shared/version.js version — see common/net.js.

{ "api": 2 }

GET /api/v3/manifest

Returns every definition file the server currently holds, keyed by its path, each with a sha256 of its bytes. The local client scans your content root the same way and compares hashes, so it uploads only the definitions that differ and deletes the ones that are gone. Definitions are the small JSON files a host is authored from: a collection's datahoster-collection.json, each column file, each user file, and each endpoint file.

{
  "api.example.com/datahoster-collections/messages/datahoster-collection.json": "b1946ac9…",
  "api.example.com/datahoster-collections/messages/datahoster-collection-columns/email.json": "9f86d081…",
  "api.example.com/datahoster-users/you.json": "3a7bd3e2…",
  "api.example.com/datahoster-endpoints/submit/datahoster-endpoint-insert.json": "2c624232…"
}

GET /api/v3/schema

Returns one host's whole definition set — every collection, column, user and endpoint file — keyed by path, with each file's parsed JSON. The local client fetches this for each host before it pushes, so it can confirm its local copy still matches the server. Because the schema is append-only, a collection or column the server holds that is missing locally (deleted, or its folder renamed) or whose definition differs is drift: that host is left untouched and the drift is reported with the exact definition to restore, while every other host still syncs. Users and endpoints are access definitions, not data, and stay freely mutable.

GET /api/v3/schema?host=api.example.com
Authorization: Bearer <token>

{
  "api.example.com/datahoster-collections/messages/datahoster-collection.json": { "datahoster-collection-name": "messages" },
  "api.example.com/datahoster-collections/messages/datahoster-collection-columns/email.json": { "datahoster-column-name": "email", "datahoster-column-type": "char64" }
}

GET /api/v3/def

Returns one definition file's current bytes — the online structure. The local client reads it when a push is refused, so it can print the server's definition beside your local one and show exactly what drifted. A file the server does not hold answers 410, which the local client reads as "new, not a change".

GET /api/v3/def?path=api.example.com/datahoster-collections/messages/datahoster-collection-columns/email.json
Authorization: Bearer <token>

PUT /api/v3/def

Uploads one definition file, creating or replacing it. The server validates it before writing — its schema — and enforces the same append-only rule the local client pre-checks: a new collection is created empty; a new column added to an existing collection is backfilled; but an existing collection or column may never change — an identical re-push is a no-op, and any change to one is refused 409 with the online and local structures in the body (add a new, differently-named collection or column instead). Users and endpoints carry no data and may be replaced freely.

PUT /api/v3/def?path=api.example.com/datahoster-users/you.json
Authorization: Bearer <token>
Content-Type: application/json

{ "datahoster-user-token": "k7Qw2f9xJ4mB6pR8tZ1nD3sV5hL0aYc" }

{ "stored": true }

DELETE /api/v3/def

Removes a definition that exists on the server but is gone locally — a user or endpoint you deleted. Deleting a definition that is already absent is still a success. A collection or column can never be deleted — the schema is append-only, so its delete is refused 409. This is also why a local deletion or a renamed folder can never drop online data: the client sees the collection or column missing, reports the drift, and skips that host rather than removing anything. Dropping stored data is out of scope of the sync.

DELETE /api/v3/def?path=api.example.com/datahoster-endpoints/old-form/datahoster-endpoint-insert.json
Authorization: Bearer <token>

{ "deleted": true }

POST /api/v3/import/check

The change pre-check for imported records. The local client sends a compact binary blob, one 24-byte row per record: the record's 16-byte id followed by the float64 modification time of the file it came from. The server replies with the 16-byte ids whose stored file time it has not seen, so the local client sends the full record only for those. It saves bandwidth without giving up correctness: the server still compares data on the PUT, so a file whose time changed but whose rows did not writes nothing.

The blob is also the collection's whole desired import state, so the server reconciles here: any record it holds a token for (one the import brought in) whose id is absent from the blob is deleted the usual way and its token dropped. Records with no token (a public insert) are never touched. The local client runs this step even under --full, so a removed record is always reconciled away; an X-Import-Pruned response header reports how many were deleted.

POST /api/v3/import/check?host=api.example.com&collection=frameworks
Authorization: Bearer <token>
Content-Type: application/octet-stream

<id(16) + mtime(float64) rows>

PUT /api/v3/import

Upserts one imported record. The body is the record's JSON in its import forms; the server coerces it against the collection's columns (the same shared code the local client validated with), writes it under the id in the query, and attributes it to the built-in datahoster-import user. It appends a new version only when the record differs from the current one, and always records the file's mtime so the next pre-check can skip it. This is an admin route, so it bypasses the endpoint permissions that guard the public insert.

PUT /api/v3/import?host=api.example.com&collection=frameworks&id=9a3c…&mtime=1782864000123
Authorization: Bearer <token>
Content-Type: application/json

{ "name": "React", "language": "JavaScript", "released": 2013, "stars": 229000 }

{ "stored": true, "changed": true }

GET /api/v3/data

Lists every file in a host's columnar data tree (datahoster-data/) with its size, so the local client can mirror it. The store is append-only: an immutable full chunk is never rewritten, and only the current tail chunk grows. Comparing each file's size to its local copy is all the local client needs to know what to fetch. The local client mirrors these files verbatim into <contentRoot>/_data/<host>/datahoster-data/…, byte-identical to the server, and the Data Logs tool reads them back through the storage engine.

The mirror copies bytes, so it cannot itself judge whether they read back cleanly. After each mirror the local client loads every collection through the storage engine and reads it: a value that fails to decode is reported by host and collection, and the Data Logs tool shows the same error in place of that collection's records. A decode failure is surfaced, never rendered as an empty collection.

GET /api/v3/data?host=api.example.com
Authorization: Bearer <token>

{
  "files": [
    { "path": "messages/datahoster-data-records/datahoster-data-record-00000000.dat", "size": 56000 },
    { "path": "messages/datahoster-data-columns/email/datahoster-data-column-text-00000000.dat", "size": 12000 },
    { "path": "messages/datahoster-data-columns/email/datahoster-data-column-text-00000000-00000000.jsonl", "size": 4211 }
  ]
}

GET /api/v3/data/file

Streams the bytes of one data file. A Range header is honoured so the local client appends only the tail grown since last time; an immutable full chunk is fetched once, whole. Because the store is append-only, the local client always knows what it already has: it fetches a file whole when it is missing locally, or from its local size when the server's copy is larger.

GET /api/v3/data/file?host=api.example.com&path=messages/datahoster-data-columns/email/datahoster-data-column-text-00000000-00000000.jsonl
Authorization: Bearer <token>
Range: bytes=4096-

Synchronisation Process

A run is stateless: the local client recomputes everything from your content root and the server's manifest each time, and only ever sends the differences. It validates the whole host locally first, then validates the whole push plan against the online data before touching anything, so a broken or blocked set never reaches the server. It then pushes, deletes, imports and mirrors.

What you can change

The schema is append-only: it only ever grows. This one rule is enforced on both sides — the local client checks each host against the server's online schema before it pushes, and the server answers 409 — so the two can never disagree.

Drift is reported and enforced per domain. If one host's local copy diverges from its online schema — a collection or column missing or changed — that host is left exactly as it is online and the drift is printed with the definition to restore, while every other host still syncs normally.

Validation

Before it sends anything, the local client parses every local definition, so a malformed set never reaches the server. Because a host is only its small definition files, the checks are cheap and run entirely on your machine; the server parses each one again on write, so the two can never disagree. A run that fails validation pushes nothing.

• Only dotted host folders are scanned; loose files, dotfiles and the _data and _tools folders are ignored and never pushed.
• Every definition sits at a legal path inside a host's collection, user or endpoint folder.
• Every definition file parses against its schema: a valid type, a valid name, and a well-formed structure.
• Every collection and column name inside a file matches the folder or file it sits in.

The append-only rule is checked separately, against each host's online schema, in the plan step: a host whose local copy has dropped or changed a collection or column is skipped and its drift reported, while the other hosts push normally.

Errors

On the wire, a failure carries its outcome in the HTTP status line with a short { "error": … } body; without a valid token every reply is instead the same 404 wall, so a probe learns nothing.

Before it sends anything, the local client validates the whole host locally and pre-checks the push plan; a run that cannot proceed stops with one of the errors below and pushes nothing. Each names the definition at fault so you can fix it and run again. Names in the messages are illustrative.

Drift names the domain and the definition to restore, and leaves that host untouched while the others sync:

$ node datahoster-local.js --sync --config config.json
[datahoster] syncing /srv/data -> https://198.51.100.10
[datahoster] push: 2 uploaded, 0 deleted

[datahoster] api.example.com: NOT SYNCED — the schema is append-only and the local copy diverges from the server. Restore the definitions below (or, to change one, add a new differently-named collection/column instead):
  column "email" was changed locally (append-only — add a new one instead). api.example.com/datahoster-collections/messages/datahoster-collection-columns/email.json:
    server: {"datahoster-column-name":"email","datahoster-column-type":"char64"}
    local:  {"datahoster-column-name":"email","datahoster-column-type":"char32"}

[datahoster-local] fatal: 1 domain(s) not synced — schema drift (see above)

A local validation failure stops on the first bad definition, before any request:

$ node datahoster-local.js --sync --config config.json
[datahoster] sync failed: api.example.com/datahoster-collections/messages/datahoster-collection.json: name "message" does not match its folder "messages"