Search

The reference search box on this site runs on Datahoster. Every keystroke calls a public select endpoint on data.datahoster.org, a data host whose one collection holds every searchable term. The endpoint does the filtering, sorting and limiting; the page just draws the rows. The terms are loaded with import files, one per kind of thing, so the search is data, not code.

data.datahoster.org/
  datahoster-collections/
    search/                          one row per searchable term
      datahoster-collection.json
      datahoster-collection-columns/
        name.json
        sortkey.json
        category.json
        url.json
        description.json
      datahoster-collection-import/   the terms, grouped by kind
        types.json
        files.json
        folders.json
        fields.json
        operators.json
        headers.json
  datahoster-endpoints/
    search/                          full results, parameterised by the query
      datahoster-endpoint-select.json
    suggest/                         up to 10 names for the header dropdown
      datahoster-endpoint-select.json
The whole search host: one collection, its import files, and one public endpoint.

The collection

A search result is four fields: a name (what you match on and see), a category (Type, File, Folder, Field, Operator, Header), a url to open, and a one line description. A fifth column, a lower-case sortkey, exists only to order the results. The column types are chosen for how each is used.

name — char64

The searchable field, and what you see. A char type is held in memory, so the endpoint can contains it. 64 bytes covers the longest key name.

sortkey — char64

The name in lower case. The endpoint sorts on this, so the order is case-insensitive, the way the search has always sorted. It is only sorted on, never returned.

category — char16

A short label, also in memory, so results could be filtered by kind later. Returned with every row.

url — char64

Where a hit links. Short and fixed, returned with the row.

description — text

One line, sometimes long. text is kept on disc and only returned, never matched, which is exactly right here.

{
  "datahoster-column-name": "name",
  "datahoster-column-type": "char64"
}
The name column: a fixed-size string, so it can be searched and sorted.

The import files

Each file in datahoster-collection-import holds the terms for one kind of thing, as an array of records. A record's keys are the collection's columns; its id is the term's name, so the same term keeps the same row across syncs and reordering a file changes nothing. Editing the search is editing these files.

[
  {
    "id": "char64",
    "name": "char64",
    "sortkey": "char64",
    "category": "Type",
    "url": "reference/types/",
    "description": "A fixed string of up to 64 bytes of UTF-8; held in memory."
  },
  {
    "id": "integer",
    "name": "integer",
    "sortkey": "integer",
    "category": "Type",
    "url": "reference/types/",
    "description": "A whole number."
  }
]
Two entries from types.json; the other files have the same shape.
{
  "id": "ETag",
  "name": "ETag",
  "sortkey": "etag",
  "category": "Header",
  "url": "usage/endpoints/select/#change-detection",
  "description": "Identifies a select's result so a caller can check whether it changed."
}
A single-record file, headers.json, written as one object.

The endpoint

One public select answers the search. It takes the query as a parameter term, keeps rows whose name contains it, sorts by sortkey (the lower-case name, so the order is case-insensitive), and limits the result. Because it is public, no token is needed.

{
  "datahoster-endpoint-select-collection": "search",
  "datahoster-endpoint-select-columns": [
    { "datahoster-endpoint-select-column-of": "name" },
    { "datahoster-endpoint-select-column-of": "category" },
    { "datahoster-endpoint-select-column-of": "url" },
    { "datahoster-endpoint-select-column-of": "description" }
  ],
  "datahoster-endpoint-select-parameters": [
    { "datahoster-endpoint-select-parameter-name": "term", "datahoster-endpoint-select-parameter-type": "text" }
  ],
  "datahoster-endpoint-select-filter": [
    ["name", "contains", { "param": "term" }]
  ],
  "datahoster-endpoint-select-sort": [
    ["sortkey", "asc"]
  ],
  "datahoster-endpoint-select-limit": 100,
  "datahoster-endpoint-select-public": true
}
The search endpoint: filter by contains, sort by name, public.

Calling it

The search box GETs the endpoint with the typed query, at <domain>/<endpoint>. The response is a JSON array of rows under the endpoint's column names, already filtered and sorted, which the page renders as the results table. A search that matches nothing returns an empty array.

GET https://data.datahoster.org/search?term=char
Searching for char.
[
  { "name": "char16", "category": "Type", "url": "reference/types/", "description": "A fixed string of up to 16 bytes of UTF-8; held in memory." },
  { "name": "char32", "category": "Type", "url": "reference/types/", "description": "A fixed string of up to 32 bytes of UTF-8; held in memory." },
  { "name": "char64", "category": "Type", "url": "reference/types/", "description": "A fixed string of up to 64 bytes of UTF-8; held in memory." },
  { "name": "char8", "category": "Type", "url": "reference/types/", "description": "A fixed string of up to 8 bytes of UTF-8; held in memory." }
]
The rows the page turns into the results table, in sorted order.

Two more touches make it feel instant. The response carries an ETag, so a browser re-fetching the same query skips the download when nothing changed. And the page keeps the site's built-in index as a fallback: if the endpoint cannot be reached, offline or while browsing local files, the search runs against that copy instead, so it always works.

The header dropdown

The small search box in the header shows a live dropdown as you type. It calls a second, lighter endpoint, suggest, the same select but returning only the name and its url, limited to ten rows, so the list stays short. Each name links straight to its page; pressing Enter, or the See all results link at the foot of the dropdown, opens the full reference search instead.

GET https://data.datahoster.org/suggest?term=char
The dropdown query: up to ten matching names.

Typing is debounced so a flurry of keystrokes makes one request, and only one request runs at a time: a term typed while a request is in flight is held and sent the moment that one returns, so requests never pile up and the dropdown never lags behind what you typed. Its show and hide is pure CSS, driven by the box's focus.

Keeping it current

To add or change a search term you edit an import file and sync. The local client loads only what changed and deletes any term you removed, so the live search always matches the files. It is the same loop a site would use to publish its own search index from its content.