char/avril

(wip) why'd you have to go and make things so complicaaaaated?

git clone https://git.t4t.associates/char/avril

Charlotte Somadd `avril:kv` module78f4748

main
7.3 KiB168 linesraw

avril:kv

avril:kv is an embedded persistent key-value store: named keyspaces of ordered byte-string keys mapping to byte-string values, with cross-keyspace atomic batches guarded by per-entry versionstamps. Think "a SQLite-shaped hole for durable program state" — user profiles with a secondary email index, a build cache, a job queue's bookkeeping — without leaving the avril:std world.

The canonical API surface is the generated avril:kv declarations (from exts/kv/src/public.ts); what follows describes the model, not the signatures.

import { openKv } from "avril:kv";

const db = await openKv("users.kv", { access: "exclusive", create: true });
const users = db.keyspace("users");
await users.set("u1", JSON.stringify({ name: "Ada" }));
console.log(await users.getText("u1"));
await db.close();

Storage is a redb file. The on-disk format is an internal detail, not a public contract: the engine and its encoding may change between releases, so don't open the file with other tools.

Opening

openKv(path | file URL, { access, create }) resolves path against the working directory (a file: URL works too, via its href). create defaults to true unless access is "read-only".

  • shared (default) opens and closes the file per operation. Cheap to hold and safe to share, but other processes may interleave between your operations — use atomic batches when that matters.
  • exclusive holds one read-write handle. Opening fails promptly if another process holds the file, so this is the right choice for servers, daemons, and hot paths that want stable, uncontended access.
  • read-only never creates or modifies the file. Opening a missing file succeeds and simply reads empty.

Authority is capability-checked at open and re-checked live on every operation, like file handles: narrowing capabilities with pledge() (even pledge({ read: false }) after opening) makes subsequent operations on already-open handles fail. close() is synchronous and idempotent; await using db = await openKv(...) or an explicit db[Symbol.asyncDispose]() awaits the actual close.

Keyspaces

A database holds any number of named keyspaces — partitions that behave like one table each. db.keyspace(name) returns a lightweight handle (identity-stable per database: the same name gives the same object); db.keyspaces() lists names sorted; clear() empties one keyspace while leaving the others alone, and a cleared keyspace keeps existing as an empty space. Closing the database invalidates its keyspace handles.

Keys

A key is an arbitrary byte string: a string is its UTF-8 encoding, a Uint8Array is copied as-is. "abc" and new Uint8Array([97, 98, 99]) name the very same key. The empty string and the empty byte array are valid keys and sort before every other key.

Keys order lexicographically by byte. Strings therefore sort by their UTF-8 encoding, and byte keys interleave with string keys purely by byte content — there is no cross-type ordering because there are no types. A shorter key sorts before any key extending it. list({ prefix }) scans exactly the keys starting with those bytes, so prefix: "user:" scans the whole "user:1", "user:2", … family while excluding "admin:1". An empty prefix (the default) scans everything.

Limits, enforced with TypeError: strings must be well-formed (lone surrogates rejected, not replaced) for both keys and values.

Values

A value is an arbitrary byte string, exactly as stored: get and list return the raw bytes, byte-for-byte, with no decoding and no envelope. set with a string stores its UTF-8 encoding; set with a Uint8Array stores a copy of its bytes. getText decodes the stored bytes as UTF-8 (and throws a TypeError when they are not valid UTF-8).

Richer values are userland's job: serialize to bytes first — JSON.stringify into a string, or a binary codec such as jsr:@char/cbor into a Uint8Array — and store the bytes. What comes back is the same bytes.

Limits: values are capped at 4 MiB (RangeError). Anything that is neither a string nor a Uint8Array — numbers, objects, functions, symbols, undefined, null — is rejected with a TypeError.

Operations and versionstamps

get returns an entry { keyspace, key, value, versionstamp } or null; getText is the same entry with the value decoded as UTF-8; getMany answers in input order; set resolves with the new { versionstamp }; delete and clear resolve when committed (deleting a missing key changes nothing, not even the versionstamp). Entry keys are returned as Uint8Arrays. list takes prefix, start, end, limit (default and cap 10000), reverse, and yields one snapshot per iteration — breaking out early just stops yielding from the already-materialized rows.

A versionstamp is a bigint: the database's commit sequence number for the commit that last wrote the entry. Equality is the intended use; ordering is also meaningful, but only between stamps of the same database. Every write mints a new one, and all writes in one atomic commit share theirs.

Atomic batches

db.atomic((tx) => { ... }) runs check/set/delete across any of the database's keyspaces in one transaction:

  • tx.check(space, key, stamp | null) asserts the entry's current stamp — or, with null, that the key must not exist.
  • On any failed check the whole batch aborts: { ok: false }, and nothing is written — not even keyspace registrations.
  • Success resolves { ok: true, versionstamp } (shared by every write; null when the batch was empty or check-only, which burns no stamp).
  • Repeated writes to one key resolve last-write-wins.
  • Batches are capped at 1000 ops / 4 MiB total (keyspace names, keys and values). The tx call that would breach either cap throws RangeError.

The callback must be synchronous: returning a promise, throwing, or reusing the transaction afterwards all fail (TypeError on the async and retained cases; a throw discards the batch and propagates). Each call validates its arguments immediately; the ops are collected during the synchronous callback and committed in one operation.

The canonical use is a secondary index maintained atomically with its primary data, with a conflict-retry loop on the versionstamp:

import { openKv } from "avril:kv";

const db = await openKv("users.kv", { access: "exclusive", create: true });
const users = db.keyspace("users");
const byEmail = db.keyspace("usersByEmail");

async function createUser(id: string, email: string, name: string): Promise<void> {
  const value = JSON.stringify({ id, email, name });
  for (;;) {
    if (await byEmail.get(email) !== null) throw new Error(`email taken: ${email}`);
    const result = await db.atomic((tx) => {
      tx.check(byEmail, email, null); // still absent
      tx.set(users, "user:" + id, value);
      tx.set(byEmail, email, id);
    });
    if (result.ok) return;
    // Lost a race: someone claimed the address first. Loop back and re-check.
  }
}

await createUser("u1", "a@example.com", "Ada");

for await (const entry of users.list({ prefix: "user:" })) {
  console.log(entry.key, entry.value);
}

await db.close();

Non-goals for v1

TTL/expiry, watch/change notification, queues, cross-keyspace listing, backup/export, and a synchronous API. list is per-keyspace; there is no way to scan the whole database in one call.

1# `avril:kv`
2
3`avril:kv` is an embedded persistent key-value store: named keyspaces of
4ordered byte-string keys mapping to byte-string values, with cross-keyspace
5atomic batches guarded by per-entry versionstamps. Think "a SQLite-shaped
6hole for durable program state" — user profiles with a secondary email
7index, a build cache, a job queue's bookkeeping — without leaving the
8`avril:std` world.
9
10The canonical API surface is the generated `avril:kv` declarations (from
11`exts/kv/src/public.ts`); what follows describes the model, not the
12signatures.
13
14```ts
15import { openKv } from "avril:kv";
16
17const db = await openKv("users.kv", { access: "exclusive", create: true });
18const users = db.keyspace("users");
19await users.set("u1", JSON.stringify({ name: "Ada" }));
20console.log(await users.getText("u1"));
21await db.close();
22```
23
24Storage is a redb file. The on-disk format is an internal detail, not a
25public contract: the engine and its encoding may change between releases,
26so don't open the file with other tools.
27
28## Opening
29
30`openKv(path | file URL, { access, create })` resolves `path` against the
31working directory (a `file:` URL works too, via its `href`). `create`
32defaults to true unless `access` is `"read-only"`.
33
34- `shared` (default) opens and closes the file per operation. Cheap to
35  hold and safe to share, but other processes may interleave between your
36  operations — use atomic batches when that matters.
37- `exclusive` holds one read-write handle. Opening fails promptly if
38  another process holds the file, so this is the right choice for
39  servers, daemons, and hot paths that want stable, uncontended access.
40- `read-only` never creates or modifies the file. Opening a missing file
41  succeeds and simply reads empty.
42
43Authority is capability-checked at open **and** re-checked live on every
44operation, like file handles: narrowing capabilities with `pledge()` (even
45`pledge({ read: false })` after opening) makes subsequent operations on
46already-open handles fail. `close()` is synchronous and idempotent;
47`await using db = await openKv(...)` or an explicit
48`db[Symbol.asyncDispose]()` awaits the actual close.
49
50## Keyspaces
51
52A database holds any number of named keyspaces — partitions that behave
53like one table each. `db.keyspace(name)` returns a lightweight handle
54(identity-stable per database: the same name gives the same object);
55`db.keyspaces()` lists names sorted; `clear()` empties one keyspace while
56leaving the others alone, and a cleared keyspace keeps existing as an
57empty space. Closing the database invalidates its keyspace handles.
58
59## Keys
60
61A key is an arbitrary byte string: a `string` is its UTF-8 encoding, a
62`Uint8Array` is copied as-is. `"abc"` and `new Uint8Array([97, 98, 99])`
63name the very same key. The empty string and the empty byte array are
64valid keys and sort before every other key.
65
66Keys order lexicographically by byte. Strings therefore sort by their
67UTF-8 encoding, and byte keys interleave with string keys purely by byte
68content — there is no cross-type ordering because there are no types. A
69shorter key sorts before any key extending it. `list({ prefix })` scans
70exactly the keys starting with those bytes, so `prefix: "user:"` scans the
71whole `"user:1", "user:2", …` family while excluding `"admin:1"`. An empty
72prefix (the default) scans everything.
73
74Limits, enforced with `TypeError`: strings must be well-formed (lone
75surrogates rejected, not replaced) for both keys and values.
76
77## Values
78
79A value is an arbitrary byte string, exactly as stored: `get` and `list`
80return the raw bytes, byte-for-byte, with no decoding and no envelope.
81`set` with a string stores its UTF-8 encoding; `set` with a `Uint8Array`
82stores a copy of its bytes. `getText` decodes the stored bytes as UTF-8
83(and throws a `TypeError` when they are not valid UTF-8).
84
85Richer values are userland's job: serialize to bytes first — `JSON.stringify`
86into a string, or a binary codec such as `jsr:@char/cbor` into a
87`Uint8Array` — and store the bytes. What comes back is the same bytes.
88
89Limits: values are capped at 4 MiB (`RangeError`). Anything that is
90neither a string nor a `Uint8Array` — numbers, objects, functions, symbols,
91`undefined`, `null` — is rejected with a `TypeError`.
92
93## Operations and versionstamps
94
95`get` returns an entry `{ keyspace, key, value, versionstamp }` or `null`;
96`getText` is the same entry with the value decoded as UTF-8;
97`getMany` answers in input order; `set` resolves with the new
98`{ versionstamp }`; `delete` and `clear` resolve when committed (deleting
99a missing key changes nothing, not even the versionstamp). Entry keys are
100returned as `Uint8Array`s. `list` takes `prefix`, `start`, `end`,
101`limit` (default and cap 10000), `reverse`, and yields one snapshot per
102iteration — breaking out early just stops yielding from the
103already-materialized rows.
104
105A versionstamp is a `bigint`: the database's commit sequence number for
106the commit that last wrote the entry. Equality is the intended use; ordering
107is also meaningful, but only between stamps of the same database. Every
108write mints a new one, and all writes in one atomic commit share theirs.
109
110## Atomic batches
111
112`db.atomic((tx) => { ... })` runs `check`/`set`/`delete` across any of
113the database's keyspaces in **one** transaction:
114
115- `tx.check(space, key, stamp | null)` asserts the entry's current stamp —
116  or, with `null`, that the key must not exist.
117- On any failed check the whole batch aborts: `{ ok: false }`, and nothing
118  is written — not even keyspace registrations.
119- Success resolves `{ ok: true, versionstamp }` (shared by every write;
120  `null` when the batch was empty or check-only, which burns no stamp).
121- Repeated writes to one key resolve last-write-wins.
122- Batches are capped at 1000 ops / 4 MiB total (keyspace names, keys and
123  values). The `tx` call that would breach either cap throws `RangeError`.
124
125The callback must be synchronous: returning a promise, throwing, or
126reusing the transaction afterwards all fail (`TypeError` on the async and
127retained cases; a throw discards the batch and propagates). Each call
128validates its arguments immediately; the ops are collected during the
129synchronous callback and committed in one operation.
130
131The canonical use is a secondary index maintained atomically with its
132primary data, with a conflict-retry loop on the versionstamp:
133
134```ts
135import { openKv } from "avril:kv";
136
137const db = await openKv("users.kv", { access: "exclusive", create: true });
138const users = db.keyspace("users");
139const byEmail = db.keyspace("usersByEmail");
140
141async function createUser(id: string, email: string, name: string): Promise<void> {
142  const value = JSON.stringify({ id, email, name });
143  for (;;) {
144    if (await byEmail.get(email) !== null) throw new Error(`email taken: ${email}`);
145    const result = await db.atomic((tx) => {
146      tx.check(byEmail, email, null); // still absent
147      tx.set(users, "user:" + id, value);
148      tx.set(byEmail, email, id);
149    });
150    if (result.ok) return;
151    // Lost a race: someone claimed the address first. Loop back and re-check.
152  }
153}
154
155await createUser("u1", "a@example.com", "Ada");
156
157for await (const entry of users.list({ prefix: "user:" })) {
158  console.log(entry.key, entry.value);
159}
160
161await db.close();
162```
163
164## Non-goals for v1
165
166TTL/expiry, watch/change notification, queues, cross-keyspace listing,
167backup/export, and a synchronous API. `list` is per-keyspace; there is no
168way to scan the whole database in one call.