📐treestore-codec

The groupant beside tools/treestore/src/codec.rs in the Mova repository: what the code computes, what it never does, how to go back from its outputs, its invariants, cost and checks.

Facts

Components

Obernants

The typed relations around the forward model: the behaviour complement (what the code never does), the reverse reconstruction (every input compatible with an output) and specialised views.

Model text

Descriptant

The field codecs: an annotated article's five text fields (Markdown, sentence rows, CoNLL-U, frames, pragmatics) as typed columns. Each coded field is decoded back at write time and kept in the envelope as written if any byte differs, so the round trip is lossless by construction. Epoch 2026-11-06: on tier-1 shards 00005/00050/00099 and 1 272 random articles every field of every record was taken by its codec (0 kept raw).

Approximant

Forward, per article, in field order (each field may use the earlier ones as context):

• Markdown → md raw.
• sentence rows → header id; count; prefix literal; per row a flag byte (irregular id/n, section change, item, block literal, text literal), block delta, and the text as (offset from the cursor, length) into the Markdown.
• CoNLL-U → per tree: row delta, flags (text ≠ row text, last token SpaceAfter=No); per whitespace chunk of the text 0 when the predicted split (frozen rule + model exceptions) is right, else the token byte lengths; structure symbols (shape ids or fragments); per token, escaped (tag, rel, head) and the lemma edit-script id (or literal).
• frames → header, count, per row: row delta, frame count, pieces as table id / reference to a lemma or form of the sentence (4 kinds per token) / literal.
• pragmatics → header, count, per row: row delta, the six labels as one tuple id, confidences as hundredths. Measured 2026-11-06 (shard 00050, final model): structure 2.0 MB for 2.82 M tokens (≈5.7 bits/token), lemma 0.49 MB (1.4 bits/token), splits 0.06 MB, text offsets 0.17 MB.

Invariant

For every record r: decode(encode(r)) = r byte for byte. Per field: coded ⇒ dec_field(enc_field(v)) = v and all its streams are used up (checked in enc_field); otherwise the field stays in the envelope. MISC/SpaceAfter, # sent_id, row ids and sentence numbers are derived, not stored. Cost: O(bytes) per field plus O(chunks·chunk length) for the split predictor and O(rows·distance) for the text search in the Markdown (the cursor makes it linear for in-order sentences).

Composition

Uses cols (streams), json (members, escaping), model (tables, shapes, fragments, split exceptions). Called by store::encode_group and Shard::decode. The frames codec depends on the CoNLL-U string (decoded or kept), the tree and pragmatics codecs on the sentence rows, the sentence codec on the Markdown.

Behavior complement

Does not interpret meaning, does not normalise anything, does not code CoNLL-U with multiword ranges, empty nodes, DEPS ≠ _, MISC other than SpaceAfter=No, forms that do not tile the # text by single spaces, or rows whose column counts differ (those fields stay raw). Confidences are lossless (no quantisation).

Reverse reconstruction

decode is the exact inverse on coded fields given the same model; a coded field is recoverable only with the model whose tables, exceptions and fragments made it (the store checks the model id). Frame references are recoverable only from the decoded tree of the same sentence.

Tests

Integration test tests/roundtrip.rs (synthetic records in three structure modes, a raw-kept CoNLL-U field, sentence views); the real-data gate is treestore verify (byte equality of every record with the source).

This page in Markdown → https://md.groupants.elamur.ai/free/en/mova/tools/treestore/src/codec.rs.groupant/