📐roundtrip-wiki-rounds

The groupant beside tools/roundtrip/src/wiki.rs in the Mova repository: what the code computes, what it never does, how to go back from its outputs, its invariants, cost and checks.

Facts

Components

Obernants

The typed relations around the forward model: the behaviour complement (what the code never does), the reverse reconstruction (every input compatible with an output) and specialised views.

Model text

Descriptant

roundtrip wiki-round: one autoencoder learning round on Wikipedia text. It draws a deterministic sample from the tier-1 records of Mova's Wikipedia corpus (a fixed dev sample, a fixed test sample and the k-th fresh train window), measures the exact round trip's residual bits on dev and test with the tables in use, adapts en's order generator on the installed domain plus the window, measures again and writes round.json. It decides nothing: the Eel gates and installs. Its outputs hold Wikipedia text (CC BY-SA 4.0) and stay outside git and releases.

Forward

Pool P = articles of shards 0..S in shard order. h(t) = fnv("ae-dev\0" + t): dev if h mod 20 = 0, test if h mod 20 = 1, train otherwise. Dev and test: their articles ordered by their own keys, the first per_article sentences each, until dev sentences (the last article cut). Train: articles ordered by fnv("ae-train\0" + t); window k starts at article (k·⌈train/per_article⌉) mod |train|, epoch = the quotient, and takes articles cyclically until train sentences. Volumes: one per article, one sentence a line. Measure: pipeline::run exact, unit block, merged in volume order; bits = sum of every residual cell. Candidate: domain = last domain_max lines of (installed domain ++ window), Model::adapt(domain, 1) (linearizer retrained on the model's own base trees plus the domain's own parses; tagger and parser unchanged).

Invariant and cost

The split depends on the title alone: no round's window contains a dev or test article, and dev and test are the same in every round (checked again by the caller). A short pool is an error, never a short sample. Same inputs give byte-identical tables. Cost: reading ≤ train + 2·dev sentences from zstd frames; annotating dev and test twice; one adaptation that parses ≤ domain_max sentences and retrains the linearizer on base + domain; memory one model (≈1.1 GB peak RSS measured).

Composition

main → pool → sample (read_article) → tsv files → Annotator::load (current or base) → measure (dev, test) → next_domain → Model::adapt → Annotator::save → measure → round.json. Called by the Eel's autoencoder round (tools/eel/src/autoencoder.rs), which pins this executable's hash and checks the sample files independently.

Behavior complement

Domain: rounds and pools. Relation: the module never writes outside --out, never installs, never reads the parser model, never trains the tagger or parser; with --lin-shuffle or --lin-reverse it learns from damaged windows (controls), never both. Counterexample: articles shorter than per_article make consecutive windows overlap slightly at their ends; a wrapped pool repeats windows (epoch > 0). Evidence: unit tests below, real hand runs.

Reverse reconstruction

Domain: round.json and the sample files. Relation: from (corpus, shards, sizes, k) the sample is recomputed exactly; from the candidate tables the window is not recoverable (the domain is capped and the linearizer is a many-to-one function of its training trees).

Tests

dev_is_fixed_and_never_in_any_train_window (red when the split admits held-out articles), rounds_take_fresh_windows_until_the_pool_wraps, a_small_pool_is_an_error_not_a_short_sample, domain_keeps_the_newest_lines, sentences_and_volumes_keep_the_text.

This page in Markdown → https://md.groupants.elamur.ai/free/en/mova/tools/roundtrip/src/wiki.rs.groupant/