Elamur Groupantslangseg-engine-tokenize
The groupant beside tools/langseg/src/lib.rs in the Mova repository: what the code computes, what it never does, how to go back from its outputs, its invariants, cost and checks.
Facts
Implementation material. A Mova code model describes this code at one source revision: partial models, finite checks and their stated scope. It is not a reviewed general exposition, and its labels are shown as written, never raised.
Components
| component | label | where |
|---|---|---|
| approximant | partial | approximant |
| ast | partial | ../syntax.json |
| cost | symbolic source derived | invariant |
| descriptant | present | descriptant |
| invariant | partial | invariant |
| tests | executed bounded checks | tests |
Obernants
The typed relations around the forward model: the behaviour complement (what the code never does), the reverse reconstruction (every input compatible with an output) and specialised views.
| id | type | label | where |
|---|---|---|---|
behavior_complement | behavior_complement | partial source derived model | behavior_complement |
reverse_reconstruction | reverse_reconstruction | partial source derived model | reverse_reconstruction |
utf8 | utf8 | partial source derived model | utf8 |
italic | italic | partial source derived model | italic |
apostrophe | apostrophe | partial source derived model | apostrophe |
suffix_cost | suffix_cost | partial source derived model | suffix_cost |
Model text
descriptant
Descriptant
Prospective epoch
2026-10-31 source review predicted: Scan char_indices; markers toggle italic ids, short tags skipped, script starts at strong char then consumes matching continuations. Resource model: O(B²) worst-case '<'^n suffix find scans; ordinary bounded tags O(B); O(B) chars/lower buffers. External prospective.md predates test additions and preserves APIs, effect order, hazards and interaction hypotheses.
Observed epoch
Bounded real helper checks in tests.md constrain these models; syntax is a partial lexical projection with no macro expansion, name resolution or proof. Production expressions unchanged; five inline test modules added. Compiled mutants are temporary and exactly restored. Existing malformed-input and nonfinite domains remain explicit, not silently repaired. Parent composes this scope with siblings without adding overlapping costs twice.
Exact byte scope; finite correspondence only. Untested branches source-derived.
approximant
Forward relation
Scan char_indices; markers toggle italic ids, short tags skipped, script starts at strong char then consumes matching continuations.
Domain: Original UTF-8 offsets, omitted neutral prefix; italics heuristic not nested grammar.
invariant
Invariant and cost
Invariant
Original UTF-8 offsets, omitted neutral prefix; italics heuristic not nested grammar.
Cost
O(B²) worst-case '<'^n suffix find scans; ordinary bounded tags O(B); O(B) chars/lower buffers.
Limits
Exact byte scope; finite correspondence only. Untested branches source-derived. No percentage surrogate is implemented: categorical parsing/effect observations lack an admitted numerical error metric. Any future profile must define its metric/domain and use only1/2/5/10 percent caps.
models
Typed source-bound relations
Subject: exact source SHA-256 in manifest. Domain and resource premises are in invariant.md. Each model is partial, source-derived, over a single helper invocation or the described finite trace. Complements are proposed inner exclusions under those premises; reverse relations retain all compatible antecedents, with no unique inverse claim. The oracle is the named checks in tests.md; unexecuted cases remain hypotheses. No numerical approximation certificate.
behavior_complement
Neutral-only input cannot start token; multiplication/division excluded from Latin.
reverse_reconstruction
Spans admit arbitrary skipped gaps/markup consistent with byte positions.
utf8
Starts char_indices; ends next char byte or text.len; ascii lowercase preserves offsets.
italic
Repeated same marker toggles once; entry increments span id; HTML opens separately.
apostrophe
Hebrew inner doublequote needs next Hebrew; apostrophe accepted only when next non-Hebrew.
suffix_cost
k<12 bounds skip only after full suffix.find('>'); unmatched openers cause quadratic sum.
tests
Checks and evidence
Named source/helper checks: script_runs_and_transliteration, wave31::empty_ties_and_switch_objective.
Wave31 on2026-10-31:30 distinct exact tests passed twice debug and twice release, with equal sorted named outcomes. This node claims only its listed subset; empty subset means source review only. Six actual source-behavior mutants compiled, failed assertions, restored exact bytes/hash, and passed focused green reruns. No assertion-only mutants; existing swapped-input controls are separate. Receipt bundle: $MOVA_DATA/runs/octopus-wave31-corpus-2026-10-31 (native-commands.json, source-test-map.json, mutation-results.json, .status/.log). No live corpus, teacher, frozen assessment, download or production data used.
Exact byte scope; finite correspondence only. Untested branches source-derived.
This page in Markdown → https://md.groupants.elamur.ai/free/en/mova/tools/langseg/src/lib.rs.groupant/tokenize.groupant/