📐algorithm-byte-pair-encoding

The groupant beside algorithms/src/byte_pair_encoding/mod.rs in the Mova repository: what the code computes, what it never does, how to go back from its outputs, its invariants, cost and checks.

Facts

Components

Obernants

The typed relations around the forward model: the behaviour complement (what the code never does), the reverse reconstruction (every input compatible with an output) and specialised views.

Model text

descriptant

Epoch memory — 2026-10-30

Original bounded byte byte_pair_encoding. Reference: Gage (1994), C Users Journal 12(2), A New Algorithm for Data Compression. Prospective equations preceded implementation. Exact semantics and observed checks are distinct from syntax binding. Round-trip cannot alone validate two mutually wrong codecs; retain hand-derived streams and malformed checks. No timing claim or percentage approximant.

model

Mathematical combiner

For source S, partial syntax P(S), forward relation F and typed relations O, the sidecar records (hash(S),P(S),F,O,I). Source identity binds descriptions to bytes but does not prove F. Tests intersect compatible implementations with finite observations. Composition with callers requires domain caps and exact model identity; no global Mova accuracy claim follows. For fixed model, token expansion is unique but inverse tokenization is ambiguous: literals [97,98] and merged [256] both decode ab when first merge is (97,98). Learned model cannot be reconstructed from one encoded stream; unchanged corpus tokens admit many models.

approximant

Forward, invariant, and cost

Count adjacent overlapping pairs. Choose frequency>=2 maximum with lexicographically smallest tie; substitute left-to-right nonoverlapping occurrences. ID 256+i is merge i, whose children have smaller IDs. Encode applies stored substitutions in order; decode expands their DAG.

O(Mnlog n) training, O(M*n) encode, O(M+tokens+output) decode, O(n+M+output) storage.

Input/token cap 65536 before merge cap 3840. Decode checks all token IDs before budget. Expansion sizes saturate at 65537, so integer addition <=131074. Output cap min(user,65536). Private merges prohibit malformed/cyclic models; allocations may panic/abort.

Invariant: Any output that violates recursive expansion under this exact model is excluded; token IDs >=256+merge_count cannot succeed. Models are not interchangeable: token 256 may expand to ab or cd.

Original implementation informed by Gage (1994), C Users Journal 12(2), A New Algorithm for Data Compression; no copied implementation. Exact categorical semantics; no useful certified percentage surrogate is attached. Patience minimality is explicitly unsupported. Finite fixtures do not constitute universal proof.

models

Typed relations

behavior_complement

Any output that violates recursive expansion under this exact model is excluded; token IDs >=256+merge_count cannot succeed. Models are not interchangeable: token 256 may expand to ab or cd.

reverse_reconstruction

For fixed model, token expansion is unique but inverse tokenization is ambiguous: literals [97,98] and merged [256] both decode ab when first merge is (97,98). Learned model cannot be reconstructed from one encoded stream; unchanged corpus tokens admit many models.

transition_relation

Count adjacent overlapping pairs. Choose frequency>=2 maximum with lexicographically smallest tie; substitute left-to-right nonoverlapping occurrences. ID 256+i is merge i, whose children have smaller IDs. Encode applies stored substitutions in order; decode expands their DAG.

finite_arithmetic

Input/token cap 65536 before merge cap 3840. Decode checks all token IDs before budget. Expansion sizes saturate at 65537, so integer addition <=131074. Output cap min(user,65536). Private merges prohibit malformed/cyclic models; allocations may panic/abort.

resource_observation

O(Mnlog n) training, O(M*n) encode, O(M+tokens+output) decode, O(n+M+output) storage. No callbacks (count zero), no IO, no randomized state. Input errors are values; allocator failures and stack exhaustion are environmental and not caught.

deterministic_ties

Traversal is increasing index; maps are ordered. Myers and decoders have no optimization ties; LZW freezes dictionary rather than clearing it; LZ77 keeps nearest distance; BPE keeps lexicographically smallest pair; patience LIS keeps earliest predecessor and endpoint. These are per-algorithm rules, not arbitrary hash iteration.

tests

Checks

Run cargo test -p mova-algorithms byte_pair_encoding::tests in debug/release twice. Preserve normalized transcripts, compiled negative control, and exact restoration under $MOVA_DATA/runs/octopus-wave25-sequences-2026-10-30. No benchmark evidence yet.

prospective

Prospective model — 2026-10-30

Original bounded byte byte_pair_encoding implementation. Before implementation: exact transform, deterministic traversal, no callbacks; input cap 65536 (diff 4096), dictionary 4096, output cap min(caller,65536). Allocation may panic/abort; resource checks prevent size-driven overflow but do not certify allocator success. Myers uses bit-vector edit row recurrence with wrapping addition; LZW extends previous phrase by next first byte, freezing at dictionary capacity; LZ77 greedy longest previous match permits overlap; BPE repeatedly merges most frequent pair, lexicographically smallest tie, applying learned merges in order; patience diff recursively anchors unique shared bytes by increasing subsequence and uses delete/insert fallback. Expected costs respectively O(n), O(n * phrase length), O(nwindowmatch), O(mergesnlog n), O(n^3) conservative. Correctness is exact distance or lossless reconstruction; patience edit count is not guaranteed minimal. No percentage approximant justified. Independent DP/exhaustive and round-trip/malformed checks planned; no timing claim.

validation

Observed validation — 2026-10-30

The five-module batch adds 14 native tests. All pass twice debug and twice release; sorted named test records match across all four runs. Initial integrated library run: 632 passed. Strict library Clippy passed. Each production file has a compiled mutant rejected by a named assertion, followed by byte-for-byte restoration and a passing module rerun: Myers score+1/exhaustive_dp; LZW flipped decoded byte/known_and_malformed; LZ77 match length-1/greedy_overlap_and_invalid; BPE tie comparison >=/ties_and_model_identity; patience anchor Insert instead of Equal/unique_anchor_and_nonminimal_witness.

Each of ten source/test files has paired identical partial syntax projections, refreshed after all mutants were restored. Receipts: $MOVA_DATA/runs/octopus-wave25-sequences-2026-10-30/{focused-debug-1.log,focused-debug-2.log,focused-release-1.log,focused-release-2.log,normalized-,mutations.log,mutant-,restored-*,syntax.log,clippy.log}. No timing or speedup evidence; allocator failures untested. These finite observations and lexical source bindings do not establish a semantic proof or global Mova quality improvement.

This page in Markdown → https://md.groupants.elamur.ai/free/en/mova/algorithms/src/byte_pair_encoding/mod.rs.groupant/