Elamur Groupantslangseg-latin
The groupant beside tools/langseg/src/latin.rs in the Mova repository: what the code computes, what it never does, how to go back from its outputs, its invariants, cost and checks.
Facts
Implementation material. A Mova code model describes this code at one source revision: partial models, finite checks and their stated scope. It is not a reviewed general exposition, and its labels are shown as written, never raised.
Components
| component | label | where |
|---|---|---|
| Approximant | partial | Approximant |
| Ast | partial | syntax.json |
| Cost | symbolic source derived | Cost |
| Descriptant | present | Descriptant |
| Invariant | partial | Invariant |
| Tests | executed bounded checks | Tests |
Obernants
The typed relations around the forward model: the behaviour complement (what the code never does), the reverse reconstruction (every input compatible with an output) and specialised views.
| relation | type | label | where |
|---|---|---|---|
| Behavior complement | behavior complement | partial source derived model | Behavior complement |
| Reverse reconstruction | reverse reconstruction | partial source derived model | Reverse reconstruction |
| Word mixture | word mixture | partial source derived model | Word mixture |
Model text
Descriptant
The Latin table of the language segmenter: per-language lexicons with counts and character 4-gram models give
log P(w | L) for every Latin word; a Viterbi pass with a switch cost labels the words of a run, and a sentence goes
to the argmax of its summed scores. Builders read a form<TAB>count lexicon, the English in-code dictionary
(en::dict::forms) and CoNLL-U files (addresses left out). Deterministic: lexicons are BTreeMaps and the character
counts are only looked up.
Approximant
Forward: P(w|L) = (1−β_L)·(c_L(w)+½)/N'_L·[w ∈ lex_L] + β_L·P_char(w|L), maximised over the readings of w
(whole; split at an inner apostrophe or hyphen), summed over parts in log space. Token scores are shrunk to their
mean by the weight ω ∈ {0, cap, 1} (address pieces, names/acronyms, other). Labels: argmax_path Σ_i s_i(y_i) − Σ_i cost_i·[y_i ≠ y_{i−1}], cost_i = switch/4 across . ? ! ; : or a quote mark, else switch. Sentence:
argmax_L Σ_i ω-shrunk s_i(L) (ties to the lower table index).
Invariant
P_char is a distribution over strings ending in the boundary mark (Witten–Bell over seen alphabet + 1), so
P(w|L) ≤ 1; β_L ∈ [0.01, 0.5]; a member of a lexicon always has positive dictionary mass; the Viterbi path is
optimal for its objective (the same recursion as Segmenter::viterbi, generalised to K languages); every output label
is a table language; a zero weight gives a token no influence on the sentence decision.
Cost
Table build O(Σ_L |lex_L| · |w| · ORDER) hash inserts (≈0.9 s for 240 876 + 43 083 types, 2026-11-06); a word
O(R · K · |w| · ORDER) lookups for R readings; a run of n words O(n·K) Viterbi after scoring.
Behavior complement
It does not identify languages outside its table (a German word goes to the closer of en/fr); it does not use context beyond the switch cost and punctuation in the gap; it does not learn at run time; it reads nothing from disk after the build (the French lexicon is compiled in); it never labels a non-Latin token.
Reverse reconstruction
From a label path one recovers only the set of score matrices for which that path is optimal (a polytope in score
space), not the words; from a sentence label, the half-space Σ s(L) ≥ Σ s(L'). A lexicon file recovers counts of
lower-case forms, not sentences or casing; the CoNLL-U builder's output admits every corpus with the same form counts.
Word mixture
The two-term mixture behaves as a back-off: a frequent member is decided by counts (function words), an unknown word
only by β_L·P_char, so a language with a larger hapax share pays less for unknown words; measured β: en 0.0391,
fr 0.0628.
Tests
Unit checks: char_model_is_a_distribution_on_short_strings, readings_split_clitics, addresses_stay_out,
tiny_table_switches_at_a_quote (with the negative control: an infinite switch cost keeps one language).
Measurement and its negative control (shuffled lexicons): langseg latin-eval, README of langseg.
Provenance
This page in Markdown → https://md.groupants.elamur.ai/free/en/mova/tools/langseg/src/latin.rs.groupant/