📐corpus-statistics

The groupant beside models/corpus-statistics.groupant in the Mova repository: what the code computes, what it never does, how to go back from its outputs, its invariants, cost and checks.

Facts

Components

Obernants

The typed relations around the forward model: the behaviour complement (what the code never does), the reverse reconstruction (every input compatible with an output) and specialised views.

Model text

descriptant

Corpus statistics descriptant

Current mechanism

This parent joins the shared events::book_body dependency with two distinct consumers and records poly as a separate extraction branch. The useful shared object is a selected raw body; downstream alphabetic histograms, parsed event tuples and detokenized story TSVs observe different units. The hierarchy is an ownership/model grouping, not a newly implemented processing pipeline.

Combiner inputs and output

Combine the authoritative source bindings and their partial syntax/model/evidence artifacts with core-model.md and the typed views. The original ten views retain their three-branch scope; three additional views extend selected-body gating, lexical mining feedback and tagged reconstruction. The first family entry specifies conditional excluded observations; the second retains compatible branch-tagged inputs, dependency state and ambiguous histories. Composition preserves the actual raw text, histogram totals, floating values and branch tag. The output of this modeling step is a source-inspected interaction specification and explicit remaining obligations, not an automatic reconstruction solver or a semantic correctness certificate.

Retained review observations, 2026-10-30

The marker policies are intentionally recorded separately: events/register can yield an empty body on reversed markers while poly rejects the order. The poly repair now runs before slicing, parser acquisition and output writes. Its whole-call cost includes total raw bytes, including arbitrarily long excluded prefixes; counting selected text alone understated work.

Register invariants concern histogram totals and score ordering. Event-specific sentence-index and role-incidence statements cannot serve as register invariants. The reviewed >2^53 singleton counterexample refuted unconditional singleton-NaN prose; the corrected child retains actual floating denominator conditions and a targeted native witness. A subsequent native regression exposed a concrete count-preservation bug: casting score inputs back to u64 changed emitted counts and minimum-count decisions above 2^53. The integrated fix retains raw ca/cb, applies ca >= min and emits exact counts. Cases at 2^53+1 and 2^53+3 cover downward and upward rounding boundaries; the old implementation compiled and failed the large-count assertions before correction. The floating z formula and NaN policy remain unchanged.

Prospective joins and evidence boundary

The source model predicts that a public register file-reading run applies exactly the same body bytes as events::book_body for each readable snapshot, but this parent has not executed that full runtime/I/O join. Native child tests exercise pure body/count composition, numeric fixtures and role/token projections; they do not establish successful corpus publication or synchronized cross-branch snapshots. Potential future checks would use explicit immutable input files, compare body policies, and bind generated artifacts to input hashes. Those checks are proposed, not executed or authorized by this document.

No benchmark, approximate error budget, parser-quality improvement, teacher call, GPU result or universal proof is inferred from this parent.

Book observation and lexical feedback epoch

The hierarchy now also owns bookans and idioms. A real bounded join feeds the selected events body directly to the book copy gate and distinguishes excluded prefix/suffix text from selected story windows; quote rejection remains independent. This adds correspondence for supplied immutable bytes without claiming a full book/teacher pipeline or unique source-passage recovery.

Idiom mining's apparent literal-oddness signal already incorporates existing idiom suppression. Preserve this feedback dependency and the process's once-loaded global hot snapshot. Re-exporting candidate seeds does not imply they became active, and same TSV inputs can classify differently under a different loaded lexical snapshot. Older event/register/poly views remain valuable within their three-tag scope; the new preimage view extends the universe without overwriting those scoped observations.

catalog

Corpus statistics typed interaction views

Revision: source bindings in manifest.json, 2026-10-30. The first ten views retain their original three-branch event/register/poly scope; the final three add book-answer and idiom-mining tags. Each relation is scoped to the operation-tagged domain in core-model.md. Finite child observations and source-inspected joins are distinguished from unexecuted runtime/I/O composition. No relation is a general Rust proof.

nonbehavior

Type: behavior_complement.

In the universe of branch-tagged input/result/effect observations, the source-derived exclusion relation includes a register histogram increment caused solely by an unreadable input; acceptance of a NaN register z by z >= zmin; event extraction being implicitly substituted for alphabetic register counting; and poly acquiring the parser or writing its destination after its own marker-order validation has already failed. These are conditional inner descriptions of excluded behavior. They do not exclude untested filesystem failures or declare all three branches semantically equivalent. Evidence: inspected caller order, native register filtering and poly early-error sentinel fixture.

compatible-sources

Type: reverse_reconstruction.

For register output retain every pair of ordered readable/failed input-instance lists whose actual body projections and lowercase histograms, f64 scores, thresholds and formatting produce that output. Word order, rejected characters, failed paths and excluded text are generally unrecoverable; arbitrary prefix edits are allowed only when they do not change the first-marker selection. For event summaries additionally retain annotation/repair outcomes and omitted postings/tails; for poly retain heading selections and whitespace-compatible selected lines. These are constraint-defined preimages, not an implemented inverse enumerator. If outputs from multiple calls are asserted to share one raw snapshot, intersect their preimages on that same text and dependency state. Dropping the operation tag gives the union of branch-compatible explanations, not unique provenance.

marker-policy-separation

Type: policy_difference_relation.

The shared events/register body helper searches marker substrings anywhere: start is after the first following newline or defaults to zero; end is the first END occurrence or text length; a reversed end yields an empty body via max(start,end). Poly instead selects lines beginning with START/END, requires both, rejects reversed order before slicing, and selects an exact heading plus an optional later exact end heading. A reversed-marker input can therefore give an empty register contribution while poly returns Err. Prefix/heading/line-boundary edits are not generally invariant across these policies. Evidence: native event body slices and poly malformed/repeated-heading fixtures; a simultaneous file-reading comparison of both public APIs is unexecuted.

token-identity-boundary

Type: representation_relation.

Register splits the selected body at every nonalphabetic scalar, admits original runs of UTF-8 byte length greater than one, then lowercases. Thus ASCII A is discarded, single-scalar é survives, İ expands to i plus combining dot, and co-op becomes two runs. Events instead uses its sentence filter and annotator tokens, then selected noun/verb lemmas and roles. Poly records detokenized sentence text and counts tokens containing any alphanumeric character. Neither equality of printed text nor equality of a word spelling identifies equal token units across these branches. Evidence: the register Unicode histogram and child token/event fixtures; general tokenizer/lemma correspondence remains open.

occurrence-versus-semantics

Type: observation_projection.

Register H measures admitted surface-run multiplicity and ignores sentence order and dependency roles. Event C measures accepted predicate/role tuples; actor weight sums role occurrences, so one noun used as patient and recipient contributes twice, and a first rejected role match can mask a later admissible noun. Consequently equal H does not imply equal C, equal C does not recover H, and neither score/count measures semantic completeness or truth. Reconstruction must retain the relevant ordering and parser state when event output is observed. Evidence: native role-masking and repeated-role aggregation fixtures; no new corpus-level equivalence experiment.

floating-count-boundary

Type: numeric_representation_relation.

Original u64 histogram counts ca/cb are retained for emitted row counts and the exact ca >= min gate. Separate copies are converted individually to f64 for scoring, while na+nb is added in u64 before conversion. Those floating paths can diverge above 2^53; they no longer feed a cast-back count gate. The inherited targeted witness uses singleton counts na=2^53+1 and nb=2^53+2: alpha=999.9999999999998, both computed complementary denominators=2.2737367544323206e-13 and z=0.0. A singleton therefore need not be excluded as inf-inf NaN; the familiar small singleton result requires its actual computed zero denominators. The witness constrains this case only, not every large count. Additional native cases at ca=2^53+1 and ca=2^53+3 preserve exact returned counts, admit min=ca and exclude min=ca+1. Compiled old-code witnesses fail because the former cast-back implementation rounded counts and misapplied minimum thresholds. The correction preserves raw integer counts and their gate while leaving the floating z formula and NaN policy unchanged; integer overflow remains excluded.

selection-and-display

Type: lossy_encoding_relation.

Selection precedes formatting: source f64 z must compare >=zmin and the original u64 register count ca must meet min exactly. Rows order by descending total_cmp(z), then lexical word, not by count ties. TSV rounds z to two decimal places and rates to one, so different selected score values can print alike and displayed equality with a threshold cannot reconstruct the original comparison. With a nonempty multi-key register and an empty reference, admitted finite scores can coexist with a reference rate 0/0; finite z does not guarantee finite formatted rates. Child threshold/tie and large-count gate fixtures are finite evidence. Selected row count fields preserve the exact input u64 values; z and per-million rates still use floating arithmetic. Full TSV rendering, including large-count decimal output, remains source-inspected only.

publication-gap

Type: effect_trace_relation.

Register skips unreadable/non-UTF-8 inputs, but its header reports attempted path counts; final create/write/flush errors propagate. Events skips failed book reads and can discard per-book publication errors while successful-instance totals advance; later final-output errors can follow earlier successful files. Poly rejects extraction errors before destination writes, but its later write is not atomic. Thus an Ok result is not a common certificate that all inputs were consumed, all outputs exist, or an all-or-nothing transaction occurred. Evidence: source-inspected I/O control flow and the bounded poly sentinel fixture only. Successful public file pipelines, injected late failures and concurrent writes are unexecuted here.

input-instance-identity

Type: provenance_relation.

Duplicate readable register path instances contribute repeatedly to the histogram; event corpus document support likewise counts successful input instances, not unique filesystem identities. Equal event filename stems can collide on disk while still contributing separately to pooled counts. Poly headers carry the named raw path and selections but no content digest. Therefore equal labels, paths or event counts do not establish that two branches used the same bytes or that outputs form one synchronized corpus snapshot. Preserving shared-snapshot identity requires external provenance absent from these APIs. Evidence: source-inspected folds/publication naming; duplicate-path and collision I/O experiments remain unexecuted.

branch-cost-composition

Type: resource_composition.

Each selected branch charges its own whole-input bytes and dependency work, as defined in core-model.md. The direct register-to-body call is one helper invocation inside counting, not an extra complete event pass. Conversely a separate poly preparation incurs another complete raw read and scan, even if its selected story is tiny. No fictitious sequential poly->events->register pipeline, shared tokenization cache or cancellation of repeated reads is inferred. Serial peak and concurrent sum-space accounting require the actual invocation schedule; none is implemented by this grouping. Evidence: source-level loops and data retention; timing, allocation peaks and throughput remain unmeasured.

selected-body-gate

Type: selected_snapshot_gate_join.

For a supplied immutable raw string x, B=events::book_body(x) and G=bookans::Gate::new(B) share exactly the selected body bytes. The gate stores normalized seven-word window hashes and scans answer windows; its independent quote rule rejects matched quote pairs of at least four words even without book membership. No source-path provenance is created by this composition.

selected_book_body_is_the_copy_gate_input builds prefix/body/suffix text with explicit markers, verifies body selection, and checks copied body prefixes of length 0..10 against COPY_RUN=7. Excluded nine-word prefix/suffix strings pass the copy test; a novel quoted phrase fails independently. This is a bounded pure-helper join, not the full bookans::run/teacher/retry/log path. Hash collisions, case/punctuation normalization and adjacent hashes from different source positions prevent interpreting every flagged run as one uniquely located verbatim passage. Gate acceptance is neither factuality nor legal certification.

lexical-mining-feedback

Type: seed_feedback_boundary.

idioms::run consumes OBJ cell/event exports plus dictionary/MWE inputs, but classification additionally consults Matrix::global().score(v,n,OBJ). That score already uses existing global idiom/light-verb/collocation knowledge to suppress literal OBJ oddness (clamping a known score to at most one). Therefore the miner's literal_odd predicate is suppression-aware, not a raw independent selectional-violation measurement. Earlier accepted lexicon can alter later mining classification even when the supplied TSV bytes are unchanged.

Global hot overlays are loaded through OnceLock: the in-process first loaded snapshot, environment/path selection and fixed seeds belong to the input context. Writing a new idioms.tsv does not establish that an already loaded process rereads it or that exported rows become active knowledge. Dictionary evidence bypasses corpus min/min_books but still requires OBJ count>=3; duplicate pair rows replace pair counts while total/marginals accumulate. Particle exports can overwrite a matching final pair. These source-inspected relations are not an executed feedback/training loop. Child finite tests cover dictionary/MWE parsing and ordering, not full mining, output activation, concurrent publication or a factual assessment of inferred idioms.

additional-branch-preimages

Type: tagged_branch_reconstruction.

Extend the parent's operation-tagged observation universe with book gate/ retrieval and idiom-mining tags, while retaining the original event/register/ poly views on their explicitly scoped three-branch domain. The complementary inner exclusions now include a selected-body gate result contradicting its normalized window/quote policies and a mining classification contradicting the threshold/source-order policy under the same loaded lexical snapshot. These are partial exclusions, not the complement of all runtime failures.

For a book gate observation retain every raw text/body selection, hash-window set, answer normalization and quote structure producing it; equal flags cannot recover unique source passages. Retrieval also retains index multiplicity, rank ties and neighborhood expansion. For idiom output retain ordered accepted/ discarded dictionary rows, corpus marginals, duplicate pair overwrites, thresholds and loaded global knowledge compatible with that output. The preimage of an untagged result is a union of branch-compatible inputs. When two observations are asserted to share a corpus or lexicon, intersect on that exact snapshot; do not multiply independent fibers or infer shared provenance from filenames. No complete inverse enumerator or five-branch pipeline is implemented.

book-segmentation-policy

Type: extraction_policy_relation.

read::split_book has its own START/END, heading, paragraph normalization and service-removal rules. Its length admission precedes final paragraph filtering, so an emitted story may retain less than the admission length. This policy is not equivalent to the events/register body or poly policy. A reverse fiber retains discarded whitespace, excluded material, policy and input snapshot; no unique original book follows.

supplied-tree-summary

Type: lossy_syntax_projection.

read::skeleton consumes a supplied dependency tree and original forms, selecting roles and phrases while losing structure. Reverse reconstruction admits all compatible trees and forms. A plausible or empty summary does not establish annotation quality. Parser, teacher and publication are separate effectful paths; bounded supplied-input tests do not cover the full book workflow.

primer-answer-reconstruction

Type: ocr_marker_and_numeric_fiber.

OCR chunks select the rightmost answer marker and consume a dotted first answer before recognizing the next item label. Story/question/answer segmentation then feeds number normalization and checked integer reachability. Invalid or overflowing arithmetic branches are omitted. Ambiguous absent page answers, permissive numeric grammar and multiple arithmetic preimages remain; finite fixtures do not establish general OCR accuracy.

tests

Evidence and pending correspondence

Date: 2026-10-30. This parent adds no executable test or Rust source. Its original three-branch evidence is source inspection plus the already recorded finite evidence of those children; the expansion evidence below adds the bounded selected-body/gate join. Binding hashes identify the inspected integration source snapshots; root integration refreshes ancestry and registry bindings.

• world/src/events.rs.groupant/model.md#tests and its child evidence: body-marker fixtures, sentence boundaries, role selection/masking and event aggregation. Event book parsing and publication are not executed by the new native fixtures.
• world/src/register.rs.groupant/model.md#tests and its current evidence: alphabetic-byte/lowercase histogram, explicit body-to-count helper composition, interior pooled-score/swap pairs, thresholds and score ties, small degeneracies, the large singleton floating witness at 2^53+1/2^53+2, and exact integer count/minimum boundaries at 2^53+1 and 2^53+3. The old production cast-back implementation compiled and failed both large-count native tests; the correction retains raw u64 counts for output and minimum filtering while preserving floating scoring. Seven register tests plus five events tests pass twice in debug and release with identical normalized observations in the child evidence. File-reading counts, successful TSV output, invalid UTF-8 I/O and formatted nonfinite rates remain outside this bounded runtime evidence.
• world/src/poly.rs.groupant/model.md#tests and evidence.json: finite sentence partition/continuation and heading/extraction fixtures, reversed-marker rejection and an isolated public early-error call preserving a sentinel destination. Successful annotator acquisition/tokenization/TSV publication is not exercised there.

The real register caller edge counts -> events::book_body -> count_body was inspected. Its native pure-helper fixture is narrower than the public runtime path: it does not read corpus files through counts. There is no executed three-branch pipeline in that original evidence; the new selected-body/gate oracle below is a narrower two-helper join. The existing child mutants and repeated runs remain child-level evidence and are not promoted to parent end-to-end validation.

Structural validation checks the original ten declarations plus the three expansion views, artifact references, integrated immediate children and source hashes after registration. It cannot execute the listed source relations or prove the grouping's semantics. Parent AST extraction, successful publication traces, input-snapshot provenance and measured combined costs remain open.

Expansion evidence

Read the actual integration world/tests/groupant_joins.rs harness and the new bookans/idioms source-side models. selected_book_body_is_the_copy_gate_input checks one explicitly selected raw snapshot, body prefixes of lengths 0..10, excluded nine-word prefix/suffix strings, and independent novel quotation rejection. The integrator reports an initial debug pass of all six harness tests; only this fixture directly supports the corpus parent's new join. Final profiles, counts and controls are appended by integration.

Idiom dictionary/MWE ordering tests remain leaf evidence. Full mining, suppression feedback across fresh processes, seed activation and publication are source-inspected gaps. The global hot OnceLock means an in-process rewrite is not a valid experiment for assuming automatic reload. No combined five- branch pipeline, teacher call or semantic quality evaluation was executed here.

Integrated native evidence, 2026-10-30

The integrated checkout passes 138 focused native tests twice in debug and twice in release, with identical normalized test identities/results. This includes 61 new leaf tests, six new actual-API joins and 71 selected existing regressions. The six joins all pass after the reversed-operand control was restored. Leaf source controls, the original defective branches and the late-invalid-index-filter mutation fail the named assertions in their respective evidence. All registered child identities and transitive hashes are reconciled in the final manifests. These checks do not execute a full reader/parser, teacher, school corpus or GPU benchmark, and do not constitute a whole-world lint result.

core-model

Corpus statistics interaction model

Domain

Operation-tagged calls to corpus event statistics, register statistics, Gutenberg story preparation, book-answer gating/retrieval, or idiom mining. The input includes ordered path instances, readable UTF-8 contents or read failures, selected headings, options, and dependency state. Successful allocations and no integer overflow are assumed. Floating operations retain Rust f64 behavior, including rounding, infinity and NaN. Parser, dictionary, repair, tokenizer and filesystem behavior are conditional dependencies, not supplied by this parent.

Immediate ownership children include corpus-events, corpus-register, world-poly, world-bookans and world-idioms; final IDs/bindings are reconciled by integration. This hierarchy groups related observations, not an execution schedule. The original forward/cost paragraphs below describe the first three branches; the expansion section supplies the additional two.

Forward

For register statistics, the inspected call edge is register::run -> counts -> events::book_body -> count_body; each readable path contributes its body to a lowercase histogram and total. run supplies two such maps/totals to ranked_rows, then formats selected rows as TSV. No event extraction, dependency parse or sentence splitting occurs in this branch.

For corpus events, book_stats reads a book, applies the same events::book_body, splits sentences with the event-specific policy, tokenizes/annotates/repairs, and accumulates ordered role events and sentence postings. run aggregates successful book instances and conditionally publishes per-book and corpus artifacts. Counts are observations of this extraction policy, not facts about all characters or actions in the story.

Poly is a separate branch: prep reads a complete raw file, calls story_paragraphs, acquires the annotator only after extraction succeeds, tokenizes/detokenizes paragraph sentences, then writes TSV. Neither register nor events calls poly. A user may separately pass a produced file elsewhere, but no such adapter or its semantics is modeled or executed here.

Composition

Let B be events::book_body, H the alphabetic counting fold, and R the actual floating ranking/selection relation. On each readable register input instance x, counting uses H(B(x)); unreadable inputs contribute no delta. The two folds supply the same maps and totals to R and formatting. This equality of intermediates is the inspected cross-file join; it does not allow replacing H with event tokenization or B with poly heading selection.

The event branch also consumes B(x), but applies sentence filtering and dependency-based event projection. Its output is not an alternative implementation of H. The poly extraction relation P has a different marker/heading/error policy. Branch identity must remain in observations and reverse queries. Joint reconstruction must retain a shared raw text when two separately invoked branches are hypothesized to have read the same snapshot; independent child preimages cannot simply be multiplied. Concurrent read snapshots and a shared-output transaction are absent from the implementation.

Invariant

Without u64 overflow, register histogram totals equal admitted alphabetic-token multiplicities before f64 projection. Empty/ASCII-singleton rejection happens before lowercase expansion. Event histograms count extraction occurrences, and repeated role occupants contribute repeatedly to actor/interactions summaries. These two multiplicity units cannot be equated.

Ranked register rows use descending total_cmp(z), then word order for equal scores; NaN z never passes the source comparison. Selected row counts retain exact original u64 histogram values, and the minimum-count gate compares those integers directly. TSV integer fields format the retained counts without a floating round trip; scoring and per-million rates still undergo f64 conversion and can lose precision above 2^53. Poly early extraction errors precede parser acquisition and output writes. A successful event/register return does not establish that every attempted input was readable or every per-book artifact was published.

Cost

Let N be total raw bytes read across the actually invoked path instances, K admitted register tokens, V union vocabulary, W_s annotated words in event sentence s, E event occurrences, D distinct event keys, L poly input lines, B selected poly bytes, T token characters and O output bytes. Register work includes O(N) byte scanning and O(K log(V+1)+V log(V+1)) map/sort operations, plus key-byte comparisons and output. Event extraction has the child's conservative O(sum_s W_s^3) scan bound plus counting/sorting O(E log(D+1)+D log(D+1)), raw/token/string bytes, and opaque annotation/repair costs. Poly whole-call work includes O(N+L+B+T+O), plus opaque annotation and I/O latency; excluded header bytes still belong to N.

For several separately invoked branches, add their actual work, including repeated reads of the same path: no shared read cache or unified controller exists here. Serial peak space is the maximum live branch footprint plus retained outputs; concurrent footprints may add. Poly retains the raw N-byte string alongside line references, selected paragraphs and output. Event worker/parser memory and filesystem latency remain unmeasured. These are symbolic bounds, not runtime measurements or speedup claims.

Limits

Source inspection establishes the stated call sites and policy differences, not runtime execution of the full file-reading joins. Existing child tests give finite pure-helper correspondence and one poly early-error I/O witness. The original three-branch evidence includes no new end-to-end corpus run, parser load, successful TSV publication, injected publication failure, concurrency measurement or semantic accuracy result. The later bounded body/gate join is recorded separately below. Parent AST is not materialized; child partial projections do not establish a semantic call graph. No approximate executable profile or error budget is attached.

Additional branch composition

Book-answer gating consumes normalized windows of the book bytes supplied to Gate::new, and retrieval combines sentence-index occurrences with lexical hits before top-30 selection, neighborhood expansion and source-order output. The tested events::book_body -> Gate join binds one selected snapshot explicitly; the parent does not assert that every book-answer entry point automatically uses this adapter. Teacher responses, retries and print/log effects remain separate conditional behavior.

Idiom mining reads cell/event/dictionary layers, applies ordered gates and classification, then exports detailed rows and a final pair map. Classification depends on existing global idiom suppression in Matrix::score. The first loaded global hot overlay and fixed seeds are part of the state: output publication is not activation or proof of a completed learning loop. Event/register token units and mining counts remain distinct. Equal file paths do not establish equal contents or an atomic multi-file snapshot.

For gate inputs of B normalized book bytes and A answer bytes, expected fixed- window hashing/scanning work is O(B+A) plus output; collisions and allocations remain implementation concerns. Retrieval sorts K candidates in O(K log K) after index/sentence scanning. Mining over B input bytes, U unique pairs, E event rows and R retained rows has conservative map/sort work O(B+(U+E)log(U+E)+R log R), excluding key lengths and global lookup. Add costs only for actually invoked branches, preserving repeated reads and once-loaded initialization; no shared cache or throughput claim is implied.

Parent behavior complements and reverse reconstruction now include these two tags as described in additional-branch-preimages. The first ten catalog views continue to constrain their original three-branch observations. The six-test integration harness has an initial debug pass reported by the integrator; only its selected-body/gate fixture addresses this corpus subtree directly. Final repeated profiles and compiled controls are integration evidence.

Story reading expansion

The read child adds book segmentation and supplied-tree summaries. Retain its policies separately from book_body and poly. Compose costs only along executed branches, including repeated full scans; parser, teacher and publication costs remain conditional external terms. Earlier views retain their narrower domains. Source binding alone does not bind external lexical tables or annotate a tree.

This page in Markdown → https://md.groupants.elamur.ai/free/en/mova/models/corpus-statistics.groupant/