Elamur Groupantsmova-seaspider-robots
The groupant beside tools/seaspider/src/robots.rs in the Mova repository: what the code computes, what it never does, how to go back from its outputs, its invariants, cost and checks.
Facts
Implementation material. A Mova code model describes this code at one source revision: partial models, finite checks and their stated scope. It is not a reviewed general exposition, and its labels are shown as written, never raised.
Components
| component | label | where |
|---|---|---|
| approximant | partial | Forward approximant |
| ast | partial | syntax.json |
| composition | partial | Composition |
| cost | partial | Cost |
| descriptant | present | Descriptant |
| invariant | partial | Invariant |
| tests | finite evidence | tests |
Obernants
The typed relations around the forward model: the behaviour complement (what the code never does), the reverse reconstruction (every input compatible with an output) and specialised views.
| id | type | label | where |
|---|---|---|---|
behavior_complement | behavior_complement | partial source derived model | behavior_complement |
reverse_reconstruction | reverse_reconstruction | partial source derived model | reverse_reconstruction |
Model text
model
mova-seaspider-robots
Descriptant
RFC 9309 robots.txt parsing, product-token group selection and the longest-match allow/disallow decision with the deciding rule. Source: tools/seaspider/src/robots.rs.
Forward approximant
Strip BOM, cut at 500 KiB on a char boundary, drop comments, group consecutive User-agent lines; tokens lowercased with any '/version' suffix dropped. rules_for(token): merged own groups, else '' groups, else none; empty Allow/Disallow ignored; patterns percent-canonicalised and compiled to anchored regexes ('' -> '.*', final '$' anchors); crawl-delay = max of finite non-negative values. decide(path): '/robots.txt' allowed; longest matching pattern wins, Allow wins a tie, no match allows.
Invariant
decide is a pure function of (rules, canonical path); a path matched by no rule is allowed; for matched rules the verdict equals the allow flag of a maximal-length pattern, Allow preferred at equal length.
Cost
Parse O(bytes); rules_for O(lines) plus regex compilation; decide O(rules x path) with linear-time regex matching.
Composition
Used by crawl::fetch_robots (once per origin per TTL) and robots_decision (per URL and redirect hop).
behavior_complement
Domain: arbitrary text and paths. Not modelled: non-standard directives other than crawl-delay (Sitemap, Host, Clean-param are ignored), the Spider's peer-crawler consent, case-insensitive path matching. Counterexample: with 'User-agent: *\nDisallow: /' and an own group allowing '/', the own group wins and '/page' is allowed (own_group_replaces_star_and_ignores_version). Evidence status: source-derived model with finite checks, not a proof.
reverse_reconstruction
From a verdict with its rule one recovers the deciding pattern and its kind; the robots.txt text is not recoverable (other rules, comments and groups are lost). Every robots.txt containing that maximal rule and no longer matching rule is a compatible antecedent. No unique inverse is claimed.
tests
Checks
Eight unit tests: star fallback, own group with version suffix and a negative control for another robot, grouped and merged agents, longest match and Allow tie, '*' and '$', crawl-delay and empty Disallow, /robots.txt and percent forms, no groups.
Command: cargo test --manifest-path tools/seaspider/Cargo.toml --offline --locked (27 unit + 9 end-to-end tests against a local 127.0.0.1 HTTP server; two identical runs on 2026-11-04). These are finite fixtures; the only live run was the 2026-11-04 smoke against the project's own site.
This page in Markdown → https://md.groupants.elamur.ai/free/en/mova/tools/seaspider/src/robots.rs.groupant/