Auditing the American English translation: ontology, agent harness, tests, and the workbench

This is the procedure behind the American English edition (rbs, public id rbs-en-us, version 2026-10-19-v2.1). It is written so a person who was not in the room can check the claim: a verse was drafted from the frozen Hebrew, Aramaic, or Greek, held to an ontology, measured by a pre-registered harness, and is not approved until three different people say so. The companion essay is A machine translated the Bible from Hebrew and Greek. This paper names the steps, the files, the checks, and the screens.

Nothing here is a claim that a human exegete has approved the text. The public edition is a draft. minimumStatus on the release is proposed. Of 31,085 verses, 27,509 have no blocking check and 3,576 still do. Approval is a receipt on the workbench, and that queue is the point of the last section.

What an auditor can open

A number in this paper is quoted only when the run, the set, and the judge are named beside it. A run whose judge is unnamed is not a number to compare.

The English approach, in the terms the registry uses

The source edition is wlc-ne1904: the Hebrew and Aramaic of the Westminster Leningrad Codex and the Greek as the public-domain interlinear tables carry them, frozen as one JSON file per chapter under evaluations/bible-translation/source/tokens/. Each verse is content-committed. The ingest refuses a re-ingest whose upstream revision moved. A new source revision is a new edition id, not an edit of this one.

The target is American English, variety en-US, brief brief-amer-v1 at 2.2.0. The working store the models write into is rbs-v2. The public name is RBS. The family licence is CC BY 4.0. The acknowledgement on the edition says it is a draft, model-made, from the Hebrew, Aramaic, and Greek.

The unit of work is a pericope, not a fixed window of verses. Section headings in the source markup (\s1, and the acrostic \qa) cut the Bible into 3,817 chunks, median eight verses. A pericope longer than fourteen verses splits at a paragraph; a one- or two-verse piece joins its neighbour inside the same heading, never across one. Positions only are stored (evaluations/bible-translation/source/pericopes.json). The headings' English is not shown to the drafter.

The drafter is shown the source tokens (text, transliteration, parsing, Strong's number), the lexicon entry each number opens, the knowledge-base rendering of each name, the clause outline, the dictionary sense at this occurrence, and the book's attributed setting. It is not shown an English edition of the passage. get_reference_renderings is marked for evaluation only. The plan never feeds it to a model step. WEB, ASV, and KJV are read by the graders. chrF is reported per edition and never pooled. A judge of meaning sees a reference answer flagged as reference, not as the target.

The production shape that drafted the second complete Bible is haiku-haiku-sonnet-3: Claude Haiku 5.5 drafts and revises the first two rounds; Claude Sonnet 5 runs only the third round, and only on a chunk the checks have not cleared. Referents are off. There is no critic in that shape. The critic was measured, lowered independence from published English, and was taken out until its revision is sent back through the same revise loop. That measurement is in the test section.

Export picks one text per verse: an approved verse wins; otherwise a verse with no invented words; otherwise the fewest blocking checks at that verse; otherwise a model-reviewed verse; otherwise the later proposal. A machine status of checked means the checks ran. It does not mean they passed. model-reviewed means a second model family accepted a clean draft. It is not approval.

The ontology, as the Skills graph draws it

Open Bible Translation, Ontology, Graph. The page is the Model lane (classes and relationships), not the Data lane (example records). Scope starts at the domain. "From the Original Languages" is one of thirteen modules. The screenshot below is that page on 20 October 2026, signed in, with that module selected: eleven classes in the module, six of them declared outside it because a class may sit in more than one module when the graph's own rule allows it.

The thirteen modules, in the order the cluster file gives them (ontology/bible-translation.clusters.ttl). An auditor who opens a module on the graph should see these classes and no others as members.

"From the Original Languages" is the module the English task is named by. Its classes:

The scripture knowledge base is not this graph. It is its own named graph, urn:skills:context:scripture:kb, so a context-page count does not walk 85,000 person triples. On 17 October 2026 it held 3,129 persons, 1,007 places (870 geocoded), 5,168 key terms, 45 hand-authored theological concepts whose definition is the rendering rule, 16 divine names, and 34 units. Later concept work raised the authored concept count; the essay states 184, of which 57 were written by hand and 127 were drafted from dictionary domains and then checked by code. Share-alike sources (dictionaries, narrative events) live in a separate named graph and are joined by identifier. The build refuses a source whose licence is not on the allow-list.

What the graph is for, in the English loop: a check does not invent a rule. names reads the node's decided rendering. terminology-consistency reads the concept's stems, and only on the dictionary sense that denotes the concept. divine-name holds only when the Strong's number is beside its partner or is exclusive. A change to a class or a rendering is a change to the source Turtle, then preflight, then a versioned release. The graph on the site is a view of that source. It does not authorize a write.

The skill artifacts that direct the harness

A skill is a SKILL.md in the corpus, canonical id skill:bible-translation/<name>. The runtime loads the ones that govern a model step by that id (GET /skill?id=… on skills-a2a), strips the YAML frontmatter, and keeps the body with the content commitment. On the trace, governedBySkill is skill:bible-translation/<name>@<commitment>. An auditor resolves that id in the corpus and checks the body against the system prompt of the activity. A corpus miss does not invent a skill: the plan falls back to a built-in brief and the run records a warning.

The frontmatter is the contract. capability is the operation. covers names the ontology classes the skill may produce. excludes names the classes it must not touch. tools names the MCP tools it may call. utterances include asks it must refuse (isNot). The body is the instruction the model step receives. Code then appends invariants the skill is not allowed to drop: no existing English translation of the passage, the tokens are the text, alignment rules, and — when the ask turned them on — the syntax/sense rules and the context rules.

Skills the English translate run actually loads (apps/translation-runtime/src/skills.ts):

translate-from-source tells the model the order of work and the tools those steps are: read get_source_tokens, open lookup_lexicon (a concept rule on the entry is binding; a divine-name convention is exact; a unit stays the source unit), resolve lookup_names (the referent's decided rendering, not the spelling), and on the memory arm only find_similar_approved. It forbids an English edition as source, a localisation, and a request to make the text say more than the tokens. The alignment contract in the skill is the string the checker reads.

en-us-translation-brief is the norm, not a draft. It states what is shown and never shown, the textual basis (WLC and the Greek the frozen tables carry), names and divine names through the knowledge base, register, units, and ambiguity. A change of brief is a new version a custodian cuts. A run does not edit it. The digest of the brief is what a decision record should name.

Skills that name the tools, but whose bodies are not injected into the translate plan. The code is the execution. The skill is the written contract an auditor compares to the tool:

Skills that belong to the other task, Irish-English localisation of the BSB, and do not direct this harness: assemble-passage-context, rules-pass, localise-segments, structure-and-meaning-checks. A translate arm on a localise ask is refused, and the reverse.

The agent harness, step by step

One passage is one POST /harness/ask with task: translate, an arm, a target edition, and a runRef of the form <edition>/<osis range>. The driver is scripts/translate-bible.mts. It walks pericopes, appends a ledger outside the repository (~/eval-store/rbs/ledger.jsonl for the first edition, ~/eval-store/rbs-v2 for the second), and can resume. --retry-blocking re-runs only chunks whose last checks still block. The PROV record of the chunk — every activity, its inputs, outputs, model, and token counts — is ~/eval-store/<store>/runs/<ref>.json. Those words are not committed.

The plan the production arm builds, with analysis and context on and the critic off, is this. A step whose when is false is recorded as skipped with a null output. It is not deleted.

What each step reads, which skill directs it, and what an auditor should find in the run file. Tool steps are not governed by a skill body. The skill named in parentheses is the contract for that tool.

get_reference_renderings is a tool on source-fidelity-checks and is marked forEvaluationOnly. Graders and the recall gate may read WEB, ASV, KJV, and the BSB. No model step's input names it. An auditor greps the run file for that tool id under a step whose kind is model and expects no hits.

The revise instruction tells the model that a match with a published Bible is an observation, not an order to paraphrase. Where a check names a concept rule, the word of the key term stays the rule's word. The construction around it may change. That rule was added after John 3:18, where a recall gate had pushed a reviser from "believes" / "saved" into "trusts" / "escapes."

Checks an auditor can re-run without a model, against a stored draft:

The checker version is stamped on every checks object (CHECKER_VERSION in source.ts). Two results are compared only inside one version. As of this paper the English whole-Bible release 2026-10-19-v2.1 was cut under the checker the registry records on that release; later checker edits (including the closed-lexicon rule used for Tar Melbene) do not rewrite that release. A re-check is a new file, not an edit of the old one.

Prompt caching is part of the audit, not a performance note. The passage material is the message prefix. The system prompt and that prefix are marked ephemeral on Anthropic. The tool description is identical on the draft and every revision, because tools sit in front of the system in the cache key. usage.cacheReadTokens on the activity is the evidence the revision read the prefix from cache. A run that rewrote the tool description and shows zero cache reads failed this check.

Cost, so the shape can be audited against the bill. The strong model drafting every verse was about $0.076 a verse. The second complete draft, cheap model throughout except seventeen long passages and one passage a content filter refused, was about $49 for 31,085 verses, about $0.0016 a verse. The measured production shape before that spend, haiku-haiku-sonnet-3 on the development set, was $0.0076 a verse, about $236 for the Bible before a retry pass. Those are plan costs on the dev set, not the invoice of the whole-Bible run. The ledger is the invoice.

The test harness

The harness measures the checker before it measures the model. That order is the audit.

Floor and ceiling set the rules. The first development run put a public-domain human translation (the World English Bible) and the interlinear gloss through the checks. The WEB failed a blocking check on 67% of verses. The gloss failed on 80%. The Bibles were not wrong. Five rules were: curly apostrophes, one "not" demanded per Hebrew negator, numbers that did not compose, a divine-name rule that fired on "poor in spirit," and dropped tokens that English absorbs. The rules were tuned until both read 0.99. Only then were model drafts scored. A rule change that is not re-run on the floor and the ceiling is not a finished change.

Three sets, never pooled.

A localise arm on a set whose task is translate is refused. Graders are one list. A translate smoke that grades only scope is a bug, and it happened once, when the durable object's one-run path carried its own grader list.

Arms, declared before the run. control-web proposes the WEB verse as it stands. It is the ceiling. Its chrF against itself is excluded. gloss lays the tables' glosses in gloss order, one alignment per token. It is the floor, and on alignment checks it is the ceiling, because it is aligned by construction. model-src, model-lex, and model-mem are the model variants. Later arms (haiku-sonnet-revise, haiku-haiku-sonnet-3, haiku-sonnet-revise-critic, haiku-sonnet-revise-ref) were added as named shapes and measured on dev before any of them became the default. An arm that is not in the arms file is not a result.

Graders read the proposal, not the first draft. For a period every deterministic grader read the first checks step, so a revise arm was scored on text its revisions had replaced. tr-dev-prod-1 then reported coverage 0.64 on chunks whose proposal had blocking: 0. Graders now read proposal.checks first. A test pins it. scripts/translation-regrade.mts <id> re-runs the deterministic graders over a stored result and recomputes summaries. Judge verdicts are kept. An auditor who finds a coverage number from before that fix should ask for the regrade file.

Planted errors price a rule change. Regression v2 planted 839 cases. Three new kinds scored 0 of 256 on the first reading because the catch map had no entry for them, not because the checker missed them. The map is now total, held by a harness test. Re-read under that map: added theological term 1.00 (118/118), dropped emphatic 1.00 (30/30), flipped person 0.80 (89/111). Overall 0.93 (780/835, four excluded). Flipped person stays the known gap: there is no person–number check. The essay's later statement that the catch rate held at 0.93 through checker .17 is the number to re-check against tr-regression-v3-1 if a rule has moved since.

The held-out number, with its judge. Run tr-heldout-prod-1, set bible-translation-translate-v1-heldout@1, judge Claude Sonnet 5.5, tier strong, one run per arm, 17 October 2026.

A second judge, gpt-5.5, read the same stored runs: fidelity 0.792 against Sonnet 5.5's 0.780. Fluency rankings agreed that the draft sat above the WEB. The two judges are not pooled.

The critic arm, measured on dev after the regrade, raised chrF against the WEB by 0.077 and lowered independence by 0.107, both intervals clear of zero, at 2.4 times the cost. That is why it is not in the production shape. The referent pass, measured as haiku-sonnet-revise-ref, moved nothing that cleared zero, at 2.8 times the cost. It stays optional.

What the harness refuses. A repeat that is not a fresh context. An arm without the gateway off, because a gateway profile was cutting answers and swapping the judge. A result with no verdict and no failureCause. A planted kind with no catch-map entry (it would score 0 forever). Comparing two checker versions as if they were one.

The workbench

No verse is approved by the harness. The workbench at workbench.faithnet.io is the collaborative review. The edition under review is a setting, not a hard-coded string: GET /workbench/edition returns the edition id and the version. For the public English text that is rbs-v2 and 2026-10-19-v2.1. A proposal whose runRef is from another edition is marked, not mixed in.

A run is synced into the queue by scripts/workbench-sync-run.mjs against the store that produced the release. Each verse becomes a proposal item. The item carries the text, the alignments, the checks, the arm, the models, and the run. The reviewer sees the source beside the draft.

Three different people take a verse to approved. The server holds the rule. The buttons in the review pane only offer the act the server will accept.

A receipt is over a hash of the exact text and alignments. An edit after approval makes every earlier receipt visibly stale. The chair can see that. Undo is not a delete.

Ask, beside the verse. A reviewer can ask a stronger model whether the verse is wrong or the checker is. The answer comes back as one of: accept, edit and accept, reject, defer, or file a change request. The model is named beside the answer. The button does the act. It does not apply a knowledge-base change by itself.

Change requests. A finding against a rule, a name, or the memory is a ChangeRequest: the verse, the check, the proposed change. The proposer does not accept it. The approving role is data on the kind (or the chair). Accepted is not applied. Applying it means editing the ontology source, passing preflight, and shipping a release whose changelog names the findings it closes. The skills text, the checker's language profile, and the plan are regenerated from that source. The release is measured on the same pre-registered set as the one before it. A release with no measurement is not a release.

The worked example already in the essay is the ruling on Ἰουδαῖοι in John: "the Jews," never "the Jewish leaders," never "the Judeans," with a note where the authorities are meant. That ruling is a record on the concept. Checker .14 holds drafts to it. The verses of the first draft that had chosen otherwise are listed for revision. An auditor should find the ruling, the checker version, and the verse list, and should not find a silent string replace.

Release. The chair records a ReleaseManifest: version, scope, books, verses considered, verses shipped, verses held back, blocking count, check-clean count, decision base, brief, checker version. scripts/build-editions-registry.py projects that into editions.json. The explorer, the landing page, and the workbench read that file. They do not each keep a version constant. For English today the manifest says version 2026-10-19-v2.1, scope whole, 31,085 verses, 3,576 blocking, 27,509 check-clean, minimum status proposed.

A checklist

1. Open the graph URL above. Confirm the module "From the Original Languages" lists SourceToken, TokenAlignment, TranslationBrief, and ReferenceEdition.
2. On a model activity, resolve governedBySkill. It is skill:bible-translation/translate-from-source@<commitment>. Fetch that id from the corpus and confirm the system prompt starts with that body, then the brief, then the invariants.
3. Confirm get_reference_renderings is not an input of draft or any revise step, and that tokens.prompt contains no verse of a published English Bible.
4. Confirm the proposal's checks.checkerVersion matches the release manifest's checker version, or that a later re-check is a separate file.
5. Confirm final kept the pass with fewer failures when a later revise got worse. The step output says which candidate won.
6. Confirm the last checks carry recallBlocks, and that a verse still identical to a held edition is blocking.
7. Confirm the held-out table above against evaluations/results/translation/ for tr-heldout-prod-1, judge Sonnet 5.5, and that the gpt-5.5 reading is a separate file.
8. Confirm the regression catch map has an entry for every planted kind, and that flipped person is still reported near 0.80 rather than silently dropped.
9. On the workbench, confirm the header names 2026-10-19-v2.1 and edition rbs-v2. Accept a verse as one person, validate as a second, and confirm the first person cannot approve it.
10. Edit an accepted verse and confirm the earlier receipt is marked stale.
11. File a change request and confirm the proposer cannot accept it, and that "accepted" does not change the ontology until a release is cut and measured.

If any of those fail, the edition's stage on the public page is the honest one: a draft, with the open check named beside the verse.