Add ERC: Preregistered Acceptance Criteria

[richard7463]

Summary

This PR adds a draft ERC for Preregistered Acceptance Criteria.

Existing agent verification standards reach a verdict by recomputation: derive
the artifact again and compare digests. That only works where a verdict is a
function of the artifact. Most of what agents pay for is judged rather than
computed — a generated deliverable, a human review, an on-site inspection.

This ERC freezes acceptance criteria and typed evidence obligations on-chain
before evidence exists, and requires the final attestation to be itemized
against that frozen list (MET / WAIVED / UNMET / NOT_APPLICABLE). A verdict that
declares an outcome satisfied while a required obligation is recorded UNMET is
rejected at the contract level rather than merely flagged; so is a verdict that
records failure while no obligation is unmet.

Scope is deliberately narrow: registry interface, three off-chain document
schemas (criteria, evidence bundle, attestation), a packed itemized-verdict
encoding, and contract-enforced consistency. Evidence formats, payment, and
identity are out of scope, so it composes with existing standards (ERC-8126
verdicts, ERC-8183 settlement, ERC-8004
identity) instead of competing with them.

Discussion

Ethereum Magicians: https://ethereum-magicians.org/t/erc-8412-preregistered-acceptance-criteria/29609

Validation

• Preamble follows EIP-1 (title/description length, discussions-to, type,
  category, created, requires).
• Draft content reviewed against the recomputation boundary and the
  outcome-switching literature (ICMJE registration, COMPare) cited in the
  motivation.
• Reference implementation and conformance vectors in assets/erc-8412/:
  registry contract, executable model, reference off-chain verifier, 43 on-chain
  cases (E1–E14) with a generated Foundry suite, 23 off-chain document packages
  (O1–O5), and a mutation test confirming every invariant is exercised.
  python assets/erc-8412/tools/run_all.py runs all checks without dependencies.

[eip-review-bot]

File ERCS/erc-8412.md

Requires 1 more review from Editors: @g11tech, @jochem-brouwer, @samwilsn, @xinbenlv

[abcoathup]

Suggested change

	eip: 8411
	eip: 8412

Assigning next sequential EIP/ERC/RIP number.
Numbers are assigned by editors & associates. Please don't attempt to self assign.

Please also update the filename.

[abcoathup]

Suggested change

	discussions-to: https://ethereum-magicians.org/t/pre-erc-discussion-preregistered-acceptance-criteria/29609
	discussions-to: https://ethereum-magicians.org/t/erc-8412-preregistered-acceptance-criteria/29609

Updated Eth Magicians title with assigned number

[renezander030]

The Reference Implementation section says the registry contract and conformance vectors are the next deliverable before Review is requested. I would like to contribute the vectors, if that is useful and not already in progress.

Proposed shape: a JSON fixture set covering E1-E9 and O1-O4, each case naming the invariant it exercises, plus a minimal conforming registry to run them against so the fixtures are executable rather than descriptive. Either landing in assets/erc-8412/ here, or in a standalone repo you vendor, whichever you prefer.

One thing worth settling before the vectors are written, because a fixture can only pin behaviour the text has already decided. The expiry boundary is currently specified with two different comparisons:

• E7: attestOutcome MUST revert once block.timestamp > expiry, so an attestation at exactly expiry is permitted.
• E8: resolveExpired MUST revert before expiry, so a resolution at exactly expiry is permitted too.

In the block where block.timestamp == expiry both calls are available. E8 guards one ordering, reverting if an attestation exists, but there is no mirror: E3 bounds attestations to one per preregistration rather than one attestation after a resolution, and E9 only rejects None as an input verdict. So resolveExpired can write ExpiredUnresolved and a later attestOutcome in the same block can overwrite it with Satisfied without violating anything in section 7. Section 9 hands ExpiredUnresolved straight to settlement layers as non-satisfaction, so this is reachable by any contract acting on ExpiredResolved.

Two lines close it: make E7 and E8 test the same comparison so the windows are disjoint by construction, and give attestOutcome the guard E8 already has in the other direction, reverting when a terminal verdict is recorded. Same shape one level down, section 9 says resolveExpired applies when neither an attestation nor a superseding preregistration exists, but E8 only names the attestation.

(Also raised on the discussions thread, post #5.)

Happy to write the vectors either way. I just do not want to encode a boundary the text has not chosen yet.

[babyblueviper1]

Independently reread E3/E7/E8/E9 against @renezander030's point above — the gap is real, not just a boundary-comparison nitpick.

At block.timestamp == expiry with no prior attestation: E7 doesn't revert (> not >=), E8 doesn't revert (expiry isn't strictly before expiry, and no attestation exists yet to trigger its other guard). Both calls are live in the same block. resolveExpired firing first writes ExpiredUnresolved; nothing then stops attestOutcome firing after it in the same block and writing Satisfied over it — E3's "one attestation per preregistration" and E9's "reject verdict == None" both still hold, since neither speaks to whether a resolution already exists.

This isn't just a same-block tie — tightening the comparison operators (>=/<=) only removes the exact-timestamp overlap, it doesn't touch the underlying gap: nothing in section 7 makes ExpiredUnresolved itself terminal. E8 already has the right shape for this ("MUST revert...if an attestation exists") — it's just one-directional. E7 needs the mirror: attestOutcome MUST also revert if the preregistration already has a non-None verdict (from a prior resolveExpired, not just a prior attestation). That closes it independent of exact timestamp comparisons, and matches the doc's own stated goal in section 9 (ExpiredUnresolved as attributable, settlement-relevant state) — right now that state is reachable but not actually final.

Worth pinning as its own invariant (something like E10: attestOutcome MUST revert if a resolution already exists for this preregistrationId) before @renezander030's conformance vectors get written — a fixture can only pin behavior the text has already decided, same point already made above.

[babyblueviper1]

Recomputed at head 85da18f, running tools/mutation_test.py locally against the on-chain vectors:

• All 15 single-invariant mutants are caught (E1-E14 + DUP).
• The replay of the original spec text (E8 < expiry, no E10) is caught by 3 cases: second-attestation, resolve-at-expiry, boundary-no-overwrite.

Reading PreregisteredCriteria.sol, this closes the gap from the 09-13 thread (raised first by @renezander030) on both halves:

• The boundary. attestOutcome is live only while block.timestamp <= expiry (E7), and resolveExpired only once block.timestamp > expiry (E8). The two can no longer both be callable in the same block.
• Finality. attestOutcome now reverts with E10_VerdictFinal whenever any verdict exists, including the ExpiredUnresolved that resolveExpired writes. So even if the time windows ever overlapped again, a resolution can't be overwritten.

One question, not a finding: in the mutation run, disabling E10 alone is caught by a single vector (second-attestation). The resolve-then-attest order is also blocked by E7, so no vector depends on E10 for that path. Is E10 meant as defence in depth there? If so, a vector with E7 relaxed (or a resolve-then-attest case under a mutant that disables both E7 and E10) would pin that E10 is independently load-bearing for resolutions, not only for double attestation.

[github-actions]

The commit 85da18f (as a parent of dc44de1) contains errors.
Please inspect the Run Summary for details.

[vijaygopalbalasa]

Hi Richard, I run Judge Protocol, a deterministic evaluator for ERC-8183 jobs on Arc testnet. It looks like a live instance of the computable corner of this ERC, so I wanted to share it as a data point.

How it maps today:

• Preregistration: the client writes the criteria into the ERC-8183 job description at createJob, so they are on chain and timestamped before any deliverable exists. The verdict commits to keccak256 of the canonical criteria JSON, and the evaluator contract can also bind a registered hash per job and reverts any verdict whose criteria differ.
• Obligations: each check (length, contains, schema, checksum, http-endpoint) is a typed obligation with its own result, and the signed verdict commits to an evidence hash over the per-check results. The decision rule is a weighted score against a threshold; at threshold 100 it is ALL_REQUIRED.
• Refutability: anyone can recompute the hashes and the itemized results in a browser from chain data (plus the file, for a deliverable hosted off chain): https://judge-protocol-verifier.vercel.app. The exception is a live http-endpoint probe, which is recorded rather than re-derivable, close to the indeterminacy your section 9 handles.

Honest limits: 24 verdicts so far, all on my own test jobs, testnet only, and only computable checks, so it covers the recomputable subset your motivation contrasts with, not judged evidence like photos.

If useful, I'd be glad to add an ERC-8412 profile: emit the criteria document with our checks as obligations, register its digest, publish the itemized verdict in your packed encoding, and run it against your vectors. Code: https://github.com/vijaygopalbalasa/judge-protocol

[richard7463]

Thanks @renezander030, @babyblueviper1 and @vijaygopalbalasa — and apologies for the slow reply. The boundary finding is correct. Working through it, and then trying to make O1–O4 executable, turned up several related gaps. All fixes are in the latest commit. The interface changed, so anything built against the previous text needs the new signatures.

1. Expiry boundary and finality (the 09-13 finding)

• E8 now reverts if block.timestamp <= expiry. E7 is unchanged, so at exactly expiry only attestation is possible; the windows are disjoint.
• New E10: once any terminal verdict is recorded, including ExpiredUnresolved, nothing can record or replace a verdict.

2. Attestation front-running

With one immutable attestation per preregistration and no on-chain verifier binding, anyone could occupy the slot first with an arbitrary verdict. preregister now takes address verifier, and attestOutcome reverts unless msg.sender == verifier (E3, E14). The verifier may be a contract, which is how committees and appeal windows compose without touching the core.

3. Supersession is now enforceable

supersedes lived only in the criteria document. preregister now takes bytes32 supersedes and the registry records supersededBy. E12: same author and taskRef, only once, only from no verdict or Indeterminate. E13: a superseded preregistration cannot be attested. §9 notes that a pre-attestation supersession without operator consent SHOULD be treated as cancellation by the author, since by O1 it voids in-flight evidence.

4. Symmetric protection

E11: NotSatisfied or Indeterminate requires at least one UNMET. E4 protects the payer from a false pass; E11 protects the operator from a false fail. E9 now also rejects attesting ExpiredUnresolved directly.

5. Making O1–O4 executable exposed undefined pieces

• decisionRule had no grammar, so O4 could not be checked. §2 now defines ALL_REQUIRED and ALL_REQUIRED_AND_AT_LEAST(k); custom rules are reverse-DNS namespaced.
• Indeterminate contradicted O4. A rule over outcomes can only yield Satisfied/NotSatisfied, so every Indeterminate was refutable. Undecided obligations are now encoded UNMET and listed in undecided; O4 says Indeterminate is valid only if the rule does not yield Satisfied.
• The attestation document and waiver record had no schema. §4 defines both; waivers are EIP-712 signatures bound to registry, chain and preregistrationId.
• Nothing checked on-chain flags against the document. A required obligation registered as optional escapes E4. O5 now requires the documents to match chain state, flags included.

6. Reference implementation and vectors

To pin the text I drafted a reference implementation: the registry, an executable model, a reference off-chain verifier, 43 on-chain cases (E1–E14, one invariant each), 23 off-chain packages (O1–O5) with real digests and EIP-712 signatures, and a generated Foundry suite. A mutation test switches off each invariant in turn and confirms a vector fails; it also replays the old boundary text and catches the overwrite you reported. python tools/run_all.py runs everything with no dependencies.

On E10 as defence in depth (@babyblueviper1)

Yes — deliberate, and your reading is right that E7 already covers the resolve-then-attest order on its own.

E10 is there so that finality does not depend on the boundary arithmetic staying correct. The two fixes are separately load-bearing on purpose: if someone later relaxes E7 back to >=, E10 still refuses to overwrite a recorded verdict. Your point that no vector pins that is fair — a case that only fails when E7 and E10 are both disabled would be the right shape, and the suite should have one. I'll add it as a follow-up rather than change test files inside this review cycle.

Thank you also for re-running the mutants yourself. That the old text is caught by three distinct cases is the most useful thing anyone has told me about this draft.

Judge Protocol (@vijaygopalbalasa)

This is the composition case the ERC is written for. A deterministic evaluator on ERC-8183 jobs, with criteria written at createJob and an itemised verdict committing to an evidence hash, is the shape §2 and §4 describe. Your handling of the live http-endpoint probe as recorded rather than re-derived matches §9's intent: that outcome is indeterminate by construction and should be recorded, not recomputed.

Please do add a profile. vectors/offchain/ has complete document packages built to run against, and §5's packed encoding is two bits per obligation. If it lands, this ERC gets its first independent implementation of the outcome side — worth more to the discussion than anything I can add to the text myself.

@renezander030, none of this replaces your offer; it is a first draft so the text was decided before anyone encoded it. I would value your review of the vectors, and especially adversarial cases I have not thought of. Happy to credit you in the vectors, and to discuss co-authorship if you would like to keep contributing.

[vijaygopalbalasa]

Thanks Richard. The profile is in: https://github.com/vijaygopalbalasa/judge-protocol/blob/master/docs/ERC-8412.md (built against ff9fbc7).

How a Judge job maps:

• One obligation per check, in order. Deterministic checks are DOCUMENT with application/octet-stream, and the evidence item is the provider's committed deliverable (its digest is the bytes32 they submitted to ERC-8183). The live http-endpoint probe is an AGENT_LOG bound to the judge's key and carried as the judge's own record, so it is recorded rather than re-derived, as you put it.
• taskRef = keccak256(abi.encode(chainId, acp, jobId)). The client preregisters right after createJob and before the provider submits, so every evidence item is younger than the criteria (O1).
• A check is required when failing it alone fails the job. decisionRule is ALL_REQUIRED whenever that is exact (always at threshold 100), ALL_REQUIRED_AND_AT_LEAST(k) when the optional checks have equal integer weights, and otherwise a namespaced io.github.vijaygopalbalasa.judge.weighted-threshold-v1, defined in the doc.
• PASS and REJECT become Satisfied and NotSatisfied, with MET or UNMET per check. The judge never attests Indeterminate: when it cannot rule it abstains, and after expiry anyone can resolve the record to ExpiredUnresolved, which lines up with the ERC-8183 refund.

Live on Arc testnet: I deployed your PreregisteredCriteria.sol unchanged at 0x48c3a1812F2dFc762a80dbD5c65e9C7B0BB25ae4 and ran two jobs end to end (create, preregister, submit, judge and settle, attest): 186777 is Satisfied (outcomes 0x50) and 186778 is NotSatisfied (0x40). The packages are in docs/erc8412 in your vector format, with the chain state read back from the registry, and verify.py returns valid with nothing unchecked on both.

Against your vectors: a separate JS canonicalizer reproduces your digests for all 69 documents in the 23 off-chain packages, and a JS port of verify.py reports exactly the expected violations for each of them. Both run in our CI, along with your verify.py over the packages we emit.

Two things came out of doing it:

1. Mapping our pass rule onto ALL_REQUIRED turned up a bug on our side. The score was rounded, so a failing check with a tiny weight (1 against 1000) could still score 100 and pass a threshold of 100. Writing the rule down as ALL_REQUIRED is what exposed it. It is fixed, and none of our 24 existing verdicts sat on that edge.
2. Since documents carry integers only (§2), criteria with a fractional weight cannot be expressed, so the profile refuses them rather than rounding. That seems right to me; I am flagging it as a practical consequence for anyone mapping weighted scores.

Limits, plainly: testnet, my own test jobs, and the live judge does not publish an ERC-8412 record for every ruling yet (the two runs go through a script that drives the same engine). If the interface moves again I will follow it.

[vijaygopalbalasa]

Following up on the one limit in my last comment: the live judge now attests ERC-8412 records itself, with no script in the loop.

• JudgeAttestor (0x78E87A8E43e8E2784C12bF39eB6e2ea7C990fB15, source-verified) is the verifier a client names in preregister, so your E3/E14 binding holds: it records an attestation only when a key JudgeEvaluator trusts signed it, and anyone can relay it. It is the "verifier may be a contract" case.
• The client preregisters with one kit call right after createJob; the kit fetches the criteria document from the judge and checks it before sending. When the hosted judge settles the ERC-8183 job, it attests on your registry in the same run, and GET /api/erc8412?jobId=N rebuilds the package from chain data.
• Jobs 186779 (Satisfied, 0x50) and 186780 (NotSatisfied, 0x40) went through exactly that path on the hosted judge. Both packages are valid under your verify.py with nothing unchecked: docs/erc8412.

One implementation hazard, since you asked for adversarial cases. A review of our code found that the judge's criteria hashing, which rebuilt objects in JavaScript by assignment, silently dropped a member named __proto__ (the assignment hits the prototype setter), so two different criteria sets could share one hash. It is fixed (null-prototype objects, and such criteria are now refused), and no record ever used one. For this ERC: your reference jcs keeps such a member, and so does ours, byte for byte, but a JS port that copies documents with Object.assign drops it and gets a different digest (spread keeps it). A vector with an otherwise valid document carrying a __proto__ member, say inside constraints, would catch that class of port.

Also, the judge's whole-word matching is now pinned to a Unicode 17.0.0 table instead of the runtime's, so a contains obligation's MET or UNMET no longer depends on the Node version.

Still testnet only, and both jobs are my own test jobs.

[babyblueviper1]

@richard7463 both follow-ups are in a PR against your branch: richard7463#1 (assets only, no spec or contract change).

• E10 as defence in depth for resolutions. attest-after-expired-unresolved already fails only when E7 and E10 are both off. The mutation table now runs that pair and requires that case to catch it. 17 mutations, all caught.
• @vijaygopalbalasa's __proto__ case is now a valid package, valid-proto-member. I checked it in Node on the vector itself: Object.assign and key-by-key copies drop the member (11 keys become 10), while JSON.parse and spread keep it. So a port that copies with assignment misreads a valid package as an O5 digest mismatch.

run_all.py passes.