Skip to content

Commit ca9939b

Browse files
committed
Stop data acquisition on verified screened-input counts
1 parent 11bf8b5 commit ca9939b

8 files changed

Lines changed: 382 additions & 9 deletions

File tree

‎config/data-generation-v4.json‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
{
2+
"schema": "caissa-data-protocol-v4",
3+
"status": "prospective acquisition amendment before full training or decisive matches",
4+
"supersedes": "config/data-generation-v3.json acquisition stopping only",
5+
"teacher_source_commit": "9ad1d9cbe18de3596d55b1ae209cb4317d74600b",
6+
"teacher_sha256": "cd33e7f3f925bb522569c34283cc9f20c710d9e1bc50760bda62ccacbf2994ed",
7+
"policy": "searched-v2",
8+
"nodes": 2000,
9+
"maximum_pairs": 200000,
10+
"workers": 8,
11+
"seed": 271828,
12+
"maximum_plies": 512,
13+
"wall_seconds_across_sessions": 86400,
14+
"child_cpu_scheduling_seconds_across_sessions": 180000,
15+
"maximum_pair_journal_gib": 40,
16+
"minimum_free_gib": 10,
17+
"target_distinct_training_inputs": 10100000,
18+
"minimum_distinct_training_inputs": 10000000,
19+
"minimum_completed_games": 100000,
20+
"minimum_completed_search_fraction": 0.95,
21+
"stop_rule": "Stop new pairs when the committed preparation index reaches the target, at least 100000 completed games and nonempty validation/test splits. Drain at most eight in-flight pairs. Require preparation to consume every committed pair and confirm at least 10000000 actual distinct training inputs before declaring acquisition complete. If later collisions invalidate the minimum, acquisition is incomplete.",
22+
"confirmation_timeout_seconds": 10800,
23+
"resume_rule": "Keep the same corpus CONFIG, executable, seed, search policy, node budget and maximum pair request. Preserve the original session wall origin and sum actual child CPU over every gracefully completed session. Refuse strict CPU-budget resumption after missing session accounting.",
24+
"independence": "Preparation counts are operational stop signals, not final evidence. A separate verifier reconstructs every original game, split, feature, target and lineage and enforces the hard floors again.",
25+
"retention": "Retain every completed pair, both colors, all outcomes, prior protocol and session journals. Do not delete earlier pairs or reset cumulative budgets.",
26+
"strength_rule": "This data acquisition amendment changes no strength sample or neural-gain threshold. All decisive strength matches remain fixed sample, separately frozen before execution.",
27+
"heldout_studies": "config/studies.json",
28+
"heldout_openings": "config/strength-openings.json"
29+
}

‎docs/BUILD_STATUS.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,14 @@ The native rules, classical evaluation, PVS search, bounded UCI, Chess960, PGN v
1212
| G04 Protocol | 10000 adversarial sequences, stop p99 below 50 ms; unread-output bound test | Final supported-load campaign |
1313
| G05 Search | Independent tiny minimax, bounds and mate tests | Full heuristic strength ablations |
1414
| G06 Network | 100000 v2 probes bit-identical; maximum observed float error 11.37 cp | Freeze and test final-model error bound |
15-
| G07 Learning | Original corpus generating; three 20000-update diagnostic seed runs completed | 10 million distinct training inputs after screening; three full runs |
15+
| G07 Learning | Three absolute-evaluator diagnostic seeds completed; residual pilot and stronger searched corpus execute | 10 million distinct training inputs after screening; three full runs |
1616
| G08 Neural contribution | Failed diagnostic: 4W/8D/84L against classical | Required 4000-game full positive contribution |
1717
| G09 External strength | Warm short diagnostics include 0W/0D/48L against unrestricted Stockfish 19 | Required 2000-game campaign across three settings/opponents |
1818
| G10 Parallel systems | Root sharing, cancellation and race-instrumented native checks | Frozen 1/2/4/8 profiles and strength comparisons |
1919
| G11 Recovery | SIGKILL training/corpus/arena recovery tests executed | Source-bound replay of recovery records |
2020
| G12 Browser | Live import, analysis, variation edit and export exercised | Complete fresh/constrained/keyboard campaign |
2121
| G13 Performance | Evaluator and protocol timings recorded | Complete frozen scalar/optimized resource profiles |
22-
| G14 Semantic verifier | Rehashed illegal PV, swapped colors, wrong inference and false draw rejected | Full campaign schema and independent checks |
22+
| G14 Semantic verifier | Rehashed illegal PV, swapped colors, wrong inference and false draw rejected; separate full-data and strength checkers implemented | Remaining full campaign schema and independent checks |
2323
| G15 Reproduction | Clean source diagnostic passed; Linux/macOS CI passes | Full saved-model/game/report reproduction |
2424
| G16 Release | Blocked by incomplete and failed full gates | All required gates and final identity audit |
2525

@@ -29,4 +29,6 @@ The initial 8-hour generation ceiling was not enforced and elapsed wall time exc
2929

3030
The larger diagnostic preparation has 2181990 training positions from 65272 completed games. Its older split predates functional-input reflection screening, so it cannot satisfy the full data gate. A later v2 preparation independently replays games, quarantines cross-split functional equivalents and excludes frozen study/opening inputs.
3131

32+
The legacy 500-node data generator was stopped after a quality audit found that only 18.53% of a 400-game pilot's samples completed a search depth. The replacement original teacher uses 2000 nodes and seeded root ordering. Its 400-game pilot completed a depth for 98.30% of samples and passed complete independent dataset reconstruction. The new full corpus is separate. [The counted acquisition protocol](../config/data-generation-v4.json) preserves cumulative budgets and requires at least ten million actual screened training inputs. Generation, preparation and small diagnostic recovery tests do not yet pass the full learning gate.
33+
3234
The saved `models/diagnostic-v2.cai` is original diagnostic data, not a release-strength model. No full pass, final Elo or Stockfish parity claim follows from the current measurements.

‎docs/DATA_PROTOCOL_V3.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,11 @@ The current searched policy uses the original faster search, one principal varia
99
All completed games and both result colors remain. Incomplete games have no manufactured outcome. Depth-zero samples are disclosed, not quietly filtered to improve the result. The full observed searched fraction must exceed 95%. Actual screened training inputs must exceed ten million and completed games one hundred thousand. A shortfall or exhausted budget remains incomplete.
1010

1111
Generation writes complete pair journals atomically. Preparation follows the fixed pair range, independently replays it, screens frozen studies/openings, aggregates identical neural inputs and quarantines cross-split collisions. Its transactions can recover after interruption. Numerical/model selection and strength settings must be frozen separately before the new full training and decisive matches; this data protocol does not certify a model or playing strength.
12+
13+
## Counted acquisition amendment
14+
15+
[Protocol v4](../config/data-generation-v4.json) changes acquisition stopping before full training or decisive matches. Two hundred thousand remains the maximum pair request. Preparation now reports exact distinct input counts online, including decreases when later games create cross-split collisions. Generation stops scheduling at 10.1 million training inputs and at least 100000 completed games, drains bounded work, and waits for the preparation transaction to consume every committed pair. Completion requires at least ten million actual training inputs at that final boundary. The separate dataset verifier reconstructs and checks that boundary and all hard floors again.
16+
17+
This avoids spending the remaining budget on an arbitrary pair count after enough usable data exists. It does not stop a strength match early. It preserves the original executable, game policy, maximum request, cumulative wall origin, measured child CPU, disk limits and every completed game. The earlier fixed-pair protocol and session records remain. A timeout, exhausted budget, missing accounting or invalidated final count is incomplete.
18+
19+
A diagnostic interrupted both processes, resumed the same corpus and SQLite transaction, retained every completed pair byte, preserved cumulative budgets and independently reconstructed the final dataset. The first recovery check exposed a microsecond change in the stored wall origin; resumption now carries forward the exact original origin. These small tests do not satisfy the full data gate.

‎docs/RECOVERY.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,11 @@ Sources: [corpus](../training/corpus.py), [trainer](../training/train.py), [aren
3131
`training.prepare --follow` consumes the fixed pair range while the original corpus writer is active. It never reads a temporary pair. The source SHA inventory, identity aggregates, contributor lineage, counts and histograms commit in one SQLite transaction every 100 pairs. A stopped corpus writer with a missing pair is an error, not successful preparation. The final NumPy arrays are exported only after the full requested range has been consumed.
3232

3333
After interruption, rerun the same preparation command with `--resume`. Completed source bytes are rechecked before reuse, and changed corpus configuration or held-out manifests are rejected. An interrupted array export is regenerated from the committed replay index. `DATASET.json` appears only after complete export and hashes; a completed dataset is immutable. A forced SIGKILL after the first 100-pair transaction reproduced every array and lineage byte and all counts/histograms from an uninterrupted 200-pair preparation.
34+
35+
## Screened acquisition and stalls
36+
37+
For a protocol with an explicit screened acquisition target, use the same `--stop-prepared`, `--target-training-inputs`, `--minimum-training-inputs` and `--minimum-completed-games` settings on resumption. The maximum pair request remains immutable. Online counts are transactionally updated when inputs become quarantined. A stopped target run is complete only after all committed pair files are replayed and its final proof agrees with the prepared index. The full verifier independently enforces ten million training inputs and one hundred thousand completed games.
38+
39+
Stop preparation first, then send SIGTERM to the producer and allow its bounded in-flight children to finish. Inspect `INTERRUPTED.json` and the matching finished `SESSIONS.jsonl` record before resuming. Restart the producer, then preparation with `--follow --resume`. A missing finished CPU record blocks strict budget resumption. Killing a writer does not authorize resetting its time or CPU limit.
40+
41+
Monitor `PROGRESS.json`, `PREPARE_PROGRESS.json`, checkpoint pointers and recent errors. A live PID alone proves no useful work. Long exports and independent replay have different units from pair generation; inspect the active phase before declaring a stall. Preserve failure logs, use bounded recovery attempts, and switch to independent work while dependencies progress. Repeating the same failed command indefinitely is not recovery.

‎tests/acquisition_target.py‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
"""End a diagnostic corpus on real screened counts, then independently replay it."""
2+
3+
import argparse, json, os, subprocess, sys, time
4+
from pathlib import Path
5+
from verification.dataset import verify_dataset
6+
7+
8+
def main():
9+
parser = argparse.ArgumentParser()
10+
parser.add_argument("--engine", default="build/caissa")
11+
parser.add_argument("--output", type=Path, required=True)
12+
parser.add_argument("--interrupt", action="store_true")
13+
args = parser.parse_args()
14+
args.output.mkdir(parents=True, exist_ok=False)
15+
corpus = args.output / "corpus"
16+
dataset = args.output / "prepared"
17+
producer = [
18+
sys.executable,
19+
"-m",
20+
"training.corpus",
21+
"--engine",
22+
args.engine,
23+
"--pairs",
24+
"5000",
25+
"--nodes",
26+
"500",
27+
"--workers",
28+
"2",
29+
"--policy",
30+
"searched-v2",
31+
"--max-seconds",
32+
"300",
33+
"--stop-prepared",
34+
str(dataset / "PREPARE_PROGRESS.json"),
35+
"--target-training-inputs",
36+
"5000",
37+
"--minimum-training-inputs",
38+
"4000",
39+
"--minimum-completed-games",
40+
"1",
41+
"--output",
42+
str(corpus),
43+
]
44+
preparation = [
45+
sys.executable,
46+
"-m",
47+
"training.prepare",
48+
str(corpus),
49+
str(dataset),
50+
"--follow",
51+
]
52+
processes = []
53+
preserved = {}
54+
with (
55+
(args.output / "producer.log").open("w") as one,
56+
(args.output / "preparation.log").open("w") as two,
57+
):
58+
try:
59+
source = subprocess.Popen(producer, stdout=one, stderr=subprocess.STDOUT)
60+
processes.append(source)
61+
deadline = time.monotonic() + 30
62+
while not (corpus / "CONFIG.json").exists():
63+
if source.poll() is not None:
64+
raise ValueError("producer exited before initialization")
65+
if time.monotonic() > deadline:
66+
raise TimeoutError("producer configuration missing")
67+
time.sleep(0.02)
68+
prepare = subprocess.Popen(
69+
preparation, stdout=two, stderr=subprocess.STDOUT
70+
)
71+
processes.append(prepare)
72+
if args.interrupt:
73+
deadline = time.monotonic() + 60
74+
while (
75+
len(list(corpus.glob("*/*.json"))) < 8
76+
or not (dataset / "PREPARE_PROGRESS.json").exists()
77+
):
78+
if source.poll() is not None or prepare.poll() is not None:
79+
raise ValueError("acquisition exited before interruption")
80+
if time.monotonic() > deadline:
81+
raise TimeoutError(
82+
"no acquisition checkpoint before interruption"
83+
)
84+
time.sleep(0.02)
85+
# Preparation rolls back its uncommitted tail. The producer drains
86+
# bounded children and records CPU before its replacement starts.
87+
prepare.terminate()
88+
prepare.wait(timeout=10)
89+
source.terminate()
90+
if source.wait(timeout=120) != 2:
91+
raise ValueError(
92+
"graceful acquisition interruption was not recorded"
93+
)
94+
import hashlib
95+
96+
preserved = {
97+
str(path.relative_to(corpus)): hashlib.sha256(
98+
path.read_bytes()
99+
).hexdigest()
100+
for path in corpus.glob("*/*.json")
101+
}
102+
source = subprocess.Popen(
103+
producer, stdout=one, stderr=subprocess.STDOUT
104+
)
105+
processes.append(source)
106+
deadline = time.monotonic() + 30
107+
while len((corpus / "SESSIONS.jsonl").read_text().splitlines()) < 3:
108+
if source.poll() is not None or time.monotonic() > deadline:
109+
raise ValueError("resumed producer did not start")
110+
time.sleep(0.02)
111+
prepare = subprocess.Popen(
112+
preparation + ["--resume"], stdout=two, stderr=subprocess.STDOUT
113+
)
114+
processes.append(prepare)
115+
if source.wait(timeout=300) != 0:
116+
raise ValueError("target producer did not complete")
117+
if prepare.wait(timeout=180) != 0:
118+
raise ValueError("target preparation did not export")
119+
finally:
120+
for process in processes:
121+
if process.poll() is None:
122+
process.terminate()
123+
for process in processes:
124+
try:
125+
process.wait(timeout=10)
126+
except subprocess.TimeoutExpired:
127+
process.kill()
128+
process.wait()
129+
summary = json.loads((corpus / "SUMMARY.json").read_text())
130+
manifest = json.loads((dataset / "DATASET.json").read_text())
131+
assert summary["completion_mode"] == "screened-target" and summary["pairs"] < 5000
132+
assert manifest["splits"]["train"] >= 4000
133+
assert (
134+
summary["completion_proof"]["counts"]["distinct_train_inputs"]
135+
== manifest["splits"]["train"]
136+
)
137+
assert (
138+
len(json.loads((dataset / "SOURCE_PAIRS.json").read_text())) == summary["pairs"]
139+
)
140+
if args.interrupt:
141+
import hashlib
142+
143+
for relative, expected in preserved.items():
144+
assert (
145+
hashlib.sha256((corpus / relative).read_bytes()).hexdigest() == expected
146+
)
147+
sessions = [
148+
json.loads(line)
149+
for line in (corpus / "SESSIONS.jsonl").read_text().splitlines()
150+
]
151+
finished = [event for event in sessions if event["event"] == "finished"]
152+
assert len(finished) == 2
153+
assert (
154+
sessions[2]["previous_child_cpu_seconds"]
155+
== finished[0]["child_cpu_seconds_this_session"]
156+
)
157+
assert (
158+
sessions[2]["campaign_wall_origin_unix"]
159+
== sessions[0]["campaign_wall_origin_unix"]
160+
)
161+
assert summary["campaign_child_cpu_seconds"] == sum(
162+
event["child_cpu_seconds_this_session"] for event in finished
163+
)
164+
verified = verify_dataset(
165+
dataset,
166+
corpus,
167+
Path("config/studies.json"),
168+
Path("config/strength-openings.json"),
169+
)
170+
result = {
171+
"status": "passed",
172+
"profile": "diagnostic",
173+
"pairs": summary["pairs"],
174+
"maximum_pairs": 5000,
175+
"actual_train_inputs": manifest["splits"]["train"],
176+
"source_and_preparation_confirm_same_boundary": True,
177+
"independent_replay": verified["status"],
178+
"resumed_completed_pair_bytes_preserved": len(preserved),
179+
"cumulative_budget_recovery": args.interrupt,
180+
"full_acceptance": False,
181+
}
182+
(args.output / "TARGET.json").write_text(json.dumps(result, indent=2) + "\n")
183+
print(json.dumps(result))
184+
185+
186+
if __name__ == "__main__":
187+
main()

0 commit comments

Comments
 (0)