Skip to content

Commit 10ce971

Browse files
committed
Enforce corpus resource ceilings and record recoverable session accounting
1 parent a28a53b commit 10ce971

4 files changed

Lines changed: 283 additions & 49 deletions

File tree

‎config/protocol-v2.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"schema": "caissa-experiment-protocol-v2",
3+
"purpose": "Prospective full campaign; earlier diagnostics and protocol deviations retained",
4+
"network": {"format": 2, "feature": 1, "width": 64, "dense": [32,16], "scale": 1024, "clip": 2047, "maximum_float_error_cp": 32, "probes": 100000, "probe_seed": 86028157, "neural_weight": 10},
5+
"training": {"seeds": [17,41,73], "steps": 20000, "batch": 2048, "threads": 2, "device": "cpu", "lr": 0.001, "outcome_weight": 0.5, "checkpoint_every": 250},
6+
"data": {"pairs": 200000, "teacher_nodes": 500, "workers": 8, "maximum_output_gib": 40, "minimum_free_gib": 10, "prospective_resume_wall_seconds": 43200, "minimum_training_inputs": 10000000, "minimum_completed_games": 100000, "functional_input_screening": true},
7+
"matches": {"milliseconds": 20, "threads": 1, "hash_mib": 16, "warmup_nodes": 1, "transport_allowance_ms": 150, "maximum_plies": 512, "neural_pairs": 2000, "external_pairs": [334,333,333], "external_settings": ["Stockfish 19 Skill Level 0", "Stockfish 19 UCI_Elo 2000", "Stockfish 19 unrestricted"], "opening_file": "config/strength-openings.json", "bootstrap_resamples": 20000, "bootstrap_seed": 104729, "confidence": 0.95, "minimum_neural_gain_elo": 50, "minimum_neural_lower_elo_exclusive": 0},
8+
"selection": "Seed/checkpoint selected on validation only; test unopened until selection. Blend/architecture frozen by this prospective protocol, never chosen from full-match outcomes.",
9+
"historical_deviation": "v1 generation wall ceiling was not enforced; historical CPU hours are unavailable. This protocol does not retroactively pass v1. Future session wall/monotonic/child CPU times are recorded independently."
10+
}

‎docs/EXPERIMENT_PLAN.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,7 @@ Match time controls and final width must be frozen after speed pilots in a subse
4343
Network arithmetic follows `docs/notes/quantization.md`. Diagnostic untrained maximum float/integer error was 46.1 cp. That observation does not freeze a trained-network tolerance. A justified trained-model tolerance must precede its held-out acceptance run. Record p50/p95/p99 command latency, evaluator ns/call, nodes/s, TT bytes, depth and cancellation for scalar and native kernels. UCI stop p99 must be below 50 ms under supported load. Race-instrumented execution is required separately from locking design review.
4444

4545
Independent verification must reject rehashed illegal PVs, swapped colors/results, wrong network outputs and invalid draw claims, plus missing artifacts, source mismatch and corrupt bytes. No saved pass boolean can substitute for these subprocess checks.
46+
47+
## Later version
48+
49+
[Prospective protocol v2](EXPERIMENT_PLAN_V2.md) records the unenforced v1 resource ceiling and freezes the screened-input, q1024, width, mixture and match settings for future full runs. Earlier diagnostics retain their original identities and limitations.

‎docs/EXPERIMENT_PLAN_V2.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Prospective protocol v2
2+
3+
This version governs new full training, numerical and strength runs. [The machine-readable protocol](../config/protocol-v2.json) is an input to source/evidence identity. Earlier diagnostics remain diagnostics. The original corpus teacher and its numerical labels stay unchanged.
4+
5+
The v1 eight-hour generation ceiling was not enforced and elapsed time exceeded it. Historical CPU hours were not captured. Wall duration and monotonic process duration differ on this host. That missing historical measurement cannot be reconstructed from a linear estimate. New sessions record Unix wall time, monotonic duration and reaped-child CPU seconds separately. A started session without a finish record remains incomplete timing evidence.
6+
7+
Generation continues toward the original fixed 200000 opening pairs, with eight workers, a prospective twelve-hour wall ceiling for the resumed session, at most 40 GiB of pair journals and at least 10 GiB free disk. A bounded eight-worker pilot completed 200 pairs in 25.13 seconds while other generation was active; this is a capacity observation under contention, not an isolated scaling claim. A budget stop retains completed pairs and returns nonzero. A smaller corpus cannot satisfy the full data requirement.
8+
9+
The active transformer width is 64 with dense stages 32 and 16, q1024 and activation clip 2047. Diagnostic width 32 and 128 trials had worse validation loss and the wider network cost more per inference. The v2 arithmetic reduced the 100000-probe observed maximum float error from the coarse format to 11.37 cp. Freeze a 32 cp maximum for a new independently seeded 100000-probe final-model set, allowing approximately 2.8 times the observed diagnostic maximum. This is an empirical acceptance tolerance, not a universal mathematical error bound. A failure stays a failure. Scalar/native/reference integer agreement remains exact.
10+
11+
Pure neural evaluation lost badly. A diagnostic ten-percent mixture scored 44W/16D/36L at equal nodes, estimated +29.02 Elo with interval [-58.45,120.41]. Weights 25 and 50 scored worse. Freeze weight 10 for the prospective neural-contribution hypothesis. Its full campaign must still exceed the unchanged classical evaluator by estimated 50 Elo with lower 95% limit above zero. The small pilot does not establish that effect, and equal-time overhead can erase an equal-node gain. No baseline evaluation term is removed or weakened.
12+
13+
Train seeds 17, 41 and 73 from independent random initialization on the fully screened v2 dataset with identical 20000-update, batch-2048 CPU budgets. Select saved checkpoints and then the seed by validation loss only. Keep all seeds. Aggregate repeated same-input targets before training and quarantine cross-split input collisions. The frozen study and 1000-family strength opening manifests must be excluded before any full training.
14+
15+
The decisive neural campaign uses 2000 reversed-color pairs at 20 ms/move, one thread and 16 MiB per engine, with a one-node warmup and 150 ms transport allowance. It uses the same source executable for both evaluations. The external campaign totals 1000 pairs across the three pinned Stockfish 19 settings in the machine-readable protocol. It compares the same final candidate with weak, intermediate and unrestricted play. Only one external engine codebase is represented; disclose that scope. Executable hashes, actual options and all failures belong to the result.
16+
17+
The statistical unit is an opening family containing reversed-color pairs. Bootstrap 20000 times with seed 104729 and retain the degenerate-sample bound guard. Completion is fixed-sample; there is no favorable early stopping. Report censored endpoints and do not infer absolute Elo from an opponent setting. The final candidate remains classical if a neural mixture has not demonstrated the required contribution.
18+
19+
Independent full verification must enforce counts, lineage, bytes, source identities, numerical checks and complete raw match replay. This protocol is a hypothesis, not a pass receipt.

0 commit comments

Comments
 (0)