Skip to content

Commit ec49637

Browse files
committed
Train declared residual mixtures and retain all failed learning-seed audits
1 parent 4951493 commit ec49637

143 files changed

Lines changed: 977 additions & 30 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ An original C++23 chess engine with incremental neural evaluation, a paired-matc
88

99
## Run
1010

11-
Native prerequisites: C++23 compiler, CMake 3.25 or newer and Ninja. The laboratory uses uv, Python 3.12 and Node 24. Apple arm64 and Ubuntu x86-64 builds run in [CI](https://github.com/Aneesh495/caissa-chess/actions/workflows/ci.yml).
11+
Native prerequisites: C++23 compiler, CMake 3.25 or newer and Ninja. The laboratory uses uv, Python 3.12 and Node 24. Apple arm64 and Ubuntu x86-64 builds run in [CI](https://github.com/Aneesh495/caissa/actions/workflows/ci.yml).
1212

1313
```sh
1414
make bootstrap

‎docs/BUILD_STATUS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ The native rules, classical evaluation, PVS search, bounded UCI, Chess960, PGN v
1212
| G04 Protocol | 10000 sequences independently replayed, stop p99 8.47 ms; six rehashed semantic controls rejected; unread-output bound test | Repeat and bind final executable |
1313
| G05 Search | Independent tiny minimax, bounds and mate tests; exhaustive neural cutoff controls | Full heuristic strength ablations |
1414
| G06 Network | 100000 v2 probes bit-identical; maximum observed float error 11.37 cp | Freeze and test final-model error bound |
15-
| G07 Learning | Three absolute-evaluator diagnostic seeds completed; residual pilot and stronger searched corpus execute | 10 million distinct training inputs after screening; three full runs |
15+
| G07 Learning | Original diagnostic seeds complete; all three mixture-aware strong-teacher small pilots overfit and are retained as failed; full searched corpus progresses | 10 million distinct training inputs after screening; three full runs |
1616
| G08 Neural contribution | Failed diagnostic: 4W/8D/84L against classical | Required 4000-game full positive contribution |
1717
| G09 External strength | Warm short diagnostics include 0W/0D/48L against unrestricted Stockfish 19 | Required 2000-game campaign across three settings/opponents |
1818
| G10 Parallel systems | Root sharing, cancellation and race-instrumented native checks | Frozen 1/2/4/8 profiles and strength comparisons |

‎docs/notes/deployment-objective.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Training at the deployed neural percentage
2+
3+
The earlier networks optimized their full neural output and were later mixed at
4+
ten percent during play. `--neural-weight` now lets residual training optimize
5+
and validate the declared mixture directly. Its default of 100 preserves the
6+
existing experiments. Lower percentages require residual format 3 and the
7+
original classical baseline arrays. The new configuration records both the
8+
percentage and its arithmetic contract.
9+
10+
`training/model.py` applies the native full-score clamp before mixing. During
11+
quantized-aware training, straight-through output rounding and signed percentage
12+
truncation retain gradients at the declared percentage. Native evaluation stays
13+
integer and separate. Hidden-layer QAT remains a floating approximation; this
14+
training objective does not replace the final independent numerical-error gate.
15+
The zero residual still produces exactly the original classical baseline.
16+
17+
Twenty-five known output cases covered signed half ties, percentage truncation
18+
and saturation. Autograd checks confirmed gradients at five percentages and
19+
zero gradients after saturation. A separate historical-source comparison ran
20+
200 updates with the old and new default objective: parameters, optimizer,
21+
scheduler, sampler, Python/NumPy/Torch RNG, validation and configuration matched
22+
exactly. Resuming the completed historical budget kept its configuration and
23+
original initialization bytes. This does not claim a new forced-interruption
24+
recovery campaign.
25+
26+
The new three-seed diagnostic used the stronger original teacher's 400-game
27+
pilot: 17654 training inputs and 2563 validation inputs, seeds 17/41/73, 1000 real
28+
updates each, batch 512, CPU with two threads, and four checkpoints per seed.
29+
Test arrays were not opened. All seeds overfit this small sample:
30+
31+
| Seed | Untrained validation loss | Best trained validation loss |
32+
|---|---:|---:|
33+
| 17 | 0.37834248 | 0.38147594 |
34+
| 41 | 0.37834248 | 0.38170146 |
35+
| 73 | 0.37834248 | 0.38182412 |
36+
37+
The independent checker rejected the learning-improvement endpoint. Its first
38+
failure stopped at the first seed. The checker now completes the provenance
39+
audit for every seed and export before returning the same failed endpoint with
40+
all observed losses. It independently evaluated all twelve checkpoints and
41+
verified optimizer/sampler/initial-state lineage. A positive older learning
42+
sample still passed; four rehashed semantic mutations still failed for their
43+
intended reasons. This forensic continuation did not retrain, change selection,
44+
extend the original wall ceiling, or turn a negative result into a pass.
45+
46+
No playing-strength claim follows from these losses. The stronger data and the
47+
failed smaller prototype remain separate from the pending full three-seed
48+
campaign. Classical evaluation remains the default.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"scope": "diagnostic objective and failed-learning forensics; no full acceptance",
3+
"base_commit": "49514939eea6c06840af0d10e1b869812d6cc0d9",
4+
"source_files": {
5+
"training/model.py": "8197f6f564486bbefeb0567f3437a07aaba64d9fc8e9316e8c3afc16747e6b52",
6+
"training/train.py": "7101097995cdbd8a705d0b1bdf20311ccdcdcede73fcd634962013568ce85a1e",
7+
"verification/learning.py": "420e8cc93e17bffc1214d5662f177d5d5dacc9ff9cdc8c2635928df7d556700e",
8+
"tests/deployment_objective.py": "467d60ce672091f2dfe2310810ff4641f77af3fedda9ea6383d2cb2e5f18e017"
9+
},
10+
"full_acceptance": false
11+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"status":"completed forensic checks; learning improvement remains failed","details":null,"seconds_this_campaign":124.9736419589608,"stages_started":3,"utc_unix":1792734684.569779,"full_acceptance":false}
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{"event":"campaign-started","utc_unix":1792734559.596179,"maximum_seconds":852.6205430030823}
2+
{"event":"stage-started","stage":"retain-failed-learning-gate","sequence":1,"command":["/Users/aneeshkrishna/Documents/Codex/2026-10-22/ex/outputs/caissa-chess/.venv/bin/python","-c","import json,subprocess,sys; r=subprocess.run(sys.argv[1:],capture_output=True,text=True,timeout=120); print(r.stdout,end=''); print(r.stderr,end=''); o=json.loads(r.stdout); assert r.returncode==1 and o['reason']=='learned evaluator did not improve held-out loss over initialization' and len(o['observed']['failed_learning_seeds'])==3 and o['observed']['all_checkpoints_independently_evaluated']==12; print(json.dumps({'actual_verifier_exit':r.returncode,'learning_gate':'FAILED','all_seed_provenance':'replayed'}))","/Users/aneeshkrishna/Documents/Codex/2026-10-22/ex/outputs/caissa-chess/.venv/bin/python","-m","verification.learning","--dataset","data/searched-2000nodes-prepared","--runs","runs/deployed-mixture-pilot/seed17","runs/deployed-mixture-pilot/seed41","runs/deployed-mixture-pilot/seed73","--selection","runs/deployed-mixture-pilot/selection","--baseline","runs/deployed-mixture-pilot/baselines","--output","runs/deployed-mixture-forensics/FAILED_LEARNING.json"],"utc_unix":1792734559.596757,"timeout_seconds":130,"maximum_output_bytes":33554432}
3+
{"event":"stage-finished","stage":"retain-failed-learning-gate","sequence":1,"command":["/Users/aneeshkrishna/Documents/Codex/2026-10-22/ex/outputs/caissa-chess/.venv/bin/python","-c","import json,subprocess,sys; r=subprocess.run(sys.argv[1:],capture_output=True,text=True,timeout=120); print(r.stdout,end=''); print(r.stderr,end=''); o=json.loads(r.stdout); assert r.returncode==1 and o['reason']=='learned evaluator did not improve held-out loss over initialization' and len(o['observed']['failed_learning_seeds'])==3 and o['observed']['all_checkpoints_independently_evaluated']==12; print(json.dumps({'actual_verifier_exit':r.returncode,'learning_gate':'FAILED','all_seed_provenance':'replayed'}))","/Users/aneeshkrishna/Documents/Codex/2026-10-22/ex/outputs/caissa-chess/.venv/bin/python","-m","verification.learning","--dataset","data/searched-2000nodes-prepared","--runs","runs/deployed-mixture-pilot/seed17","runs/deployed-mixture-pilot/seed41","runs/deployed-mixture-pilot/seed73","--selection","runs/deployed-mixture-pilot/selection","--baseline","runs/deployed-mixture-pilot/baselines","--output","runs/deployed-mixture-forensics/FAILED_LEARNING.json"],"exit":0,"reason":"completed","utc_start_unix":1792734559.596757,"utc_end_unix":1792734582.5969138,"monotonic_seconds":23.000000333006028,"wall_seconds":23.000158071517944,"reaped_child_cpu_seconds":6.5721680000000005,"retained_output_bytes":3223,"discarded_output_bytes":0,"log":"retain-failed-learning-gate.log","log_sha256":"6c289ad0a7261a98c43cbf1b4b305f37a392b4a155ec6f8198ac1a9b4b414286"}
4+
{"event":"stage-started","stage":"legacy-positive-domain","sequence":2,"command":["/Users/aneeshkrishna/Documents/Codex/2026-10-22/ex/outputs/caissa-chess/.venv/bin/python","-m","verification.learning","--dataset","data/diagnostic-v2-final-screened","--runs","runs/learning-independent-pilot/seed17","runs/learning-independent-pilot/seed41","runs/learning-independent-pilot/seed73","--selection","runs/learning-independent-pilot/selection","--baseline","runs/small-classical-baselines","--output","runs/deployed-mixture-forensics/LEGACY_POSITIVE.json"],"utc_unix":1792734582.5986052,"timeout_seconds":90,"maximum_output_bytes":33554432}
5+
{"event":"stage-finished","stage":"legacy-positive-domain","sequence":2,"command":["/Users/aneeshkrishna/Documents/Codex/2026-10-22/ex/outputs/caissa-chess/.venv/bin/python","-m","verification.learning","--dataset","data/diagnostic-v2-final-screened","--runs","runs/learning-independent-pilot/seed17","runs/learning-independent-pilot/seed41","runs/learning-independent-pilot/seed73","--selection","runs/learning-independent-pilot/selection","--baseline","runs/small-classical-baselines","--output","runs/deployed-mixture-forensics/LEGACY_POSITIVE.json"],"exit":0,"reason":"completed","utc_start_unix":1792734582.5986052,"utc_end_unix":1792734603.2542112,"monotonic_seconds":20.655465667019598,"wall_seconds":20.655606985092163,"reaped_child_cpu_seconds":6.0690029999999995,"retained_output_bytes":2456,"discarded_output_bytes":0,"log":"legacy-positive-domain.log","log_sha256":"e4705ff5faf12d0cf28aba9a3f23d4f355cbcedd01996b21f6587cba0bd21c67"}
6+
{"event":"stage-started","stage":"semantic-controls","sequence":3,"command":["/Users/aneeshkrishna/Documents/Codex/2026-10-22/ex/outputs/caissa-chess/.venv/bin/python","-m","tests.learning_controls","--source","runs/deployed-mixture-pilot","--dataset","data/searched-2000nodes-prepared","--baseline","runs/deployed-mixture-pilot/baselines","--output","runs/deployed-mixture-forensics/negative"],"utc_unix":1792734603.255946,"timeout_seconds":180,"maximum_output_bytes":33554432}
7+
{"event":"stage-finished","stage":"semantic-controls","sequence":3,"command":["/Users/aneeshkrishna/Documents/Codex/2026-10-22/ex/outputs/caissa-chess/.venv/bin/python","-m","tests.learning_controls","--source","runs/deployed-mixture-pilot","--dataset","data/searched-2000nodes-prepared","--baseline","runs/deployed-mixture-pilot/baselines","--output","runs/deployed-mixture-forensics/negative"],"exit":0,"reason":"completed","utc_start_unix":1792734603.255946,"utc_end_unix":1792734684.565202,"monotonic_seconds":81.30870491696987,"wall_seconds":81.30925607681274,"reaped_child_cpu_seconds":24.757027999999995,"retained_output_bytes":3211,"discarded_output_bytes":0,"log":"semantic-controls.log","log_sha256":"fd027bc5d7039bf153b31eca859951b5dd987ff2201925d5e2f7a90f3fe4fb70"}
8+
{"event":"campaign-finished","status":"completed forensic checks; learning improvement remains failed","details":null,"seconds_this_campaign":124.9736419589608,"stages_started":3,"utc_unix":1792734684.569779,"full_acceptance":false}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"scope":"Retain failed held-out-learning endpoint while independently checking all three actual trained seeds, and prove semantic controls still reject. No retraining or new seed selection.","original_campaign":"runs/deployed-mixture-pilot","original_wall_origin_unix":1792734212.214741,"original_total_seconds":1200,"remaining_at_start":852.6205430030823}
Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
{
2+
"status": "rejected",
3+
"reason": "learned evaluator did not improve held-out loss over initialization",
4+
"seconds": 16.3049464590149,
5+
"full_acceptance": false,
6+
"observed": {
7+
"status": "failed",
8+
"profile": "diagnostic-learning-domain",
9+
"seeds": [
10+
17,
11+
41,
12+
73
13+
],
14+
"all_checkpoints_independently_evaluated": 12,
15+
"validation_positions_per_checkpoint": 2563,
16+
"baseline_scores_independently_checked": 6237,
17+
"independent_seed_observations": [
18+
{
19+
"seed": 17,
20+
"recorded_origin": "runs/deployed-mixture-pilot/seed17",
21+
"verified_directory": "runs/deployed-mixture-pilot/seed17",
22+
"initial_parameter_sha256": "c4d15b9351e3f634c2fdb294faae1848ebd552fcc3b1215bb8472edb7682fcca",
23+
"initial_checkpoint_sha256": "fbdc5be8719d8dafe6ee1519780ee5194477840a750aa1aae8d30b2f754ba501",
24+
"independent_untrained_validation": 0.37834247987436864,
25+
"independent_checkpoint_validation": [
26+
{
27+
"step": 250,
28+
"loss": 0.3814759472595078
29+
},
30+
{
31+
"step": 500,
32+
"loss": 0.39036114219106643
33+
},
34+
{
35+
"step": 750,
36+
"loss": 0.39180555345488394
37+
},
38+
{
39+
"step": 1000,
40+
"loss": 0.39137125168747144
41+
}
42+
],
43+
"selected_checkpoint_step": 250,
44+
"final_updates": 1000,
45+
"replayed_sampler_values": 512000
46+
},
47+
{
48+
"seed": 41,
49+
"recorded_origin": "runs/deployed-mixture-pilot/seed41",
50+
"verified_directory": "runs/deployed-mixture-pilot/seed41",
51+
"initial_parameter_sha256": "88582fcad13de992379130100e42bceb63f367d587e05ce20893bfbad06351cd",
52+
"initial_checkpoint_sha256": "1678dc3ae234679c89dd3b5b13aac9f7b482e579901a3e5a24f0bc7d22368a71",
53+
"independent_untrained_validation": 0.37834247987436864,
54+
"independent_checkpoint_validation": [
55+
{
56+
"step": 250,
57+
"loss": 0.38170144797952854
58+
},
59+
{
60+
"step": 500,
61+
"loss": 0.3882881156225201
62+
},
63+
{
64+
"step": 750,
65+
"loss": 0.38886578411875505
66+
},
67+
{
68+
"step": 1000,
69+
"loss": 0.38943297989518205
70+
}
71+
],
72+
"selected_checkpoint_step": 250,
73+
"final_updates": 1000,
74+
"replayed_sampler_values": 512000
75+
},
76+
{
77+
"seed": 73,
78+
"recorded_origin": "runs/deployed-mixture-pilot/seed73",
79+
"verified_directory": "runs/deployed-mixture-pilot/seed73",
80+
"initial_parameter_sha256": "fc6819a2ae2bcfd3498398408bbff0a48c95c64dbb4431e3b5a1f4d8cd33d228",
81+
"initial_checkpoint_sha256": "488162964a3bc1415afd7ad7dc0f9311dce793d32f190fce1d04421212a4ba93",
82+
"independent_untrained_validation": 0.37834247987436864,
83+
"independent_checkpoint_validation": [
84+
{
85+
"step": 250,
86+
"loss": 0.3818241171291737
87+
},
88+
{
89+
"step": 500,
90+
"loss": 0.3894911984238269
91+
},
92+
{
93+
"step": 750,
94+
"loss": 0.3919635124147008
95+
},
96+
{
97+
"step": 1000,
98+
"loss": 0.3922218595498063
99+
}
100+
],
101+
"selected_checkpoint_step": 250,
102+
"final_updates": 1000,
103+
"replayed_sampler_values": 512000
104+
}
105+
],
106+
"selected_seed": 17,
107+
"dataset_sha256": "73f27dbfb1ca044bcc86da4387afaaf74e947c3779470bd08fa5f2c8169ac7a0",
108+
"network_sha256": "d08f4bc4c463287f8365183d9a23d3f8949a7b2774382dbf01731765192a2752",
109+
"test_arrays_opened": false,
110+
"full_acceptance": false,
111+
"scope": "learning provenance and selection only; original-game replay and playing-strength gates remain separate",
112+
"failed_learning_seeds": [
113+
{
114+
"seed": 17,
115+
"untrained": 0.37834247987436864,
116+
"best_trained": 0.3814759472595078
117+
},
118+
{
119+
"seed": 41,
120+
"untrained": 0.37834247987436864,
121+
"best_trained": 0.38170144797952854
122+
},
123+
{
124+
"seed": 73,
125+
"untrained": 0.37834247987436864,
126+
"best_trained": 0.3818241171291737
127+
}
128+
]
129+
}
130+
}
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
{
2+
"status": "passed",
3+
"profile": "diagnostic-learning-domain",
4+
"seeds": [
5+
17,
6+
41,
7+
73
8+
],
9+
"all_checkpoints_independently_evaluated": 6,
10+
"validation_positions_per_checkpoint": 2297,
11+
"baseline_scores_independently_checked": 5802,
12+
"independent_seed_observations": [
13+
{
14+
"seed": 17,
15+
"recorded_origin": "runs/learning-independent-pilot/seed17",
16+
"verified_directory": "runs/learning-independent-pilot/seed17",
17+
"initial_parameter_sha256": "c4d15b9351e3f634c2fdb294faae1848ebd552fcc3b1215bb8472edb7682fcca",
18+
"initial_checkpoint_sha256": "438424f9010b91f693404e0435c8584d158252b981bd5852c6be93088913cf58",
19+
"independent_untrained_validation": 0.8006365937568024,
20+
"independent_checkpoint_validation": [
21+
{
22+
"step": 50,
23+
"loss": 0.7635073032802636
24+
},
25+
{
26+
"step": 100,
27+
"loss": 0.7512121138284141
28+
}
29+
],
30+
"selected_checkpoint_step": 100,
31+
"final_updates": 100,
32+
"replayed_sampler_values": 25600
33+
},
34+
{
35+
"seed": 41,
36+
"recorded_origin": "runs/learning-independent-pilot/seed41",
37+
"verified_directory": "runs/learning-independent-pilot/seed41",
38+
"initial_parameter_sha256": "88582fcad13de992379130100e42bceb63f367d587e05ce20893bfbad06351cd",
39+
"initial_checkpoint_sha256": "50998172801fadd45f3590505055cdc0d3d495a2b91d366cc504da88f447ae87",
40+
"independent_untrained_validation": 0.8006365937568024,
41+
"independent_checkpoint_validation": [
42+
{
43+
"step": 50,
44+
"loss": 0.7697393195650917
45+
},
46+
{
47+
"step": 100,
48+
"loss": 0.7544519393714968
49+
}
50+
],
51+
"selected_checkpoint_step": 100,
52+
"final_updates": 100,
53+
"replayed_sampler_values": 25600
54+
},
55+
{
56+
"seed": 73,
57+
"recorded_origin": "runs/learning-independent-pilot/seed73",
58+
"verified_directory": "runs/learning-independent-pilot/seed73",
59+
"initial_parameter_sha256": "fc6819a2ae2bcfd3498398408bbff0a48c95c64dbb4431e3b5a1f4d8cd33d228",
60+
"initial_checkpoint_sha256": "2ff832935de2632def9ce4dd989e7a1f2a12eaf74811dac33e329a485cea8ac0",
61+
"independent_untrained_validation": 0.8006365937568024,
62+
"independent_checkpoint_validation": [
63+
{
64+
"step": 50,
65+
"loss": 0.7699745321045454
66+
},
67+
{
68+
"step": 100,
69+
"loss": 0.7589988392957356
70+
}
71+
],
72+
"selected_checkpoint_step": 100,
73+
"final_updates": 100,
74+
"replayed_sampler_values": 25600
75+
}
76+
],
77+
"selected_seed": 17,
78+
"dataset_sha256": "e8127c2613af6bb37430427232e850f1f6a39eac05552a1d9d3d06cff0b0c2c7",
79+
"network_sha256": "9d529f0c09f3d8629729f6073866e28881a19f5b74d308e4701c2fb89837e68c",
80+
"test_arrays_opened": false,
81+
"full_acceptance": false,
82+
"scope": "learning provenance and selection only; original-game replay and playing-strength gates remain separate",
83+
"seconds": 15.611703666043468
84+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"phase":"semantic-controls","pid":46317,"command":["/Users/aneeshkrishna/Documents/Codex/2026-10-22/ex/outputs/caissa-chess/.venv/bin/python","-m","tests.learning_controls","--source","runs/deployed-mixture-pilot","--dataset","data/searched-2000nodes-prepared","--baseline","runs/deployed-mixture-pilot/baselines","--output","runs/deployed-mixture-forensics/negative"],"seconds_this_stage":80.41563554195454,"seconds_this_campaign":124.07599641697016,"retained_output_bytes":2394,"discarded_output_bytes":0,"stopping_reason":null}

0 commit comments

Comments
 (0)