Skip to content

Commit eba4201

Browse files
committed
(spec) integration test negative path migration spec
1 parent c80ac7b commit eba4201

3 files changed

Lines changed: 136 additions & 1 deletion

File tree

‎config/skills_repo.lock.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
"branch": "main",
33
"head_commit": "3bf2d37a5bd9fd665bb7d42cee5f8ba67c64d01d",
44
"origin_url": "https://github.com/MRiabov/Problemologist-skills",
5-
"recorded_at_utc": "2026-03-26T15:34:14Z",
5+
"recorded_at_utc": "2026-03-26T15:34:45Z",
66
"skills_repo_path": "skills"
77
}
Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Integration Test Negative-Path Migration Matrix
2+
3+
This document maps current integration coverage to the negative-path split that
4+
`specs/integration-tests.md` is converging toward.
5+
6+
Heuristic used here:
7+
8+
- `@pytest.mark.allow_backend_errors(...)` is treated as a strong negative-path
9+
signal unless the test is clearly a success-path contract or a special
10+
boundary case.
11+
- `INT-xxx` stays success-oriented.
12+
- `INT-NEG-###` is the target for deterministic rejection, refusal, fail-closed,
13+
and crash-containment coverage.
14+
15+
The suggested `INT-NEG` numbers below are provisional and ordered to minimize
16+
churn when the suite is split.
17+
18+
## Recommended migrations
19+
20+
### `tests/integration/architecture_p0/test_planner_gates.py`
21+
22+
| Proposed ID | Current test | Why it moves |
23+
| -- | -- | -- |
24+
| `INT-NEG-001` | `test_int_005_mandatory_artifacts_gate` | Mandatory artifact rejection is fail-closed coverage, not a success path. |
25+
| `INT-NEG-002` | `test_int_006_plan_structure_validation` | Invalid `plan.md` structure is a deterministic negative case. |
26+
| `INT-NEG-003` | `test_int_007_todo_integrity` | Invalid todo markup and incomplete items are rejection coverage. |
27+
| `INT-NEG-004` | `test_int_008_objectives_validation` | Template placeholders, schema drift, and extra fields are negative-path validation. |
28+
| `INT-NEG-005` | `test_int_009_cost_estimation_validation` | Placeholder/schema failures belong in the negative suite. |
29+
| `INT-NEG-006` | `test_int_011_planner_caps_enforcement` | Planner-cap overflow is explicit fail-closed validation. |
30+
| `INT-NEG-007` | `test_int_015_engineer_handover_immutability` | Handover mutation detection is an integrity rejection. |
31+
| `INT-NEG-008` | `test_int_018_submit_handoff_rejects_forbidden_environment_drilling` | Forbidden drilling is a direct handoff refusal. |
32+
| `INT-NEG-009` | `test_int_010_submit_handoff_rejects_missing_benchmark_drilling_cost` | Missing drilling-cost accounting is a rejection path. |
33+
| `INT-NEG-010` | `test_int_023_submit_handoff_rejects_forbidden_benchmark_attachment_joint` | Non-attachable benchmark joints are a fail-closed gate. |
34+
| `INT-NEG-011` | `test_int_010_planner_pricing_script_integration` | Over-cap pricing should live with negative pricing gates. |
35+
| `INT-NEG-012` | `test_int_010_handoff_rejects_low_quantity_that_only_passes_at_volume` | Requested-quantity manufacturability failure is negative-path coverage. |
36+
| `INT-NEG-013` | `test_int_018_validate_and_price_integration_gate` | Missing/stale validate-and-price state is a gating failure. |
37+
38+
### `tests/integration/architecture_p0/test_node_entry_validation.py`
39+
40+
| Proposed ID | Current test | Why it moves |
41+
| -- | -- | -- |
42+
| `INT-NEG-014` | `test_int_184_seeded_workspace_rejects_mismatched_benchmark_caps` | Seeded workspace cap mismatch is a validation failure. |
43+
| `INT-NEG-015` | `test_int_184_engineer_fail_fast_and_skip_target_node` | Invalid entry must fail fast and skip target-node execution. |
44+
45+
### `tests/integration/architecture_p0/test_cots_reviewer.py`
46+
47+
| Proposed ID | Current test | Why it moves |
48+
| -- | -- | -- |
49+
| `INT-NEG-016` | `test_int_016_reviewer_decision_schema_gate` | Malformed review decisions are schema rejection coverage. |
50+
| `INT-NEG-017` | `test_int_017_plan_refusal_loop` | Plan refusal and FAILED-state propagation are negative-path behavior. |
51+
52+
### `tests/integration/architecture_p0/test_int_187_heavy_crash_containment.py`
53+
54+
| Proposed ID | Current test | Why it moves |
55+
| -- | -- | -- |
56+
| `INT-NEG-018` | `test_int_187_heavy_worker_crash_containment_boundary` | Child-process crash containment is explicitly fail-closed behavior. |
57+
58+
### `tests/electronics/test_integration_electronics.py`
59+
60+
| Proposed ID | Current test | Why it moves |
61+
| -- | -- | -- |
62+
| `INT-NEG-019` | `test_int_120_circuit_validation_pre_gate` | Circuit-validation failure should be isolated as a negative case. |
63+
| `INT-NEG-020` | `test_int_121_short_circuit_detection` | Short-circuit detection is a rejection path. |
64+
| `INT-NEG-021` | `test_int_122_overcurrent_supply_detection` | Overcurrent detection is a rejection path. |
65+
| `INT-NEG-022` | `test_int_124_open_circuit_detection` | Floating-node detection is a rejection path. |
66+
| `INT-NEG-023` | `test_int_126_wire_tear_failure` | Wire-tear failure is negative-path simulation coverage. |
67+
68+
This file is lower-confidence as a migration target because it still relies on
69+
internal imports and `monkeypatch`; the negative rows are still valid
70+
classifications, but the preferred follow-up is to rewrite the coverage as
71+
HTTP-only black-box integration.
72+
73+
### `tests/integration/architecture_p0/test_int_008_objectives_validation.py`
74+
75+
| Proposed ID | Current test | Why it moves |
76+
| -- | -- | -- |
77+
| `INT-NEG-024` | `test_int_008_objectives_semantic_validation_rejects_goal_forbid_overlap` | Objective overlap is a deterministic contradiction. |
78+
| `INT-NEG-025` | `test_int_008_objectives_semantic_validation_rejects_runtime_envelope_forbid_zone_collision` | Runtime envelope intersecting a forbid zone is fail-closed validation. |
79+
| `INT-NEG-026` | `test_int_008_objectives_semantic_validation_rejects_goal_zone_outside_build_zone` | Goal-zone placement outside build zone is rejection coverage. |
80+
| `INT-NEG-027` | `test_int_008_objectives_semantic_validation_rejects_build_zone_outside_simulation_bounds` | Build-zone outside simulation bounds is a negative gate. |
81+
| `INT-NEG-028` | `test_int_008_objectives_semantic_validation_rejects_runtime_envelope_exceeding_build_zone` | Runtime envelope overflow is a fail-closed objective check. |
82+
| `INT-NEG-029` | `test_int_008_objectives_semantic_validation_rejects_negative_runtime_jitter_and_radius` | Negative motion ranges are invalid inputs. |
83+
| `INT-NEG-030` | `test_int_008_attachment_policy_rejects_legacy_attachment_method` | Legacy attachment methods are explicit rejection coverage. |
84+
| `INT-NEG-031` | `test_int_008_requires_non_empty_benchmark_parts` | Missing benchmark parts are a required-contract failure. |
85+
| `INT-NEG-032` | `test_int_008_submit_requires_explicit_reviewer_stage` | Omitted reviewer stage is a direct submit-time failure. |
86+
| `INT-NEG-033` | `test_int_008_missing_interaction_permission_fails_closed_at_handoff` | Missing interaction permission is a fail-closed handoff gate. |
87+
| `INT-NEG-034` | `test_int_008_objectives_semantic_validation_rejects_runtime_envelope_forbid_collision` | Runtime-envelope collisions are rejection coverage. |
88+
| `INT-NEG-035` | `test_int_008_objectives_semantic_validation_rejects_negative_runtime_ranges` | Negative runtime ranges are invalid objective values. |
89+
| `INT-NEG-036` | `test_int_008_drilling_contract_rejects_exceeding_hole_count` | Drill-policy overflow is explicit negative-path coverage. |
90+
| `INT-NEG-037` | `test_int_008_drilling_contract_rejects_diameter_out_of_range` | Out-of-range drill diameter is a rejection case. |
91+
| `INT-NEG-038` | `test_int_008_drilling_contract_rejects_depth_exceeding_max` | Excessive drill depth is a rejection case. |
92+
| `INT-NEG-039` | `test_int_008_no_attachment_policy_defaults_to_non_drillable` | Missing attachment policy must fail closed by default. |
93+
94+
### `tests/integration/architecture_p1/test_benchmark_workflow.py`
95+
96+
| Proposed ID | Current test | Why it moves |
97+
| -- | -- | -- |
98+
| `INT-NEG-040` | `test_int_200_benchmark_workflow_rejects_hidden_motion_handoff` | Hidden benchmark motion is a fail-closed refusal. |
99+
| `INT-NEG-041` | `test_int_202_benchmark_workflow_rejects_unsupported_motion_handoff` | Unsupported motion tokens are rejection coverage. |
100+
| `INT-NEG-042` | `test_benchmark_request_validation_rejects_invalid_objectives` | Invalid request objectives belong in the negative suite. |
101+
102+
### `tests/integration/frontend/p0/test_int_205.py`
103+
104+
| Proposed ID | Current test | Why it moves |
105+
| -- | -- | -- |
106+
| `INT-NEG-043` | `test_int_205_failed_engineer_retry_revises_same_benchmark` | Failed-run retry lineage is a negative-path UX contract. |
107+
108+
## Keep in `INT-xxx`
109+
110+
These are the main success-path or special-boundary exceptions that should stay
111+
out of the negative bucket for now:
112+
113+
- `tests/integration/frontend/p0/test_solution_evidence.py::test_int_189_engineer_run_defaults_to_solution_evidence`
114+
- `tests/integration/architecture_p0/test_planner_gates.py::test_int_005_engineer_planner_flow_emits_submit_plan_trace`
115+
- `tests/integration/architecture_p0/test_planner_gates.py::test_int_113_electronics_planner_flow_emits_submit_plan_trace`
116+
- `tests/integration/architecture_p0/test_planner_gates.py::test_int_114_benchmark_planner_flow_emits_submit_plan_trace`
117+
- `tests/integration/architecture_p0/test_planner_gates.py::test_int_204_benchmark_plan_reviewer_inspects_latest_revision_renders_before_approval`
118+
- `tests/integration/architecture_p0/test_planner_gates.py::test_int_010_validate_and_price_adds_benchmark_drilling_cost`
119+
- `tests/integration/architecture_p0/test_planner_gates.py::test_int_010_submit_handoff_accepts_cheaper_workspace_drilling_override`
120+
- `tests/integration/architecture_p0/test_planner_gates.py::test_int_019_single_part_benchmark_submit_succeeds_without_cost_gate`
121+
- `tests/integration/architecture_p0/test_planner_gates.py::test_int_018_benchmark_submit_accepts_yaml_motion_without_literal_tokens`
122+
- `tests/integration/architecture_p0/test_int_018_multipart_dfm_gate.py::test_int_018_simulate_does_not_dfm_reject_simple_multipart_benchmark`
123+
- `tests/integration/architecture_p0/test_int_018_multipart_dfm_gate.py::test_int_018_simulation_bounds_ignore_fixed_benchmark_fixtures`
124+
- `tests/integration/architecture_p0/test_missing_p0.py::test_int_005_trace_realtime_broadcast`
125+
126+
## Special handling
127+
128+
- `tests/integration/architecture_p1/test_api_fuzzing.py::test_api_fuzzing` is
129+
negative-adjacent fuzzing, but it is a broad schema sweep rather than a single
130+
deterministic rejection scenario. Keep it separate unless the suite gets a
131+
dedicated fuzz bucket or a dedicated `INT-NEG` policy for generative probing.
132+
- The electronics file still uses internal imports and `monkeypatch`; if that
133+
file is preserved, the negative rows above should move, but the long-term
134+
cleanup is to replace those internals with HTTP-only black-box integration.

‎tests/integration/mock_responses/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ Integration mock scenarios live here as one file per `INT-###` case.
66
- `content_file` paths resolve relative to this directory.
77
- The loader fails closed on invalid IDs, duplicate IDs, missing fixture files, or invalid transcript schema.
88
- Use `scripts/normalize_integration_mock_responses.py` to normalize deterministic derived fields in fixture YAMLs before editing them by hand.
9+
- Use `scripts/validate_integration_mock_response_preflight.py` to run transcript/schema validation plus node-entry preflight over the mock-response corpus.

0 commit comments

Comments
 (0)