|
| 1 | +# Integration Test Negative-Path Migration Matrix |
| 2 | + |
| 3 | +This document maps current integration coverage to the negative-path split that |
| 4 | +`specs/integration-tests.md` is converging toward. |
| 5 | + |
| 6 | +Heuristic used here: |
| 7 | + |
| 8 | +- `@pytest.mark.allow_backend_errors(...)` is treated as a strong negative-path |
| 9 | + signal unless the test is clearly a success-path contract or a special |
| 10 | + boundary case. |
| 11 | +- `INT-xxx` stays success-oriented. |
| 12 | +- `INT-NEG-###` is the target for deterministic rejection, refusal, fail-closed, |
| 13 | + and crash-containment coverage. |
| 14 | + |
| 15 | +The suggested `INT-NEG` numbers below are provisional and ordered to minimize |
| 16 | +churn when the suite is split. |
| 17 | + |
| 18 | +## Recommended migrations |
| 19 | + |
| 20 | +### `tests/integration/architecture_p0/test_planner_gates.py` |
| 21 | + |
| 22 | +| Proposed ID | Current test | Why it moves | |
| 23 | +| -- | -- | -- | |
| 24 | +| `INT-NEG-001` | `test_int_005_mandatory_artifacts_gate` | Mandatory artifact rejection is fail-closed coverage, not a success path. | |
| 25 | +| `INT-NEG-002` | `test_int_006_plan_structure_validation` | Invalid `plan.md` structure is a deterministic negative case. | |
| 26 | +| `INT-NEG-003` | `test_int_007_todo_integrity` | Invalid todo markup and incomplete items are rejection coverage. | |
| 27 | +| `INT-NEG-004` | `test_int_008_objectives_validation` | Template placeholders, schema drift, and extra fields are negative-path validation. | |
| 28 | +| `INT-NEG-005` | `test_int_009_cost_estimation_validation` | Placeholder/schema failures belong in the negative suite. | |
| 29 | +| `INT-NEG-006` | `test_int_011_planner_caps_enforcement` | Planner-cap overflow is explicit fail-closed validation. | |
| 30 | +| `INT-NEG-007` | `test_int_015_engineer_handover_immutability` | Handover mutation detection is an integrity rejection. | |
| 31 | +| `INT-NEG-008` | `test_int_018_submit_handoff_rejects_forbidden_environment_drilling` | Forbidden drilling is a direct handoff refusal. | |
| 32 | +| `INT-NEG-009` | `test_int_010_submit_handoff_rejects_missing_benchmark_drilling_cost` | Missing drilling-cost accounting is a rejection path. | |
| 33 | +| `INT-NEG-010` | `test_int_023_submit_handoff_rejects_forbidden_benchmark_attachment_joint` | Non-attachable benchmark joints are a fail-closed gate. | |
| 34 | +| `INT-NEG-011` | `test_int_010_planner_pricing_script_integration` | Over-cap pricing should live with negative pricing gates. | |
| 35 | +| `INT-NEG-012` | `test_int_010_handoff_rejects_low_quantity_that_only_passes_at_volume` | Requested-quantity manufacturability failure is negative-path coverage. | |
| 36 | +| `INT-NEG-013` | `test_int_018_validate_and_price_integration_gate` | Missing/stale validate-and-price state is a gating failure. | |
| 37 | + |
| 38 | +### `tests/integration/architecture_p0/test_node_entry_validation.py` |
| 39 | + |
| 40 | +| Proposed ID | Current test | Why it moves | |
| 41 | +| -- | -- | -- | |
| 42 | +| `INT-NEG-014` | `test_int_184_seeded_workspace_rejects_mismatched_benchmark_caps` | Seeded workspace cap mismatch is a validation failure. | |
| 43 | +| `INT-NEG-015` | `test_int_184_engineer_fail_fast_and_skip_target_node` | Invalid entry must fail fast and skip target-node execution. | |
| 44 | + |
| 45 | +### `tests/integration/architecture_p0/test_cots_reviewer.py` |
| 46 | + |
| 47 | +| Proposed ID | Current test | Why it moves | |
| 48 | +| -- | -- | -- | |
| 49 | +| `INT-NEG-016` | `test_int_016_reviewer_decision_schema_gate` | Malformed review decisions are schema rejection coverage. | |
| 50 | +| `INT-NEG-017` | `test_int_017_plan_refusal_loop` | Plan refusal and FAILED-state propagation are negative-path behavior. | |
| 51 | + |
| 52 | +### `tests/integration/architecture_p0/test_int_187_heavy_crash_containment.py` |
| 53 | + |
| 54 | +| Proposed ID | Current test | Why it moves | |
| 55 | +| -- | -- | -- | |
| 56 | +| `INT-NEG-018` | `test_int_187_heavy_worker_crash_containment_boundary` | Child-process crash containment is explicitly fail-closed behavior. | |
| 57 | + |
| 58 | +### `tests/electronics/test_integration_electronics.py` |
| 59 | + |
| 60 | +| Proposed ID | Current test | Why it moves | |
| 61 | +| -- | -- | -- | |
| 62 | +| `INT-NEG-019` | `test_int_120_circuit_validation_pre_gate` | Circuit-validation failure should be isolated as a negative case. | |
| 63 | +| `INT-NEG-020` | `test_int_121_short_circuit_detection` | Short-circuit detection is a rejection path. | |
| 64 | +| `INT-NEG-021` | `test_int_122_overcurrent_supply_detection` | Overcurrent detection is a rejection path. | |
| 65 | +| `INT-NEG-022` | `test_int_124_open_circuit_detection` | Floating-node detection is a rejection path. | |
| 66 | +| `INT-NEG-023` | `test_int_126_wire_tear_failure` | Wire-tear failure is negative-path simulation coverage. | |
| 67 | + |
| 68 | +This file is lower-confidence as a migration target because it still relies on |
| 69 | +internal imports and `monkeypatch`; the negative rows are still valid |
| 70 | +classifications, but the preferred follow-up is to rewrite the coverage as |
| 71 | +HTTP-only black-box integration. |
| 72 | + |
| 73 | +### `tests/integration/architecture_p0/test_int_008_objectives_validation.py` |
| 74 | + |
| 75 | +| Proposed ID | Current test | Why it moves | |
| 76 | +| -- | -- | -- | |
| 77 | +| `INT-NEG-024` | `test_int_008_objectives_semantic_validation_rejects_goal_forbid_overlap` | Objective overlap is a deterministic contradiction. | |
| 78 | +| `INT-NEG-025` | `test_int_008_objectives_semantic_validation_rejects_runtime_envelope_forbid_zone_collision` | Runtime envelope intersecting a forbid zone is fail-closed validation. | |
| 79 | +| `INT-NEG-026` | `test_int_008_objectives_semantic_validation_rejects_goal_zone_outside_build_zone` | Goal-zone placement outside build zone is rejection coverage. | |
| 80 | +| `INT-NEG-027` | `test_int_008_objectives_semantic_validation_rejects_build_zone_outside_simulation_bounds` | Build-zone outside simulation bounds is a negative gate. | |
| 81 | +| `INT-NEG-028` | `test_int_008_objectives_semantic_validation_rejects_runtime_envelope_exceeding_build_zone` | Runtime envelope overflow is a fail-closed objective check. | |
| 82 | +| `INT-NEG-029` | `test_int_008_objectives_semantic_validation_rejects_negative_runtime_jitter_and_radius` | Negative motion ranges are invalid inputs. | |
| 83 | +| `INT-NEG-030` | `test_int_008_attachment_policy_rejects_legacy_attachment_method` | Legacy attachment methods are explicit rejection coverage. | |
| 84 | +| `INT-NEG-031` | `test_int_008_requires_non_empty_benchmark_parts` | Missing benchmark parts are a required-contract failure. | |
| 85 | +| `INT-NEG-032` | `test_int_008_submit_requires_explicit_reviewer_stage` | Omitted reviewer stage is a direct submit-time failure. | |
| 86 | +| `INT-NEG-033` | `test_int_008_missing_interaction_permission_fails_closed_at_handoff` | Missing interaction permission is a fail-closed handoff gate. | |
| 87 | +| `INT-NEG-034` | `test_int_008_objectives_semantic_validation_rejects_runtime_envelope_forbid_collision` | Runtime-envelope collisions are rejection coverage. | |
| 88 | +| `INT-NEG-035` | `test_int_008_objectives_semantic_validation_rejects_negative_runtime_ranges` | Negative runtime ranges are invalid objective values. | |
| 89 | +| `INT-NEG-036` | `test_int_008_drilling_contract_rejects_exceeding_hole_count` | Drill-policy overflow is explicit negative-path coverage. | |
| 90 | +| `INT-NEG-037` | `test_int_008_drilling_contract_rejects_diameter_out_of_range` | Out-of-range drill diameter is a rejection case. | |
| 91 | +| `INT-NEG-038` | `test_int_008_drilling_contract_rejects_depth_exceeding_max` | Excessive drill depth is a rejection case. | |
| 92 | +| `INT-NEG-039` | `test_int_008_no_attachment_policy_defaults_to_non_drillable` | Missing attachment policy must fail closed by default. | |
| 93 | + |
| 94 | +### `tests/integration/architecture_p1/test_benchmark_workflow.py` |
| 95 | + |
| 96 | +| Proposed ID | Current test | Why it moves | |
| 97 | +| -- | -- | -- | |
| 98 | +| `INT-NEG-040` | `test_int_200_benchmark_workflow_rejects_hidden_motion_handoff` | Hidden benchmark motion is a fail-closed refusal. | |
| 99 | +| `INT-NEG-041` | `test_int_202_benchmark_workflow_rejects_unsupported_motion_handoff` | Unsupported motion tokens are rejection coverage. | |
| 100 | +| `INT-NEG-042` | `test_benchmark_request_validation_rejects_invalid_objectives` | Invalid request objectives belong in the negative suite. | |
| 101 | + |
| 102 | +### `tests/integration/frontend/p0/test_int_205.py` |
| 103 | + |
| 104 | +| Proposed ID | Current test | Why it moves | |
| 105 | +| -- | -- | -- | |
| 106 | +| `INT-NEG-043` | `test_int_205_failed_engineer_retry_revises_same_benchmark` | Failed-run retry lineage is a negative-path UX contract. | |
| 107 | + |
| 108 | +## Keep in `INT-xxx` |
| 109 | + |
| 110 | +These are the main success-path or special-boundary exceptions that should stay |
| 111 | +out of the negative bucket for now: |
| 112 | + |
| 113 | +- `tests/integration/frontend/p0/test_solution_evidence.py::test_int_189_engineer_run_defaults_to_solution_evidence` |
| 114 | +- `tests/integration/architecture_p0/test_planner_gates.py::test_int_005_engineer_planner_flow_emits_submit_plan_trace` |
| 115 | +- `tests/integration/architecture_p0/test_planner_gates.py::test_int_113_electronics_planner_flow_emits_submit_plan_trace` |
| 116 | +- `tests/integration/architecture_p0/test_planner_gates.py::test_int_114_benchmark_planner_flow_emits_submit_plan_trace` |
| 117 | +- `tests/integration/architecture_p0/test_planner_gates.py::test_int_204_benchmark_plan_reviewer_inspects_latest_revision_renders_before_approval` |
| 118 | +- `tests/integration/architecture_p0/test_planner_gates.py::test_int_010_validate_and_price_adds_benchmark_drilling_cost` |
| 119 | +- `tests/integration/architecture_p0/test_planner_gates.py::test_int_010_submit_handoff_accepts_cheaper_workspace_drilling_override` |
| 120 | +- `tests/integration/architecture_p0/test_planner_gates.py::test_int_019_single_part_benchmark_submit_succeeds_without_cost_gate` |
| 121 | +- `tests/integration/architecture_p0/test_planner_gates.py::test_int_018_benchmark_submit_accepts_yaml_motion_without_literal_tokens` |
| 122 | +- `tests/integration/architecture_p0/test_int_018_multipart_dfm_gate.py::test_int_018_simulate_does_not_dfm_reject_simple_multipart_benchmark` |
| 123 | +- `tests/integration/architecture_p0/test_int_018_multipart_dfm_gate.py::test_int_018_simulation_bounds_ignore_fixed_benchmark_fixtures` |
| 124 | +- `tests/integration/architecture_p0/test_missing_p0.py::test_int_005_trace_realtime_broadcast` |
| 125 | + |
| 126 | +## Special handling |
| 127 | + |
| 128 | +- `tests/integration/architecture_p1/test_api_fuzzing.py::test_api_fuzzing` is |
| 129 | + negative-adjacent fuzzing, but it is a broad schema sweep rather than a single |
| 130 | + deterministic rejection scenario. Keep it separate unless the suite gets a |
| 131 | + dedicated fuzz bucket or a dedicated `INT-NEG` policy for generative probing. |
| 132 | +- The electronics file still uses internal imports and `monkeypatch`; if that |
| 133 | + file is preserved, the negative rows above should move, but the long-term |
| 134 | + cleanup is to replace those internals with HTTP-only black-box integration. |
0 commit comments