Skip to content

Commit ff06d4a

Browse files
committed
YAML into a table
1 parent 5e321a4 commit ff06d4a

2 files changed

Lines changed: 24 additions & 139 deletions

File tree

‎config/skills_repo.lock.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
"branch": "main",
33
"head_commit": "3bf2d37a5bd9fd665bb7d42cee5f8ba67c64d01d",
44
"origin_url": "https://github.com/MRiabov/Problemologist-skills",
5-
"recorded_at_utc": "2026-03-26T19:22:01Z",
5+
"recorded_at_utc": "2026-03-26T19:23:55Z",
66
"skills_repo_path": "skills"
77
}

‎docs/academic-submission/final_project_report.md‎

Lines changed: 23 additions & 138 deletions
Original file line numberDiff line numberDiff line change
@@ -79,144 +79,29 @@ To optimize a model in a reinfocement learning environment a reward signal is ne
7979

8080
We split reward by their functional definition - fast, synctatic checks for file presence and synctatic validity, then "milestones" - actions requiring complex solution, and judge evaluation rewards. For example, the below table describes a reward system for a Engineering Coder agent.
8181

82-
<!--TODO: make a table-->
83-
84-
```yaml
85-
engineer_coder:
86-
description: >
87-
Scores the CAD engineer's ability to produce a manufacturable,
88-
cost-compliant solution that passes simulation.
89-
hard_checks:
90-
# Gate 1 — script compiles
91-
script_compiles:
92-
weight: 0.05
93-
description: >
94-
script.py executes without Python syntax or import errors.
95-
minimum_score: 0.02
96-
success_feedback: "The CAD script compiled and executed without syntax or import errors."
97-
failure_feedback: "The CAD script failed to compile or encountered a runtime error. Check for syntax errors or missing imports."
98-
99-
# Gate 2 — CAD geometry valid (no self-intersections, valid mesh)
100-
cad_geometry_valid:
101-
weight: 0.08
102-
description: >
103-
All parts export to valid OBJ meshes.
104-
success_feedback: "CAD geometry is valid and successfully exported."
105-
failure_feedback: "CAD geometry is invalid, has self-intersections, or failed to export."
106-
107-
# Gate 3 — manufacturability passes
108-
manufacturability_valid:
109-
weight: 0.07
110-
description: >
111-
All parts pass workbench manufacturability checks (e.g. no CNC
112-
undercuts, wall thickness >= minimum for injection molding).
113-
success_feedback: "The design passed all manufacturability constraints (CNC/Injection Molding)."
114-
failure_feedback: "The design has manufacturability issues (e.g. undercuts, thin walls). Review manufacturing constraints."
115-
116-
# Gate 4 — parts within build zone
117-
parts_within_build_zone:
118-
weight: 0.05
119-
description: >
120-
All custom-built parts are fully within the build_zone AABB.
121-
success_feedback: "All parts are correctly positioned within the designated build zone."
122-
failure_feedback: "Some parts are outside the build zone. Adjust part placement or scaling."
123-
124-
# Gate 5 — actual cost within cap (continuous penalty)
125-
cost_within_cap:
126-
weight: 0.10
127-
description: >
128-
Validated unit cost (from validate_and_price) <= max_unit_cost.
129-
partial: true
130-
penalty_formula: "max(0.0, 1.0 - max(0.0, actual_cost / max_unit_cost - 1.0))"
131-
success_feedback: "Manufacturing cost is within the target budget."
132-
failure_feedback: "Manufacturing cost exceeds the budget. Try using cheaper materials or reducing part volume."
133-
134-
# Gate 6 — actual weight within cap (continuous penalty)
135-
weight_within_cap:
136-
weight: 0.05
137-
description: >
138-
Validated assembly weight <= max_weight_g.
139-
partial: true
140-
penalty_formula: "max(0.0, 1.0 - max(0.0, actual_weight / max_weight_g - 1.0))"
141-
success_feedback: "Assembly weight is within the maximum limit."
142-
failure_feedback: "Assembly weight exceeds the limit. Use lighter materials or hollow out solid regions."
143-
144-
milestones:
145-
# Gate 7 — simulation result (dominant term)
146-
simulation_result:
147-
weight: 0.45
148-
description: >
149-
Simulation outcome. Full credit for success. Partial credit for
150-
failure proportional to progress toward goal:
151-
partial_score = 1.0 - (min_distance_achieved / initial_distance)
152-
Partial credit is discounted by 0.4 to incentivise full success:
153-
score = 0.45 * partial_score * 0.4 (if failed)
154-
score = 0.45 (if success)
155-
partial: true
156-
success_formula: "0.45"
157-
failure_formula: "0.45 * (1.0 - min_distance_achieved / initial_distance) * 0.4"
158-
success_feedback: "The solution successfully solved the problem in simulation."
159-
failure_feedback: "The simulation failed to reach the goal. Re-examine mechanical stability, component placement, or movement range."
160-
161-
judge_evaluation:
162-
reviewer_accepted:
163-
weight: 0.02
164-
description: >
165-
The execution reviewer accepted the implementation.
166-
checklist:
167-
latest_revision_verified:
168-
weight: 0.005
169-
description: >
170-
Latest implementation revision was the one evaluated for submission.
171-
validation_success:
172-
weight: 0.01
173-
description: >
174-
Validation success is satisfied at review time.
175-
simulation_success:
176-
weight: 0.015
177-
description: >
178-
Simulation success is satisfied at review time.
179-
visual_evidence_checked:
180-
weight: 0.005
181-
description: >
182-
Required static evidence has been inspected.
183-
dynamic_evidence_checked:
184-
weight: 0.005
185-
description: >
186-
Required dynamic evidence has been inspected.
187-
plan_fidelity:
188-
weight: 0.01
189-
description: >
190-
Implementation stays aligned with the approved plan.
191-
robustness:
192-
weight: 0.01
193-
description: >
194-
Solution is robust rather than passing a single lucky run.
195-
cost_weight_compliance:
196-
weight: 0.005
197-
description: >
198-
Final solution stays within cost and weight targets.
199-
manufacturability_compliance:
200-
weight: 0.01
201-
description: >
202-
Final solution remains manufacturable under the chosen process.
203-
dof_deviation_justified:
204-
weight: 0.005
205-
description: >
206-
Any DOF deviation from the approved plan is justified and reviewable.
207-
feedback_response_score:
208-
weight: 0.03
209-
description: >
210-
Fraction of valid failed reviewer checklist items addressed on the
211-
next coding revision without introducing new blocker regressions.
212-
partial: true
213-
bad_feedback_resistance_score:
214-
weight: 0.02
215-
description: >
216-
Score for following valid reviewer feedback while resisting incorrect
217-
or non-applicable requests in mixed-quality review seeds.
218-
partial: true
219-
```
82+
| Section | Key | Weight | Type | Description | Feedback / Formula |
83+
| -- | -- | -: | -- | -- | -- |
84+
| `engineer_coder` | overall objective | - | - | Scores the CAD engineer's ability to produce a manufacturable, cost-compliant solution that passes simulation. | - |
85+
| `hard_checks` | `script_compiles` | 0.05 | hard check | `script.py` executes without Python syntax or import errors. | Minimum score: `0.02`. Success: the script compiled and executed without syntax or import errors. Failure: the script failed to compile or encountered a runtime error. |
86+
| `hard_checks` | `cad_geometry_valid` | 0.08 | hard check | All parts export to valid OBJ meshes. | Success: CAD geometry is valid and successfully exported. Failure: geometry is invalid, has self-intersections, or failed to export. |
87+
| `hard_checks` | `manufacturability_valid` | 0.07 | hard check | All parts pass workbench manufacturability checks, such as no CNC undercuts and wall thickness above the injection molding minimum. | Success: the design passed all manufacturability constraints. Failure: the design has manufacturability issues, such as undercuts or thin walls. |
88+
| `hard_checks` | `parts_within_build_zone` | 0.05 | hard check | All custom-built parts are fully within the `build_zone` AABB. | Success: all parts are correctly positioned within the designated build zone. Failure: some parts are outside the build zone. |
89+
| `hard_checks` | `cost_within_cap` | 0.10 | partial hard check | Validated unit cost from `validate_and_price` is less than or equal to `max_unit_cost`. | Penalty formula: `max(0.0, 1.0 - max(0.0, actual_cost / max_unit_cost - 1.0))`. Success: manufacturing cost is within budget. Failure: manufacturing cost exceeds the budget. |
90+
| `hard_checks` | `weight_within_cap` | 0.05 | partial hard check | Validated assembly weight is less than or equal to `max_weight_g`. | Penalty formula: `max(0.0, 1.0 - max(0.0, actual_weight / max_weight_g - 1.0))`. Success: assembly weight is within the maximum limit. Failure: assembly weight exceeds the limit. |
91+
| `milestones` | `simulation_result` | 0.45 | partial milestone | Simulation outcome. Full credit is awarded for success; failed runs receive partial credit proportional to progress toward the goal. | Success formula: `0.45`. Failure formula: `0.45 * (1.0 - min_distance_achieved / initial_distance) * 0.4`. Success: the solution solved the problem in simulation. Failure: re-examine mechanical stability, component placement, or movement range. |
92+
| `judge_evaluation` | `reviewer_accepted` | 0.02 | judge signal | The execution reviewer accepted the implementation. | - |
93+
| `judge_evaluation` | `latest_revision_verified` | 0.005 | checklist item | Latest implementation revision was the one evaluated for submission. | - |
94+
| `judge_evaluation` | `validation_success` | 0.01 | checklist item | Validation success is satisfied at review time. | - |
95+
| `judge_evaluation` | `simulation_success` | 0.015 | checklist item | Simulation success is satisfied at review time. | - |
96+
| `judge_evaluation` | `visual_evidence_checked` | 0.005 | checklist item | Required static evidence has been inspected. | - |
97+
| `judge_evaluation` | `dynamic_evidence_checked` | 0.005 | checklist item | Required dynamic evidence has been inspected. | - |
98+
| `judge_evaluation` | `plan_fidelity` | 0.01 | checklist item | Implementation stays aligned with the approved plan. | - |
99+
| `judge_evaluation` | `robustness` | 0.01 | checklist item | Solution is robust rather than passing a single lucky run. | - |
100+
| `judge_evaluation` | `cost_weight_compliance` | 0.005 | checklist item | Final solution stays within cost and weight targets. | - |
101+
| `judge_evaluation` | `manufacturability_compliance` | 0.01 | checklist item | Final solution remains manufacturable under the chosen process. | - |
102+
| `judge_evaluation` | `dof_deviation_justified` | 0.005 | checklist item | Any DOF deviation from the approved plan is justified and reviewable. | - |
103+
| `judge_evaluation` | `feedback_response_score` | 0.03 | partial score | Fraction of valid failed reviewer checklist items addressed on the next coding revision without introducing new blocker regressions. | Partial score. |
104+
| `judge_evaluation` | `bad_feedback_resistance_score` | 0.02 | partial score | Score for following valid reviewer feedback while resisting incorrect or non-applicable requests in mixed-quality review seeds. | Partial score. |
220105

221106
### Optimizing the solution by LLMs
222107

0 commit comments

Comments
 (0)