Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,10 @@ flowchart LR
1. The advisor creates a falsifiable assignment with the exact required research-base SHA, baseline metrics, expected mechanism, implementation scope, and stopping rules.
2. `create_assignment` creates the student branch and draft PR, embeds a typed assignment record, and applies the routing labels.
3. The assigned student receives one OpenHands conversation for that assignment revision. New PR comments and reviews are queued durably even while a turn is active, then delivered in the next bounded turn.
4. The student commits the exact implementation, launches supervised training, and records every referenced run in W&B.
4. The student commits the exact implementation, launches supervised training,
records every referenced run in W&B, and uses `post_assignment_comment` for
material progress, questions, blockers, or replies. Each typed comment wakes
the advisor without changing the PR's branch, draft state, or labels.
5. The student calls `submit_experiment_result`; the tool validates and publishes the branch before changing the PR to `status:review`.
6. The advisor compares the evidence, then uses the corresponding operation-specific tool to merge a reproducible winner, close a useful negative result, request a new revision, or send non-revision feedback.

Expand Down
20 changes: 14 additions & 6 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ the model-facing schema. It also canonicalizes every Senpai-authored comment to
an `ADVISOR:` or `STUDENT:` prefix from that trusted role; models supply plain
comment text and cannot impersonate the other role through a payload.

Advisor operations that act on an assignment share this object:
Assignment-scoped advisor and student operations share this object:

```json
{
Expand All @@ -313,18 +313,26 @@ Advisor operations that act on an assignment share this object:
| `publish_advisor_branch` | advisor | `remote_branch_sha_before_push`, `local_commit_sha` |
| `repair_assignment_routing` | advisor | `working_state` (`wip` or `review`) and a `blockers` list containing only `blocked`, `hold`, or `needs-rebase` |
| `send_assignment_feedback` | advisor | `feedback_id`, `comment` |
| `post_assignment_comment` | student | `comment_id`, `comment` |
| `request_assignment_revision` | advisor | `new_revision_id`, `required_base_sha`, `comment` |
| `accept_result_on_current_base` | advisor | `expected_current_base_sha`, `reason` |
| `merge_experiment` | advisor | `expected_current_base_sha`, `merge_method` |
| `close_experiment` | advisor | `reason` |
| `respond_to_human_issue` | advisor or student | `issue_number`, `human_message_id`, `response` |
| `submit_experiment_result` | student | `branch`, `remote_branch_sha_before_push`, `result` |

Student publication happens only inside `submit_experiment_result`, which
derives the PR and proposed local head from the structured result, then validates
repository, assignment, revision, student, and current remote head before it can
push. Marker comments are trusted only when authored by the authenticated token
actor.
Interim student communication happens through `post_assignment_comment`. The
runtime binds the configured student identity and validates the exact open WIP
or review assignment, revision, and PR head before posting an immutable typed comment.
Exact replay is a no-op; changed text uses a new `comment_id`. The comment does
not push, change routing, or end the assignment, and its trusted marker wakes the
advisor without replaying into the student's own feedback inbox.

Terminal student publication happens only inside `submit_experiment_result`,
which derives the PR and proposed local head from the structured result, then
validates repository, assignment, revision, student, and current remote head
before it can push. Marker comments are trusted only when authored by the
authenticated token actor.

Assignment creation checks the remote base SHA, creates an isolated empty
assignment commit with `git commit-tree`, publishes with force-with-lease,
Expand Down
2 changes: 1 addition & 1 deletion plugins/senpai/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ the current role:
`repair_assignment_routing`, `send_assignment_feedback`,
`request_assignment_revision`, `accept_result_on_current_base`,
`merge_experiment`, and `close_experiment`;
- students receive `submit_experiment_result`; and
- students receive `post_assignment_comment` and `submit_experiment_result`; and
- both roles receive `get_prs` and `respond_to_human_issue`.

Each tool has one operation-specific schema without a union wrapper and a
Expand Down
4 changes: 3 additions & 1 deletion senpai_agent/controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,9 @@
from senpai_agent.workspace import StudentWorkspaceReconciler, WorkspaceDivergence


_EDGE_TRIGGERED_EVENT_KINDS = frozenset({"research_base_changed"})
_EDGE_TRIGGERED_EVENT_KINDS = frozenset(
{"research_base_changed", "student_assignment_comment"}
)
_PROMPT_TEMPLATE_VARIABLES = frozenset(
{
"ADVISOR_BRANCH",
Expand Down
4 changes: 2 additions & 2 deletions senpai_agent/github/mailbox/advisor.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
result_matches_assignment,
versioned_event,
)
from .issues import human_issue_events
from .communications import advisor_communication_events

if TYPE_CHECKING:
from .core import GitHubMailbox
Expand Down Expand Up @@ -136,7 +136,7 @@ def advisor_events(
)

events.extend(_research_base_events(mailbox, active_assignments))
events.extend(human_issue_events(mailbox, issues))
events.extend(advisor_communication_events(mailbox, active_assignments, issues))
return tuple(events)


Expand Down
26 changes: 26 additions & 0 deletions senpai_agent/github/mailbox/communications.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
"""Advisor events derived from PR and Issue communication."""

from __future__ import annotations

from collections.abc import Sequence
from typing import TYPE_CHECKING

from senpai_agent.mailbox import ControllerEvent
from senpai_agent.models import AssignmentRecord

from .issues import human_issue_events
from .student_comments import student_assignment_comment_events

if TYPE_CHECKING:
from .core import GitHubMailbox


def advisor_communication_events(
mailbox: GitHubMailbox,
assignments: Sequence[tuple[dict[str, object], AssignmentRecord]],
issues: Sequence[dict[str, object]],
) -> list[ControllerEvent]:
return [
*student_assignment_comment_events(mailbox, assignments),
*human_issue_events(mailbox, issues),
]
152 changes: 152 additions & 0 deletions senpai_agent/github/mailbox/student_comments.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
"""Trusted student assignment comments delivered to advisor controllers."""

from __future__ import annotations

import sys
from collections.abc import Sequence
from typing import TYPE_CHECKING

from senpai_agent.github.http import GitHubReadError
from senpai_agent.mailbox import ControllerEvent
from senpai_agent.models import (
AssignmentCommentRecord,
AssignmentRecord,
authoritative_marker_line,
parse_assignment_comment_markers,
render_assignment_comment_marker,
)

from .values import (
FEEDBACK_EXCERPT_BYTES,
bounded_text,
github_datetime,
object_value,
payload_digest,
)

if TYPE_CHECKING:
from .core import GitHubMailbox


def student_assignment_comment_events(
mailbox: GitHubMailbox,
assignments: Sequence[tuple[dict[str, object], AssignmentRecord]],
) -> list[ControllerEvent]:
"""Wake the advisor for trusted typed messages from assigned students."""

with_comments = [
(pull, assignment)
for pull, assignment in assignments
if pull.get("comments_url")
]
if not with_comments:
return []
try:
actor = mailbox._github.actor()
except (GitHubReadError, TypeError) as error:
_report_read_error(f"actor {type(error).__name__}: {error}")
return []

events_by_key: dict[str, ControllerEvent] = {}
for pull, assignment in with_comments:
comments_url = str(pull["comments_url"])
number = int(pull["number"])
try:
comments = mailbox._github.objects(f"{comments_url}?per_page=100")
except (GitHubReadError, TypeError) as error:
_report_read_error(f"pr={number} {type(error).__name__}: {error}")
continue
for item in comments:
event = _comment_event(
mailbox,
pull,
assignment,
item,
actor=actor,
)
if event is not None:
previous = events_by_key.get(event.dedupe_key)
if previous is None or int(event.payload["github_comment_id"]) < int(
previous.payload["github_comment_id"]
):
events_by_key[event.dedupe_key] = event
events = list(events_by_key.values())
events.sort(
key=lambda event: (
github_datetime(str(event.payload["created_at"])),
int(event.payload["github_comment_id"]),
)
)
return events


def _comment_event(
mailbox: GitHubMailbox,
pull: dict[str, object],
assignment: AssignmentRecord,
item: dict[str, object],
*,
actor: str,
) -> ControllerEvent | None:
try:
author = str(object_value(item["user"])["login"])
body = str(item.get("body") or "")
records = parse_assignment_comment_markers(body)
github_comment_id = int(item["id"])
comment_url = str(item["html_url"])
created_at = str(item["created_at"])
except (KeyError, TypeError, ValueError):
return None
if author.casefold() != actor.casefold() or len(records) != 1:
return None
record: AssignmentCommentRecord = records[0]
number = int(pull["number"])
if (
record.repo != mailbox.repo
or record.pr_number != number
or record.assignment_id != assignment.assignment_id
or record.revision_id != assignment.revision_id
or record.student != assignment.student
):
return None
marker = render_assignment_comment_marker(record)
if authoritative_marker_line(body) != marker:
return None
message = "\n".join(body.splitlines()[1:]).strip()
if not message:
return None
payload = {
"number": number,
"pr_url": str(pull["html_url"]),
"comment_url": comment_url,
"github_comment_id": github_comment_id,
"comment_id": record.comment_id,
"assignment_id": record.assignment_id,
"revision_id": record.revision_id,
"student": record.student,
"message": bounded_text(message, limit=FEEDBACK_EXCERPT_BYTES),
"created_at": created_at,
}
semantic_payload = {
"number": number,
"assignment_id": record.assignment_id,
"revision_id": record.revision_id,
"student": record.student,
"comment_id": record.comment_id,
"message": message,
}
return ControllerEvent(
kind="student_assignment_comment",
dedupe_key=(
"student_assignment_comment:v2:" + payload_digest(semantic_payload)
),
payload=payload,
)


def _report_read_error(message: str) -> None:
print(
f"SENPAI_STUDENT_COMMENT_READ_ERROR {message}",
file=sys.stderr,
flush=True,
)
4 changes: 4 additions & 0 deletions senpai_agent/github/tools/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
GitHubMutationObservation,
MergeExperimentAction,
PublishAdvisorBranchAction,
PostAssignmentCommentAction,
RepairAssignmentRoutingAction,
RequestAssignmentRevisionAction,
RespondToHumanIssueAction,
Expand All @@ -20,6 +21,7 @@
CreateAssignmentTool,
MergeExperimentTool,
PublishAdvisorBranchTool,
PostAssignmentCommentTool,
RepairAssignmentRoutingTool,
RequestAssignmentRevisionTool,
RespondToHumanIssueTool,
Expand Down Expand Up @@ -58,6 +60,8 @@
"MergeExperimentAction",
"MergeExperimentTool",
"PRManifestObservation",
"PostAssignmentCommentAction",
"PostAssignmentCommentTool",
"PublishAdvisorBranchAction",
"PublishAdvisorBranchTool",
"RepairAssignmentRoutingAction",
Expand Down
24 changes: 24 additions & 0 deletions senpai_agent/github/tools/contracts.py
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,30 @@ class SendAssignmentFeedbackAction(Action):
)


class PostAssignmentCommentAction(Action):
"""Post one interim student message to the current assignment PR."""

assignment: AssignmentVersion = Field(
description="Current student assignment revision and PR-head precondition.",
)
comment_id: str = Field(
min_length=1,
max_length=256,
description=(
"Stable ID for this message. Exact replay is a no-op; a new message "
"must use a new ID."
),
)
comment: str = Field(
min_length=1,
max_length=50_000,
description=(
"Interim progress, question, blocker, evidence, or response to post on "
"the assigned pull request."
),
)


class RequestAssignmentRevisionAction(Action):
"""Start a new revision of an existing assignment on an exact research base."""

Expand Down
20 changes: 20 additions & 0 deletions senpai_agent/github/tools/definitions.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
GitHubMutationObservation,
MergeExperimentAction,
PublishAdvisorBranchAction,
PostAssignmentCommentAction,
RepairAssignmentRoutingAction,
RequestAssignmentRevisionAction,
RespondToHumanIssueAction,
Expand All @@ -32,6 +33,7 @@
)
from .runtime import (
GitHubToolRuntime,
PostAssignmentCommentExecutor,
SubmitExperimentResultExecutor,
tool_annotations,
)
Expand Down Expand Up @@ -228,3 +230,21 @@ def create(cls, runtime: GitHubToolRuntime) -> Sequence[Self]:
"make the PR review-ready.",
SubmitExperimentResultExecutor(runtime),
)


class PostAssignmentCommentTool(
ToolDefinition[PostAssignmentCommentAction, GitHubMutationObservation]
):
"""Post one student-authored comment to its active assignment PR."""

@classmethod
def create(cls, runtime: GitHubToolRuntime) -> Sequence[Self]:
return _tool(
cls,
PostAssignmentCommentAction,
"Post assignment comment",
"Post or exactly replay one meaningful interim progress update, "
"question, blocker, evidence item, or response on this student's "
"current active assignment without pushing or changing workflow state.",
PostAssignmentCommentExecutor(runtime),
)
Loading
Loading