Skip to content

Hosted player-log tail is a hard-coded 10,000 lines (kubernetes_runner.py:1398): verbose policies lose the start of every long episode #62

Description

@arisklar6

The hosted runner captures each player container's combined stdout and stderr with read_namespaced_pod_log(..., tail_lines=10000) at coworld/src/coworld/runner/kubernetes_runner.py:1398. Any policy that writes more than 10,000 lines in an episode loses the beginning of its log, not the end. For a 9,120-tick Zero Sum match at 24 Hz that is anything above roughly 1.1 lines per tick. docs/artifacts/PLAYER_LOGS.md says "up to the last 10,000 pod-log lines" but does not state that the head is what gets dropped.

Real impact: an external researcher running a homeostatic agent in Zero Sum reports recovering only about 60% of lived ticks from hosted logs because of this cut (paper: Zenodo 10.5281/zenodo.22713650, logs and scripts at github.com/Manelenrico/g-emv/tree/main/paper3). His ask, verbatim, was "logs that aren't cut at a fixed size."

Request:

(a) Expose the tail as a manifest field (for example player.log_tail_lines, with a platform cap) or as a runner setting, so a coworld can raise it deliberately.

(b) Document the current value and the head-loss behavior next to it in PLAYER_LOGS.md.

Tradeoff, stated plainly: player logs are stored per seat per episode. At 16 seats, 32 episodes per round and a round every 10 minutes, raising the tail 10x raises log storage and read-API time by up to 10x for verbose policies; the researcher's two batches alone were about 714 MB of raw logs. The kubelet's own container log rotation (containerLogMaxSize, default 10 MiB) may still cap what the pod-log API can return regardless of tail_lines, so (a) may need to be paired with a rotation setting.

Current workaround, which should be named in the docs next to the limit: the per-slot player artifact (COWORLD_PLAYER_ARTIFACT_UPLOAD_URL, 200 MB zip, fetched with coworld episode-logs --artifact). Policies that need a complete diary should write it there rather than to stdout. I have already pointed the researcher at it.

Filing as a request, not proposing an implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions