[GSoC 2026] chatbot: report IntelOwl's own verdict on a job

[berardifra]

Description

summarize_job now returns IntelOwl's own reconciled verdict on the job's observable alongside
the metadata it already reported, plus the analyzer evidence behind it. This closes the third of the
bugs reported on Discord in July: "Summarize job" and "Evaluate job" returned identical output
because no chatbot tool ever read the findings.

The verdict is not a chatbot-side scale. The headline is obtained by running the platform's own
EvaluationEngineModule(job).run() and mapping it through the shared classify() helper, so the
chatbot says exactly the word the job-page badge shows. The LLM is never involved in producing it.

How it works

• New reader api_app/chatbot_manager/evaluation.py — a pure function evaluate_job(job)
  returning a JobEvaluation dataclass. It performs zero Job queries: the caller passes a job
  already scoped through visible_for_user, so the tool keeps the single tenancy boundary it had.
• Live recompute, not the stored scalar. The engine modules run asynchronously after the
  pipeline saves a transient un-reconciled merge, so job.data_model can be wrong for a window.
  Recomputing removes that race by construction and guarantees the chatbot cannot diverge from the badge.
• Evidence partitioning. The analyzers that ran are split into supporting / contradicting /
  silent, so "we don't know" is always attributable to named analyzers instead of an opaque shrug.
  Attribution goes through data_model_object_id rather than the report's GenericForeignKey, which
  would be a per-report N+1 — pinned by a query-count guard.
• Honest absences. A GENERIC observable (no DataModel class exists) and a job where no analyzer
  produced an evaluation both return no evaluation with an explicit reason, never a fabricated verdict.
• No 12th tool. The verdict is folded into summarize_job, so no new tool-routing decision is
  introduced (and no repeat of [GSoC 2026] Chatbot: warm KV-cache tool-selection flip picks the wrong tool ('show my recent jobs' → list_investigations) #3843). The two job chips collapse into one, "Summarize & evaluate",
  which keeps sending the already-validated "Summarize job #{id}" wording.
• The headline is echoed into the prose summary as well as carried structurally. This is a
  deliberate duplication: a live smoke against qwen2.5:3b showed the model reproduces the prose
  fields verbatim but paraphrases the structured object away — dropping the reliability and
  reporting contradicting analyzers as silent. A test pins the echo so it cannot silently regress.

Reliability is truncated, not rounded

The engine averages reliability into a float and stores it through an integer column, and Django's
IntegerField.get_prep_value truncates. Avg(5, 6) = 5.5 is persisted as 5 (suspicious);
rounding would have made the chatbot say malicious while the badge said suspicious. The reader
truncates to match, and a test pins it by replaying merge() and reading the value back from the DB.

Why the prompt word cap moves from 500 to 600

test_prompt_under_token_limit counts whitespace-separated words, not tokens. develop was
already at 492/500 before this PR: the guard had shrunk to 8 words of headroom and was rejecting
further rules rather than protecting the context window. At ~1.3-1.4 tokens per word here, 600 words
is roughly 850 tokens — about a tenth of Ollama's 8192 window, alongside the tool schemas and the
history. The prompt ships at 552. The docstring now states what is actually being measured.

Multi-tenancy

Job.get_user_events_data_model() resolves user events with the job owner's visibility, while
the requester may merely be an org-mate or a viewer of a CLEAR/GREEN job. The reader therefore
exposes only the boolean analyst_override — never the event's author, reason or tags. Nothing in
the verdict exposes more than get_job_details already does (it returns each report's full body).

Tests

• tests/api_app/chatbot_manager/test_evaluation.py — reader unit tests: partitioning, GENERIC,
  analyst override, boundary bucket, evidence caps, equality with the engine, and purity (no writes).
• tests/api_app/chatbot_manager/tools/test_summarize_job.py — envelope, no-evaluation, GENERIC, and
  the tenancy case (an invisible job is indistinguishable from a missing one — no verdict leaks).
• tests/api_app/chatbot_manager/test_query_counts.py — N+1 guard in the DataModel dimension.
• tests/api_app/chatbot_manager/test_agent.py — pins both former chip intents to the one tool.
• Frontend: QuickActions.test.jsx updated; the collapse is pinned by a negative assertion.

200 chatbot tests green, makemigrations --check clean, ruff + prettier clean, frontend
chat suite 65/65 green.

The narration of the structured verdict by qwen2.5:3b was measured end to end against a seeded
job (report and raw logs kept out of the repo): "summarize job #N" and "evaluate the results of job
#N" relay the verdict, the reliability, the named analyzers and the silent count correctly. On the
terse "is job #N malicious?" the model relays the verdict, the reliability and all three counts
correctly but does not enumerate the analyzer names — it never substitutes placeholder names. That
is the prompt's own "one paragraph unless the user asks for a list" rule winning over enumeration.

Refs #3895
(#3893 is merged — this branch is based on develop after that merge.)

[mlodic]

if there's nothing to add, I'll merge this

[berardifra]

if there's nothing to add, I'll merge this

Nothing to add, thanks.