[FAQ] Why does the dlt pipeline pull fewer rows/tables than expected right after running the agent?

[c-ibarra]

Course

llm-zoomcamp

Question

Why does the dlt pipeline pull fewer rows/tables than expected right after running the agent (dlt workshop homework, Q2/Q3)?

Answer

Freshly created spans aren't immediately queryable through Logfire's Query API (GET https://logfire-us.pydantic.dev/v1/query) — ingestion can lag anywhere from a few seconds up to roughly a minute. If the dlt pipeline runs immediately after the agent call, the SQL query against records may only return a partial trace (e.g. 2 of 6 spans), which then silently undercounts both the number of normalized tables dlt creates (Q2) and the summed gen_ai.usage.input_tokens (Q3) — without raising an error.

Fix: poll the Query API for the full expected span count before running the pipeline, instead of a short fixed sleep:

import time

def wait_for_trace(trace_id, expected_span_count, read_token, max_wait_seconds=60):
    sql = f"SELECT COUNT(*) AS n FROM records WHERE trace_id = '{trace_id}'"
    waited = 0
    while waited < max_wait_seconds:
        response = requests.get(
            "https://logfire-us.pydantic.dev/v1/query",
            headers={"Authorization": f"Bearer {read_token}"},
            params={"sql": sql},
        )
        n = response.json()["columns"][0]["values"][0]
        if n >= expected_span_count:
            return n
        time.sleep(5)
        waited += 5
    return n

A quick sanity check: compare your own summed gen_ai.usage.input_tokens against Pydantic AI's own aggregated usage attribute on the top-level agent-run span (gen_ai.aggregated_usage.input_tokens) — they should match exactly once the full trace has landed.

Checklist

• I have searched existing FAQs and this question is not already answered
• The answer provides accurate, helpful information
• I have included any relevant code examples or links