Skip to content

Log LLM provider errors with stack traces to fix silent failures - #38

Draft
calebwin with Copilot wants to merge 4 commits into
mainfrom
copilot/debug-blast-output-issue
Draft

Log LLM provider errors with stack traces to fix silent failures#38
calebwin with Copilot wants to merge 4 commits into
mainfrom
copilot/debug-blast-output-issue

Conversation

Copilot AI commented Dec 19, 2025

Copy link
Copy Markdown

Users reported BLAST appearing to run but producing no output/tokens. Root cause: LLM provider errors (invalid API keys, connection failures) were caught but logged without stack traces, making debugging impossible.

Changes

  • Core execution paths: Added exc_info=True to exception handlers in scheduler.py, executor.py, resource_factory.py
  • API endpoints: Added exc_info=True to handlers in server_api_chat_completions.py, server_api_realtime.py
  • Tests: Added test_error_logging.py to verify stack traces are logged

Example

Before:

except Exception as e:
    logger.error(f"Task {task_id} failed: {e}")  # No stack trace
    await self.complete_task(task_id, success=False)
    raise

After:

except Exception as e:
    logger.error(f"Task {task_id} failed: {e}", exc_info=True)  # Full stack trace logged
    await self.complete_task(task_id, success=False)
    raise

Now when authentication or connection issues occur, the engine log will contain full stack traces showing exactly what failed (e.g., AuthenticationError: Invalid API key with complete traceback).

Original prompt

Please debug this issue:

blast not returining any output
Image
EreQ

— 12/15/25, 10:33 PM
Are we moving on from blast?
Caleb Winston — 12/15/25, 11:21 PM
No, we're actively developing this and will get more active in the new year. We're working on a key improvement in the auto scheduler. (paper coming soon)

If you let us know your use case, we can better support it (and ensure it continues to function!)
EreQ

— 12/15/25, 11:57 PM
Right now its not even generating tokens
I thought it was a config issue but its not.
We're using it as a backend automation engine replacing the current front end
ronyw — 12/16/25, 12:55 AM
Curious why this happened, do you spot any issues in the engine log?
EreQ

— 12/16/25, 12:56 AM
There are no logs
all the metrics are at 0
The keys are working
ronyw — 12/16/25, 12:59 AM
When running blastai serve there is a log dir that gets displayed in the CLI. The engine log should contain helpful information for debugging.
❯ blastai serve
╭──────────────────────────────────────────────────────────────────────────────╮
│ Engine: http://127.0.0.1:8000 blast-logs/1eddb8f8.engine.log │
│ Web: http://localhost:3000 blast-logs/1eddb8f8.web.log │
╰──────────────────────────────────────────────────────────────────────────────╯

Are you using a different way to start the service?
EreQ

— 12/16/25, 1:00 AM
There are no errors
There are no tokens generated
ronyw — 12/16/25, 1:09 AM
Hmm, that sound suspiciously like an LLM provider/connection issue to me... although couldn't confirm without the logs. I also can't reproduce this locally on the latest commit and installing in a fresh environment
Also recommend checking if the backend service is up and running, i.e., not just the GUI is running
EreQ

— 12/16/25, 1:44 AM
So a provider error would show up?

Identify the potential root cause and things to try


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 3 commits December 19, 2025 07:07
Co-authored-by: calebwin <20741909+calebwin@users.noreply.github.com>
Co-authored-by: calebwin <20741909+calebwin@users.noreply.github.com>
Co-authored-by: calebwin <20741909+calebwin@users.noreply.github.com>
Copilot AI changed the title [WIP] Debug issue with blast not returning any output Log LLM provider errors with stack traces to fix silent failures Dec 19, 2025
Copilot AI requested a review from calebwin December 19, 2025 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants