Skip to content

Execute examples in parallel - #900

Open
AVHopp wants to merge 4 commits into
mainfrom
docs/parallelize_example_execution
Open

Execute examples in parallel#900
AVHopp wants to merge 4 commits into
mainfrom
docs/parallelize_example_execution

Conversation

@AVHopp

@AVHopp AVHopp commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

This PR makes the doc building execute the examples in parallel, significantly speeding up the overall execution.

This has been achieved as follows:

  • There is now a dedicated function for executing and converting a single example (_convert_example)
  • We do some Thread-magic that Claude came up with 😆
  • The whole job now fails when a single example fails, and it fails faster with a clearer error message, see here where I added an error into the basic example: https://github.com/AVHopp/baybe_dev/actions/runs/33525219751/job/99914223793 This does make the whole output a bit more verbose, but I think it is worth it (in particular since this makes it easier when just routing the action to an agent who can easily filter this)

Important: While this always improves the running time, it is hard to estimate by how much since this seems to depend on how much work load is currently done in other CI tests. I've seen the full doc building job to complete between 8 and 14 minutes, which is a speed-up in any case.

Compiled docs: See https://emdgroup.github.io/baybe/latest/ (compiled September 1st, around 5:30 PM). PLEASE TAKE A LOOK HERE at the examples - I looked at some of them and everything looked as expected, but please double-check that your favourite examples look like you expect them to look.


Stack created with GitHub Stacks CLIGive Feedback 💬

@AVHopp
AVHopp marked this pull request as ready for review August 20, 2026 10:46
Copilot AI lite review requested due to automatic review settings August 20, 2026 10:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the documentation example-build pipeline to execute example conversions concurrently, aiming to reduce overall docs build time while keeping the generated toctree structure intact.

Changes:

  • Refactors per-example conversion into a dedicated _convert_example() helper.
  • Collects all example conversion tasks first, then executes them in parallel via ThreadPoolExecutor.
  • Adds a max_workers parameter to control conversion concurrency.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/scripts/build_examples.py
Comment thread docs/scripts/build_examples.py Outdated
@AVHopp
AVHopp force-pushed the docs/parallelize_example_execution branch from e23fcc5 to 0268bec Compare August 20, 2026 11:01
@AVHopp
AVHopp marked this pull request as draft August 20, 2026 13:32
@AVHopp AVHopp self-assigned this Aug 20, 2026
@AVHopp
AVHopp force-pushed the docs/parallelize_example_execution branch from 0268bec to dfdf30d Compare August 31, 2026 07:59
@AVHopp
AVHopp force-pushed the docs/parallelize_example_execution branch from dfdf30d to 58fa424 Compare August 31, 2026 08:13
@AVHopp
AVHopp force-pushed the docs/parallelize_example_execution branch 2 times, most recently from 74f7bdc to 8791128 Compare August 31, 2026 11:49
@AVHopp
AVHopp force-pushed the docs/parallelize_example_execution branch from 8791128 to 0e3f154 Compare August 31, 2026 12:14
@AVHopp
AVHopp force-pushed the docs/parallelize_example_execution branch from 0e3f154 to a6a41bc Compare September 1, 2026 06:42
@AVHopp
AVHopp force-pushed the docs/parallelize_example_execution branch from a6a41bc to 6b5b791 Compare September 1, 2026 06:43
Base automatically changed from docs/refactor-apidoc_new to main September 1, 2026 14:39
@AVHopp
AVHopp force-pushed the docs/parallelize_example_execution branch 2 times, most recently from 0ed859a to 68ea474 Compare September 1, 2026 14:42
@AVHopp
AVHopp marked this pull request as ready for review September 1, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants