Skip to content

perf: caching models - #224

Merged
kuraisle merged 14 commits into
mainfrom
perf-caching-models
Oct 23, 2026
Merged

kuraisle merged 14 commits into
mainfrom
perf-caching-models

Conversation

@kuraisle

Copy link
Copy Markdown
Member
🦋 Bug Fix

PR Description

For the search routes, an embedder/retriever was set up for each request, and a new llm and pipeline for every LLM request.
To start the fix, I moved these components out to a FastAPI Depends() function, which is cached with lru_cache decorator.

This did require me to change some of the logic of how these components work.
Previously, the pipelines took the search parameters as arguments on setup.
This wouldn't work with caching pipelines, unless you wanted to keep those parameters, so now the pipelines don't take any of the search parameters on startup, but only on running them.
This part actually doesn't save much time, as the pipeline setup is cheap.
Where this does save time is that haystack shouts at you if you try to add one of their components (which is how the retrievers are initialised) to more than one pipeline.

Then some of the tests needed changing.
The main one was that the logic limiting the number of results returned is all in the generated SQL, so making a python unit test for this no longer makes any sense, so this test was moved into an integration test that runs on the database instead.
This passes and the test also checks that restricting the domain works too.

Related Issues or other material

Related #220
Related #221
Related #222
Closes #220
Closes #221
Closes #222

✅ Added/updated tests?

  • This PR contains relevant tests / Or doesn't need to per the below explanation

@kuraisle kuraisle changed the title Perf caching models perf: caching models Sep 28, 2026
@Thomas-Rowlands
Thomas-Rowlands self-requested a review October 23, 2026 11:21

@Thomas-Rowlands Thomas-Rowlands left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM Tests pass and the explanation for the changes sounds good.

@kuraisle
kuraisle merged commit 4980d53 into main Oct 23, 2026
7 checks passed
@kuraisle
kuraisle deleted the perf-caching-models branch October 23, 2026 11:29
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 1.2.5 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

2 participants