Skip to content

Wrap correctionlib._core.CorrectionSet.from_string with an LRU cache #349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ikrommyd wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Wrap correctionlib._core.CorrectionSet.from_string with an LRU cache #349

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions src/correctionlib/highlevel.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

import json
from collections.abc import Iterator, Mapping
from functools import lru_cache
from numbers import Integral
from typing import TYPE_CHECKING, Any, Callable

Expand Down Expand Up @@ -333,6 +334,11 @@ def __iter__(self) -> Iterator[str]:
return iter(self._base)


@lru_cache(maxsize=64)

@nsmith- nsmith- Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty large size, which would increase the memory usage of the process a bit depending on the size of the correction set (some are 10s MB)

@ikrommyd ikrommyd Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the typical size of those strings? I thought they would be in the kBs so I thought 64 would be fine. What number would you suggest?

def _from_string(data: str) -> correctionlib._core.CorrectionSet:
return correctionlib._core.CorrectionSet.from_string(data)


class CorrectionSet(Mapping[str, Correction]):
"""High-level correction set evaluator object

Expand All @@ -347,7 +353,7 @@ def __init__(self, data: Any):
self._data = data
else:
self._data = data.model_dump_json(exclude_unset=True)
self._base = correctionlib._core.CorrectionSet.from_string(self._data)
self._base = _from_string(self._data)

@classmethod
def from_file(cls, filename: str) -> CorrectionSet:
Expand All @@ -362,7 +368,7 @@ def __getstate__(self) -> dict[str, Any]:

def __setstate__(self, state: dict[str, Any]) -> None:
self._data = state["_data"]
self._base = correctionlib._core.CorrectionSet.from_string(self._data)
self._base = _from_string(self._data)

def _ipython_key_completions_(self) -> list[str]:
return list(self.keys())
Expand Down
Loading