feat: Add in-place TTree branch addition and row extension by Yokubas · Pull Request #1690 · scikit-hep/uproot5

Yokubas · 2026-07-27T14:42:17Z

Summary

Implements in-place modification of existing TTrees:

f["tree"].add_branches({"new_x": array1, "new_y": array2, ...}) — add one or more new branches back-filled with provided data
f["tree"].extend({"x": array1, "y": array2}) — append new entries to one or more existing branches
f["tree"].extend({"x": array1, "new_y": array2}, accept_new_fields=True) — auto-add new branches back-filled with zeros, then extend

How it works

Uses uproot's cascade machinery instead of manual byte patching:

For extend: deserializes the existing TTree using uproot's reading side (branch members, cursor positions), reconstructs a ct.Tree cascade object, then delegates to the existing cascade write machinery — which appends new baskets and patches fBasketSeek, fBasketBytes, fBasketEntry, fWriteBasket, fEntryNumber, and fEntries in the TTree blob
For add_branches: creates new branch dict via _branch_np, calls write_anew to rewrite the TTree metadata blob with the new branch included, then writes one basket per new branch. Existing basket data is never touched — the metadata blob just gains new branch headers and the basket seek arrays are updated
For accept_new_fields=True: calls add_branches with zeros for existing entries, then extends with the provided data using the updated cascade

Metadata positions (metadata_start, basket_metadata_start) are found by searching for known byte patterns in the blob rather than fixed offsets, ensuring correctness for branches added via add_branches as well as freshly created ones.

Tests (23 passing)

Basic add_branches and extend for simple TBranch files
Multiple branches added or extended in a single call
Different dtypes (float32, int32)
Preserves existing data across operations
Sequential add_branches calls across separate sessions
add_branches then extend in the same session
Multiple extend calls across separate sessions
extend after add_branches in a new session
accept_new_fields behavior
ROOT verification for simple TBranch files
Corner cases: nonexistent branch, mismatched lengths, nonexistent tree, missing branch in extend, accept_new_fields error without flag
Jagged array extend (counter branch not required from user)
Extend after many extends (fMaxBaskets > 10)

Known limitations

Only top-level branches supported — adding subfields (e.g. particle.phi) is not yet implemented
add_branches for TBranchElement files is not yet supported
File-like objects not supported (requires file path for re-reading metadata)

…/uproot5 into Yokubas/ttree-inplace-v2

codecov · 2026-07-27T15:39:22Z

❌ 4 Tests Failed:

Tests completed	Failed	Passed	Skipped
1298	4	1294	24

View the top 3 failed test(s) by shortest run time

tests/test_0498_create_leaf_branch_in_extend.py::test_counter_shadows_branch_1

Stack Traces | 0.013s run time

tmp_path = PosixPath('.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_10')

    def test_counter_shadows_branch_1(tmp_path):
        newfile = os.path.join(tmp_path, "newfile.root")
    
        with uproot.recreate(newfile) as fout:
            fout.mktree("tree", {"nb": "int32", "b": "var * float64"})
            with pytest.raises(ValueError):
                fout["tree"].extend(
                    {"nb": [1, 2, 3], "b": [[1.1, 2.2, 3.3], [], [4.4, 5.5]]}
                )
>           fout["tree"].extend({"nb": [3, 0, 2], "b": [[1.1, 2.2, 3.3], [], [4.4, 5.5]]})

fout       = <WritableDirectory '/' at 0x1457f2d93380>
newfile    = '.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_10/newfile.root'
tmp_path   = PosixPath('.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_10')

tests/test_0498_create_leaf_branch_in_extend.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <WritableTree '/tree' at 0x145aa7258650>
data = {'nb': [3, 0, 2], 'b': [[1.1, 2.2, 3.3], [], [4.4, 5.5]]}
accept_new_fields = False

    def extend(self, data, *, accept_new_fields=False):
        """
        Args:
            data (dict of str \u2192 arrays): More array data to add to the TTree.
            accept_new_fields (bool): If True, new fields in data are automatically added
                with zeros back-filled for existing entries before extending.
    
        This method adds data to an existing TTree, whether it was created through
        assignment or :doc:`uproot.writing.writable.WritableDirectory.mktree`.
    
        The arrays must be a dict, but the values of the dict can be any of the
        array/DataFrame types described in :doc:`uproot.writing.writable.WritableTree`.
        However, these types must be compatible with the established TBranch
        types, the dict must contain a key for every TBranch, and the arrays must have
        the same lengths (in their first dimension).
    
        For example,
    
        .. code-block:: python
    
            my_directory.mktree("tree6", {"branch1": numpy_dtype, "branch2": awkward_type})
    
            my_directory["tree6"].extend({"branch1": another_numpy_array,
                                          "branch2": another_awkward_array})
    
        .. warning::
    
            **As a word of warning,** be sure that each call to :ref:`uproot.writing.writable.WritableTree.extend` includes at least 100 kB per branch/array. (NumPy and Awkward Arrays have an `nbytes <https://numpy..../reference/generated/numpy.ndarray.nbytes.html>`__ property; you want at least ``100000`` per array.) If you ask Uproot to write very small TBaskets, it will spend more time working on TBasket overhead than actually writing data. The absolute worst case is one-entry-per-:ref:`uproot.writing.writable.WritableTree.extend`. See `#428 (comment) <https://github..../uproot5/pull/428#issuecomment-908703486>`__.
        """
        if self._cascading is None:
            raise RuntimeError(
                "_cascading is None — this should not happen; please report this bug"
            )
        # validate branches
        # get user-facing branch names (exclude auto-generated counter and record parent branches)
        # get record parent names to exclude their sub-fields
        _record_names = {
            bd.get("name", "")
            for bd in self._cascading._branch_data
            if bd.get("kind") == "record"
        }
        _user_branch_names = [
            bd["fName"]
            for bd in self._cascading._branch_data
            if bd.get("kind") not in ("counter", "record")
            and "fName" in bd
            and not any(
                bd["fName"].startswith(rn + "_") or bd["fName"].startswith(rn + ".")
                for rn in _record_names
                if rn
            )
        ]
        # check if data looks like a flat dict of branch arrays (not a record/awkward array)
        _data_is_flat_dict = isinstance(data, dict) and all(
            not hasattr(v, "fields") for v in data.values()
        )
        if isinstance(data, dict) and _data_is_flat_dict:
            existing_names = _user_branch_names
            # also get record parent names that the user passes as dicts
            _record_parent_names = {
                bd.get("name")
                for bd in self._cascading._branch_data
                if bd.get("kind") == "record" and bd.get("name")
            }
            new_fields = {
                k: v
                for k, v in data.items()
                if k not in existing_names and k not in _record_parent_names
            }
            missing = [b for b in existing_names if b not in data]
            if missing:
                raise ValueError(
                    f"'extend' must fill every branch with the same number of entries; missing: {missing}"
                )
            if new_fields:
                if not accept_new_fields:
>                   raise ValueError(
                        "'extend' was given data that do not correspond to any branch: "
                        + repr(next(iter(new_fields)))
                    )
E                   ValueError: 'extend' was given data that do not correspond to any branch: 'nb'

_data_is_flat_dict = True
_record_names = set()
_record_parent_names = set()
_user_branch_names = ['b']
accept_new_fields = False
data       = {'b': [[1.1, 2.2, 3.3], [], [4.4, 5.5]], 'nb': [3, 0, 2]}
existing_names = ['b']
missing    = []
new_fields = {'nb': [3, 0, 2]}
self       = <WritableTree '/tree' at 0x145aa7258650>

.../test-env/lib/python3.13.../uproot/writing/writable.py:2280: ValueError

tests/test_0498_create_leaf_branch_in_extend.py::test_counter_shadows_branch_3

Stack Traces | 0.013s run time

tmp_path = PosixPath('.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_30')

    def test_counter_shadows_branch_3(tmp_path):
        newfile = os.path.join(tmp_path, "newfile.root")
    
        with uproot.recreate(newfile) as fout:
            fout.mktree("tree", {"b": "var * float64", "nb": "int32"})
            with pytest.raises(ValueError):
                fout["tree"].extend(
                    {"nb": [1, 2, 3], "b": [[1.1, 2.2, 3.3], [], [4.4, 5.5]]}
                )
>           fout["tree"].extend({"nb": [3, 0, 2], "b": [[1.1, 2.2, 3.3], [], [4.4, 5.5]]})

fout       = <WritableDirectory '/' at 0x145aa714a580>
newfile    = '.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_30/newfile.root'
tmp_path   = PosixPath('.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_30')

tests/test_0498_create_leaf_branch_in_extend.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <WritableTree '/tree' at 0x145aa7145fd0>
data = {'nb': [3, 0, 2], 'b': [[1.1, 2.2, 3.3], [], [4.4, 5.5]]}
accept_new_fields = False

    def extend(self, data, *, accept_new_fields=False):
        """
        Args:
            data (dict of str \u2192 arrays): More array data to add to the TTree.
            accept_new_fields (bool): If True, new fields in data are automatically added
                with zeros back-filled for existing entries before extending.
    
        This method adds data to an existing TTree, whether it was created through
        assignment or :doc:`uproot.writing.writable.WritableDirectory.mktree`.
    
        The arrays must be a dict, but the values of the dict can be any of the
        array/DataFrame types described in :doc:`uproot.writing.writable.WritableTree`.
        However, these types must be compatible with the established TBranch
        types, the dict must contain a key for every TBranch, and the arrays must have
        the same lengths (in their first dimension).
    
        For example,
    
        .. code-block:: python
    
            my_directory.mktree("tree6", {"branch1": numpy_dtype, "branch2": awkward_type})
    
            my_directory["tree6"].extend({"branch1": another_numpy_array,
                                          "branch2": another_awkward_array})
    
        .. warning::
    
            **As a word of warning,** be sure that each call to :ref:`uproot.writing.writable.WritableTree.extend` includes at least 100 kB per branch/array. (NumPy and Awkward Arrays have an `nbytes <https://numpy..../reference/generated/numpy.ndarray.nbytes.html>`__ property; you want at least ``100000`` per array.) If you ask Uproot to write very small TBaskets, it will spend more time working on TBasket overhead than actually writing data. The absolute worst case is one-entry-per-:ref:`uproot.writing.writable.WritableTree.extend`. See `#428 (comment) <https://github..../uproot5/pull/428#issuecomment-908703486>`__.
        """
        if self._cascading is None:
            raise RuntimeError(
                "_cascading is None — this should not happen; please report this bug"
            )
        # validate branches
        # get user-facing branch names (exclude auto-generated counter and record parent branches)
        # get record parent names to exclude their sub-fields
        _record_names = {
            bd.get("name", "")
            for bd in self._cascading._branch_data
            if bd.get("kind") == "record"
        }
        _user_branch_names = [
            bd["fName"]
            for bd in self._cascading._branch_data
            if bd.get("kind") not in ("counter", "record")
            and "fName" in bd
            and not any(
                bd["fName"].startswith(rn + "_") or bd["fName"].startswith(rn + ".")
                for rn in _record_names
                if rn
            )
        ]
        # check if data looks like a flat dict of branch arrays (not a record/awkward array)
        _data_is_flat_dict = isinstance(data, dict) and all(
            not hasattr(v, "fields") for v in data.values()
        )
        if isinstance(data, dict) and _data_is_flat_dict:
            existing_names = _user_branch_names
            # also get record parent names that the user passes as dicts
            _record_parent_names = {
                bd.get("name")
                for bd in self._cascading._branch_data
                if bd.get("kind") == "record" and bd.get("name")
            }
            new_fields = {
                k: v
                for k, v in data.items()
                if k not in existing_names and k not in _record_parent_names
            }
            missing = [b for b in existing_names if b not in data]
            if missing:
                raise ValueError(
                    f"'extend' must fill every branch with the same number of entries; missing: {missing}"
                )
            if new_fields:
                if not accept_new_fields:
>                   raise ValueError(
                        "'extend' was given data that do not correspond to any branch: "
                        + repr(next(iter(new_fields)))
                    )
E                   ValueError: 'extend' was given data that do not correspond to any branch: 'nb'

_data_is_flat_dict = True
_record_names = set()
_record_parent_names = set()
_user_branch_names = ['b']
accept_new_fields = False
data       = {'b': [[1.1, 2.2, 3.3], [], [4.4, 5.5]], 'nb': [3, 0, 2]}
existing_names = ['b']
missing    = []
new_fields = {'nb': [3, 0, 2]}
self       = <WritableTree '/tree' at 0x145aa7145fd0>

.../test-env/lib/python3.13.../uproot/writing/writable.py:2280: ValueError

tests/test_0498_create_leaf_branch_in_extend.py::test_counter_shadows_branch_4

Stack Traces | 0.013s run time

tmp_path = PosixPath('.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_40')

    def test_counter_shadows_branch_4(tmp_path):
        newfile = os.path.join(tmp_path, "newfile.root")
    
        with uproot.recreate(newfile) as fout:
            fout.mktree("tree", {"b": "var * float64", "nb": "int32"})
            with pytest.raises(ValueError):
                fout["tree"].extend(
                    {"b": [[1.1, 2.2, 3.3], [], [4.4, 5.5]], "nb": [1, 2, 3]}
                )
>           fout["tree"].extend({"b": [[1.1, 2.2, 3.3], [], [4.4, 5.5]], "nb": [3, 0, 2]})

fout       = <WritableDirectory '/' at 0x145aa714b700>
newfile    = '.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_40/newfile.root'
tmp_path   = PosixPath('.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_40')

tests/test_0498_create_leaf_branch_in_extend.py:91: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <WritableTree '/tree' at 0x145aa7146f90>
data = {'b': [[1.1, 2.2, 3.3], [], [4.4, 5.5]], 'nb': [3, 0, 2]}
accept_new_fields = False

    def extend(self, data, *, accept_new_fields=False):
        """
        Args:
            data (dict of str \u2192 arrays): More array data to add to the TTree.
            accept_new_fields (bool): If True, new fields in data are automatically added
                with zeros back-filled for existing entries before extending.
    
        This method adds data to an existing TTree, whether it was created through
        assignment or :doc:`uproot.writing.writable.WritableDirectory.mktree`.
    
        The arrays must be a dict, but the values of the dict can be any of the
        array/DataFrame types described in :doc:`uproot.writing.writable.WritableTree`.
        However, these types must be compatible with the established TBranch
        types, the dict must contain a key for every TBranch, and the arrays must have
        the same lengths (in their first dimension).
    
        For example,
    
        .. code-block:: python
    
            my_directory.mktree("tree6", {"branch1": numpy_dtype, "branch2": awkward_type})
    
            my_directory["tree6"].extend({"branch1": another_numpy_array,
                                          "branch2": another_awkward_array})
    
        .. warning::
    
            **As a word of warning,** be sure that each call to :ref:`uproot.writing.writable.WritableTree.extend` includes at least 100 kB per branch/array. (NumPy and Awkward Arrays have an `nbytes <https://numpy..../reference/generated/numpy.ndarray.nbytes.html>`__ property; you want at least ``100000`` per array.) If you ask Uproot to write very small TBaskets, it will spend more time working on TBasket overhead than actually writing data. The absolute worst case is one-entry-per-:ref:`uproot.writing.writable.WritableTree.extend`. See `#428 (comment) <https://github..../uproot5/pull/428#issuecomment-908703486>`__.
        """
        if self._cascading is None:
            raise RuntimeError(
                "_cascading is None — this should not happen; please report this bug"
            )
        # validate branches
        # get user-facing branch names (exclude auto-generated counter and record parent branches)
        # get record parent names to exclude their sub-fields
        _record_names = {
            bd.get("name", "")
            for bd in self._cascading._branch_data
            if bd.get("kind") == "record"
        }
        _user_branch_names = [
            bd["fName"]
            for bd in self._cascading._branch_data
            if bd.get("kind") not in ("counter", "record")
            and "fName" in bd
            and not any(
                bd["fName"].startswith(rn + "_") or bd["fName"].startswith(rn + ".")
                for rn in _record_names
                if rn
            )
        ]
        # check if data looks like a flat dict of branch arrays (not a record/awkward array)
        _data_is_flat_dict = isinstance(data, dict) and all(
            not hasattr(v, "fields") for v in data.values()
        )
        if isinstance(data, dict) and _data_is_flat_dict:
            existing_names = _user_branch_names
            # also get record parent names that the user passes as dicts
            _record_parent_names = {
                bd.get("name")
                for bd in self._cascading._branch_data
                if bd.get("kind") == "record" and bd.get("name")
            }
            new_fields = {
                k: v
                for k, v in data.items()
                if k not in existing_names and k not in _record_parent_names
            }
            missing = [b for b in existing_names if b not in data]
            if missing:
                raise ValueError(
                    f"'extend' must fill every branch with the same number of entries; missing: {missing}"
                )
            if new_fields:
                if not accept_new_fields:
>                   raise ValueError(
                        "'extend' was given data that do not correspond to any branch: "
                        + repr(next(iter(new_fields)))
                    )
E                   ValueError: 'extend' was given data that do not correspond to any branch: 'nb'

_data_is_flat_dict = True
_record_names = set()
_record_parent_names = set()
_user_branch_names = ['b']
accept_new_fields = False
data       = {'b': [[1.1, 2.2, 3.3], [], [4.4, 5.5]], 'nb': [3, 0, 2]}
existing_names = ['b']
missing    = []
new_fields = {'nb': [3, 0, 2]}
self       = <WritableTree '/tree' at 0x145aa7146f90>

.../test-env/lib/python3.13.../uproot/writing/writable.py:2280: ValueError

tests/test_0498_create_leaf_branch_in_extend.py::test_counter_shadows_branch_2

Stack Traces | 0.015s run time

tmp_path = PosixPath('.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_20')

    def test_counter_shadows_branch_2(tmp_path):
        newfile = os.path.join(tmp_path, "newfile.root")
    
        with uproot.recreate(newfile) as fout:
            fout.mktree("tree", {"nb": "int32", "b": "var * float64"})
            with pytest.raises(ValueError):
                fout["tree"].extend(
                    {"b": [[1.1, 2.2, 3.3], [], [4.4, 5.5]], "nb": [1, 2, 3]}
                )
>           fout["tree"].extend({"b": [[1.1, 2.2, 3.3], [], [4.4, 5.5]], "nb": [3, 0, 2]})

fout       = <WritableDirectory '/' at 0x145aa714a3c0>
newfile    = '.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_20/newfile.root'
tmp_path   = PosixPath('.../pytest-of-ar1092/pytest-668/test_counter_shadows_branch_20')

tests/test_0498_create_leaf_branch_in_extend.py:59: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <WritableTree '/tree' at 0x145aa725a090>
data = {'b': [[1.1, 2.2, 3.3], [], [4.4, 5.5]], 'nb': [3, 0, 2]}
accept_new_fields = False

    def extend(self, data, *, accept_new_fields=False):
        """
        Args:
            data (dict of str \u2192 arrays): More array data to add to the TTree.
            accept_new_fields (bool): If True, new fields in data are automatically added
                with zeros back-filled for existing entries before extending.
    
        This method adds data to an existing TTree, whether it was created through
        assignment or :doc:`uproot.writing.writable.WritableDirectory.mktree`.
    
        The arrays must be a dict, but the values of the dict can be any of the
        array/DataFrame types described in :doc:`uproot.writing.writable.WritableTree`.
        However, these types must be compatible with the established TBranch
        types, the dict must contain a key for every TBranch, and the arrays must have
        the same lengths (in their first dimension).
    
        For example,
    
        .. code-block:: python
    
            my_directory.mktree("tree6", {"branch1": numpy_dtype, "branch2": awkward_type})
    
            my_directory["tree6"].extend({"branch1": another_numpy_array,
                                          "branch2": another_awkward_array})
    
        .. warning::
    
            **As a word of warning,** be sure that each call to :ref:`uproot.writing.writable.WritableTree.extend` includes at least 100 kB per branch/array. (NumPy and Awkward Arrays have an `nbytes <https://numpy..../reference/generated/numpy.ndarray.nbytes.html>`__ property; you want at least ``100000`` per array.) If you ask Uproot to write very small TBaskets, it will spend more time working on TBasket overhead than actually writing data. The absolute worst case is one-entry-per-:ref:`uproot.writing.writable.WritableTree.extend`. See `#428 (comment) <https://github..../uproot5/pull/428#issuecomment-908703486>`__.
        """
        if self._cascading is None:
            raise RuntimeError(
                "_cascading is None — this should not happen; please report this bug"
            )
        # validate branches
        # get user-facing branch names (exclude auto-generated counter and record parent branches)
        # get record parent names to exclude their sub-fields
        _record_names = {
            bd.get("name", "")
            for bd in self._cascading._branch_data
            if bd.get("kind") == "record"
        }
        _user_branch_names = [
            bd["fName"]
            for bd in self._cascading._branch_data
            if bd.get("kind") not in ("counter", "record")
            and "fName" in bd
            and not any(
                bd["fName"].startswith(rn + "_") or bd["fName"].startswith(rn + ".")
                for rn in _record_names
                if rn
            )
        ]
        # check if data looks like a flat dict of branch arrays (not a record/awkward array)
        _data_is_flat_dict = isinstance(data, dict) and all(
            not hasattr(v, "fields") for v in data.values()
        )
        if isinstance(data, dict) and _data_is_flat_dict:
            existing_names = _user_branch_names
            # also get record parent names that the user passes as dicts
            _record_parent_names = {
                bd.get("name")
                for bd in self._cascading._branch_data
                if bd.get("kind") == "record" and bd.get("name")
            }
            new_fields = {
                k: v
                for k, v in data.items()
                if k not in existing_names and k not in _record_parent_names
            }
            missing = [b for b in existing_names if b not in data]
            if missing:
                raise ValueError(
                    f"'extend' must fill every branch with the same number of entries; missing: {missing}"
                )
            if new_fields:
                if not accept_new_fields:
>                   raise ValueError(
                        "'extend' was given data that do not correspond to any branch: "
                        + repr(next(iter(new_fields)))
                    )
E                   ValueError: 'extend' was given data that do not correspond to any branch: 'nb'

_data_is_flat_dict = True
_record_names = set()
_record_parent_names = set()
_user_branch_names = ['b']
accept_new_fields = False
data       = {'b': [[1.1, 2.2, 3.3], [], [4.4, 5.5]], 'nb': [3, 0, 2]}
existing_names = ['b']
missing    = []
new_fields = {'nb': [3, 0, 2]}
self       = <WritableTree '/tree' at 0x145aa725a090>

.../test-env/lib/python3.13.../uproot/writing/writable.py:2280: ValueError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

…/uproot5 into Yokubas/ttree-inplace-v2

ariostas

This is looking really promising! Thanks for the hard work!

I left some comments for specific locations.

But one more general comment is that this seems to be doing a lot of manual scanning and patching. It would be better to try to do what you did for the RNTuple one. Deserializing what you need to construct a WritableTree (with self._cascading properly built) and then let the existing functionality take care of as much of the serialization as possible.

…/uproot5 into Yokubas/ttree-inplace-v2

Yokubas · 2026-07-28T12:16:59Z

This is looking really promising! Thanks for the hard work!

I left some comments for specific locations.

But one more general comment is that this seems to be doing a lot of manual scanning and patching. It would be better to try to do what you did for the RNTuple one. Deserializing what you need to construct a WritableTree (with self._cascading properly built) and then let the existing functionality take care of as much of the serialization as possible.

Thanks for the feedback! I understand the concern about manual byte patching — it's fragile and hard to maintain.

For the RNTuple case, we could reconstruct a WritableNTuple because uproot already had full read/write support for the format. For TTree, the challenge is that _cascadetree.py handles writing new trees from scratch, but there's no path to deserialize an existing TTree blob back into a WritableTree with self._cascading set.

Would you be able to point me toward what would need to change in _cascadetree.py to support this? Specifically — how to reconstruct the cascade objects (branches, baskets, etc.) from an existing serialized TTree so we can use the existing extend/write machinery?

…ce-v2

eduardo-rodrigues · 2026-08-03T13:23:57Z

Hello @Yokubas, cc @ariostas as reviewer - there are many people watching this important package and I'm sure I'm not the only one noticing that you often get trivial pre-commit updates on your commits. This results in many more email notifications, which could be trivially spared to everyone if you would run pre-commit locally. This standard way of proceding is even mentioned in the CONTRIBUTING file. Could you kindly follow the guidelines and save us all tens of meaningless email notifications? Thank you in advance.

Yokubas · 2026-08-03T13:34:06Z

Apologies for the noise! I've set up pre-commit locally now — won't happen again.

…ce-v2

…/uproot5 into Yokubas/ttree-inplace-v2

…sting_ttree

ariostas · 2026-08-06T13:42:09Z

-    def __repr__(self):
-        return "<WritableTree {} at 0x{:012x}>".format(
-            repr("/" + "/".join(self._path)), id(self)
-        )
-
-    @property
-    def path(self):
-        """
-        Path of directory names to this TTree as a tuple of strings.
-        """
-        return self._path
-
-    @property
-    def object_path(self) -> str:
-        """
-        Path of directory names to this TTree as a single string, delimited by
-        slashes.
-        """
-        return "/".join(("", *self._path, "")).replace("//", "/")
-
-    @property
-    def file_path(self) -> str | None:
-        """
-        Filesystem path of the open file, or None if using a file-like object.
-        """
-        return self._file.file_path
-
-    @property
-    def file(self):
-        """
-        Handle to the :doc:`uproot.writing.writable.WritableDirectory` in which
-        this directory can be found.
-        """
-        return self._file
-
-    def close(self):
-        """
-        Explicitly close the file.
-
-        (Files can also be closed with the Python ``with`` statement, as context
-        managers.)
-
-        After closing, objects cannot be read from or written to the file.
-        """
-        self._file.close()
-
-    @property
-    def closed(self) -> bool:
-        """
-        True if the file has been closed; False otherwise.
-
-        The file may have been closed explicitly with
-        :ref:`uproot.writing.writable.WritableFile.close` or implicitly in the Python
-        ``with`` statement, as a context manager.
-
-        After closing, objects cannot be read from or written to the file.
+    def add_branches(self, branches):
        """
-        return self._file.closed
+        Args:
+            branches (dict of str -> array): Names and data of new branches.

-    def __enter__(self):
-        self._file.sink.__enter__()
-        return self
+        Adds new branches to this TTree in-place. Only the new branch data and
+        an updated TTree header are written; existing data is never touched.
+        Works with both simple TBranch and TBranchElement files.

-    def __exit__(self, exception_type, exception_value, traceback):
-        self._file.sink.__exit__(exception_type, exception_value, traceback)
+        .. code-block:: python

-    @property
-    def compression(self):


Seems like you accidentally deleted a bunch of properties here. We should keep those

Restored the accidentally deleted WritableTree properties (__repr__, path, object_path, file_path, file, close, closed, __enter__, __exit__, compression) — they got removed when I was cleaning up _extend_inplace

ariostas · 2026-08-06T13:45:19Z

+            existing_names = [bd["fName"] for bd in self._cascading._branch_data]
+            new_fields = {k: v for k, v in data.items() if k not in existing_names}
+            missing = [b for b in existing_names if b not in data]
+            if missing:


As you saw from the failed tests, there is one exception here. When you have a jagged array it creates another branch with the same name, but with an n prefix that stores how many elements each row has. And they are automatically generated by Uproot/ROOT, so the user is not expected to pass them in.

I think also records need to be skipped. So the fix could probably be

existing_names = [bd["fName"] for bd in self._cascading._branch_data if bd.datum["kind"] not in ("counter", "record")]

Counter and record branches are now excluded from the missing check in extend() — existing_names now filters by bd.datum["kind"] not in ("counter", "record") as you suggested. Also fixed _load_existing_ttree to correctly reconstruct counter branch kind and references so that jagged array extend works in uproot.update sessions too. 22 tests passing

One thing to flag — test_writable_vs_readable_tree in test_0406_write_a_ttree.py expects a TypeError when accessing a ROOT-written TTree via uproot.update. This was the old behavior when uproot couldn't handle existing TTrees. Since our PR now supports this, the test expectation is outdated — should this test be updated to reflect the new behavior?

Yeah, let's update that test

…ce-v2

ariostas · 2026-08-07T18:54:47Z

+            existing_names = [bd["fName"] for bd in self._cascading._branch_data]
+            new_fields = {k: v for k, v in data.items() if k not in existing_names}
+            missing = [b for b in existing_names if b not in data]
+            if missing:


Yeah, let's update that test

…ce-v2

ariostas · 2026-08-12T14:25:37Z

+        key = self._file._cascading.rootdirectory.data.get_key(source, 1)
+        casc = self._file.root_directory._load_existing_ttree(key)._cascading


Do these two lines work if the tree is in a subdirectory?

Fixed the subdirectory issue — add_branches now navigates through self._path[:-1] to find the correct directory instead of always using rootdirectory. Also fixed casc._directory to point to the correct directory so write_anew works properly. Tested with a tree in a subdirectory and it works correctly

…tree

Yokubas and others added 13 commits July 27, 2026 15:16

Add in-place TTree branch addition

9e96466

Add in-place TTree branch addition with tests

81ce9b3

Add in-place TTree branch addition with tests

3f89df6

Add in-place TTree extend method

56fd418

Add in-place TTree extend and add_branches with tests

73d82ab

style: pre-commit fixes

990acb0

Rename test file to test_1690_ttree_inplace.py

fbe47d8

Merge branch 'Yokubas/ttree-inplace-v2' of https://github.com/Yokubas…

f91e25a

…/uproot5 into Yokubas/ttree-inplace-v2

style: pre-commit fixes

cdf5d21

Move imports to top level in writable.py

daab666

Merge branch 'Yokubas/ttree-inplace-v2' of https://github.com/Yokubas…

1a990e0

…/uproot5 into Yokubas/ttree-inplace-v2

Add accept_new_fields kwarg to extend

cf5532a

style: pre-commit fixes

b0e7d43

Yokubas and others added 3 commits July 27, 2026 17:53

Fix extend for multiple sessions and add basket overflow check

1b303df

Merge branch 'Yokubas/ttree-inplace-v2' of https://github.com/Yokubas…

9ab366c

…/uproot5 into Yokubas/ttree-inplace-v2

style: pre-commit fixes

314ee2f

Yokubas changed the title ~~Add in-place TTree branch addition and row extension~~ feat: Add in-place TTree branch addition and row extension Jul 27, 2026

ariostas reviewed Jul 27, 2026

View reviewed changes

Yokubas and others added 11 commits July 28, 2026 12:18

Use self._file instead of opening uproot.update again

2a2983d

Merge branch 'Yokubas/ttree-inplace-v2' of https://github.com/Yokubas…

3537114

…/uproot5 into Yokubas/ttree-inplace-v2

Fix hardcoded byte range for fBranches TObjArray bcnt search

cbb47aa

Find TTree fEntries more reliably using unique sequence

36429fe

style: pre-commit fixes

ba54114

Validate branch length matches tree in add_branches

f012ece

Merge branch 'Yokubas/ttree-inplace-v2' of https://github.com/Yokubas…

55f965d

…/uproot5 into Yokubas/ttree-inplace-v2

Clean up tests and enforce all branches in extend

6373215

style: pre-commit fixes

f97c03d

Fix fEND write size for big files (>2GB)

314fe05

Merge branch 'Yokubas/ttree-inplace-v2' of https://github.com/Yokubas…

b7c62d1

…/uproot5 into Yokubas/ttree-inplace-v2

Yokubas and others added 3 commits August 3, 2026 14:42

Merge remote-tracking branch 'upstream/main' into Yokubas/ttree-inpla…

f34b48c

…ce-v2

Use cascade machinery for extend on existing TTrees

3cb7b1d

style: pre-commit fixes

d7ae561

Yokubas added 5 commits August 5, 2026 14:30

Merge remote-tracking branch 'upstream/main' into Yokubas/ttree-inpla…

e4c8a04

…ce-v2

Rewrite add_branches using cascade machinery

af5ca84

Merge branch 'Yokubas/ttree-inplace-v2' of https://github.com/Yokubas…

7f0327b

…/uproot5 into Yokubas/ttree-inplace-v2

Remove dead _extend_inplace code and fully use cascade machinery

0fb41a2

Fix metadata_start and basket_metadata_start computation in _load_exi…

8b906e5

…sting_ttree

ariostas reviewed Aug 6, 2026

View reviewed changes

Yokubas added 4 commits August 7, 2026 12:00

Restore accidentally deleted WritableTree properties

69c7a40

Fix extend validation to skip counter and record branches

49ca9ed

Merge remote-tracking branch 'upstream/main' into Yokubas/ttree-inpla…

7aa10f2

…ce-v2

Fix extend validation to handle counter, record, and jagged branches

04d8bf1

ariostas reviewed Aug 7, 2026

View reviewed changes

Yokubas added 6 commits August 10, 2026 11:20

Merge remote-tracking branch 'upstream/main' into Yokubas/ttree-inpla…

c99184b

…ce-v2

Update test_writable_vs_readable_tree to reflect new behavior

84d300f

Use existing tree title in _load_existing_ttree instead of empty string

e5edb36

Fix basket_metadata_start formula for trees with fMaxBaskets != 10

64cb73e

Add test for extend after many extends (fMaxBaskets > 10)

495e10a

Replace BytesIO approach with sink.read + _ReadForUpdate pattern

3c0dc35

ariostas reviewed Aug 12, 2026

View reviewed changes

Yokubas added 4 commits August 12, 2026 16:54

Use fIsRange to detect counter branches instead of name pattern matching

ae1f135

Use key.cycle instead of hardcoded cycle number 1 in _load_existing_t…

85a4281

…tree

Fix subdirectory support in add_branches and _load_existing_ttree

d0f2b8e

Fix counter branch validation in extend

d790b77

		key = self._file._cascading.rootdirectory.data.get_key(source, 1)
		casc = self._file.root_directory._load_existing_ttree(key)._cascading

Uh oh!

Conversation

Yokubas commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Tests (23 passing)

Known limitations

Uh oh!

codecov Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 4 Tests Failed:

Uh oh!

ariostas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Yokubas commented Jul 28, 2026

Uh oh!

eduardo-rodrigues commented Aug 3, 2026

Uh oh!

Yokubas commented Aug 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Yokubas commented Jul 27, 2026 •

edited

Loading

codecov Bot commented Jul 27, 2026 •

edited

Loading