JaggedTensor: empty outer lists are dropped by from_data_offsets_and_list_ids, hidden from lshape/unbind, and fail CPU indexing

[swahtz]

Summary

A two-level JaggedTensor whose outer lists include an empty one (a camera with no pixels, a batch element with no samples) is only partly supported. The nested-list constructor accepts it and len() counts it, but

1. from_data_offsets_and_list_ids drops a trailing empty outer list, because it takes the outer count from list_ids[:, 0].max() + 1 (src/fvdb/JaggedTensor.cpp, from_data_offsets_and_list_ids). There is no way to pass the intended count.
2. lshape (and unbind() at the outer level) omit empty outer lists, so lshape disagrees with len(). recompute_lsizes_if_dirty starts a new outer entry only when the outer list id changes, so a list with no members never gets one.
3. On CPU, indexing an empty outer list raises IndexError: select(): index 0 out of range for tensor of size [0, 2]. On CUDA the same index returns an empty JaggedTensor, which is the behaviour I would expect on both devices.

Related: #89 (empty JaggedTensor). Found while building the sparse contributor analysis in openvdb/fvdb-reality-capture#338 on top of #799, where the output of rasterize_contributing_gaussian_ids_sparse nests cameras, then pixels, then contributors, and a camera may request no pixels.

Reproduction

import torch
from fvdb import JaggedTensor

for dev in ("cpu", "cuda"):
    t = lambda *v: torch.tensor(v, dtype=torch.int32, device=dev)

    # Intended structure: three outer lists; the last one is empty.
    nested = JaggedTensor([[t(1, 2), t(3)], [t(4)], []])
    print(dev, "nested:", len(nested), nested.lshape, [len(x) for x in nested.unbind()])
    try:
        print(dev, "nested[2]:", len(nested[2]))
    except IndexError as e:
        print(dev, "nested[2] ->", e)

    # Same structure through from_data_offsets_and_list_ids.
    data = torch.tensor([1, 2, 3, 4], dtype=torch.int32, device=dev)
    offsets = torch.tensor([0, 2, 3, 4], device=dev)
    list_ids = torch.tensor([[0, 0], [0, 1], [1, 0]], dtype=torch.int32, device=dev)
    built = JaggedTensor.from_data_offsets_and_list_ids(data, offsets, list_ids)
    print(dev, "from_data_offsets_and_list_ids:", len(built), built.lshape)

    # An empty middle list survives construction but has the same lshape and CPU indexing problems.
    mid = JaggedTensor([[t(1, 2), t(3)], [], [t(4)]])
    print(dev, "middle empty:", len(mid), mid.lshape)

Output:

cpu nested: 3 [[2, 1], [1]] [2, 1]
cpu nested[2] -> select(): index 0 out of range for tensor of size [0, 2] at dimension 0
cpu from_data_offsets_and_list_ids: 2 [[2, 1], [1]]
cpu middle empty: 3 [[2, 1], [1]]
cuda nested: 3 [[2, 1], [1]] [2, 1]
cuda nested[2]: 0
cuda from_data_offsets_and_list_ids: 2 [[2, 1], [1]]
cuda middle empty: 3 [[2, 1], [1]]

Expected

• len(nested) == 3, nested.lshape == [[2, 1], [1], []], len(nested.unbind()) == 3, and nested[2] is an empty JaggedTensor on both devices.
• from_data_offsets_and_list_ids builds the same three-list structure. Since list_ids cannot express a trailing empty list, an optional num_outer_lists argument (or a check that makes the limitation an error rather than a silent drop) would do.

Actual

len() says 3 while lshape and unbind() say 2; from_data_offsets_and_list_ids returns 2 lists; CPU indexing of the empty list raises.

Environment

• fvdb 0.7.0 built from feature/gsplat-functional-surface at 0604777 (Publish Gaussian splatting kernels as a flat public surface in fvdb.functional #799); the JaggedTensor code involved is unchanged on main
• torch 2.13.0, CUDA 13.0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition
• Python 3.12.14, Linux 7.0.0-31-generic