Repository: don't mask the original exception when unwinding with buffered chunks

[ThomasWaldmann]

Problem

Repository.close() asserts that the PackWriter has no unflushed chunks. When a command aborts with chunks still buffered, the with repository: unwind calls close() and that assertion raises AssertionError, masking the original exception. Buffered chunks also leave F_PENDING entries in the chunk index, which the close()-time index persist asserts on (chunk ... has no pack location yet).

Reproduce: open a Repository, put() one small chunk (below the pack size limit), raise inside the with-block — the AssertionError replaces the real error.

This affects the paths that put chunks without a Cache:

• ArchiveChecker.add_reference() — borg check --repair
• borg debug put-obj

Commands that use a Cache are not affected: Cache is the inner with, so Cache.close() unwinds first and flushes the pack writer via _maybe_write_chunks_index(force=True). ArchiveChecker.finish() flushes as well since #10072, but only on the success path, so an abort before that still reaches close() with a non-empty buffer.

Fix

On exception unwind, Repository.__exit__ calls PackWriter.discard() — the abort-side counterpart to flush():

• it joins a still in-flight pack store first, so a pack that was already stored gets recorded in the index. Joining first also means dropping buffered entries cannot break update_pack_info() for a chunk id that sits in the in-flight pack and in the buffer (dropping first would delete the shared index entry, update_pack_info() would then KeyError mid-pack and leave F_PENDING leftovers for the persist to trip over — masking the original error again through a different door).
• then it drops the buffered pieces and their still-pending index entries: those chunks were never stored and die with the aborted operation. Store errors from the join are logged, not raised.

close()'s assertion is unchanged: on a clean close it still catches a genuinely forgotten flush(). __exit__ is the only place that knows whether we are unwinding, so the discard decision lives there.

All abort-time index cleanup goes through one helper, PackWriter._drop_index_entries():

• it never builds the chunk index from the repo — that I/O can fail and mask the error being unwound. With no in-memory index there is nothing to delete: add() installs a chunk's entry before buffering its piece, so pending entries never outlive a dropped index. (flush() asserts the index is loaded while chunks are outstanding, so on the normal path that caller bug fails loudly instead of silently leaving a stored pack unrecorded.)
• it only deletes entries that are still pending — a resolved entry means the chunk is in a stored pack; only the aborted duplicate piece dies. To make that hold for re-puts too, ChunkIndex.add() now keeps an existing resolved pack location instead of resetting the entry to pending: a re-put of an already stored chunk stays readable, an abort cannot lose it, and a successful flush() overwrites the entry with the new copy's location via update_pack_info(), as before.
• _apply_outcome() gets the same no-index guard, so joining a pack store while aborting cannot trigger an index rebuild either.

close() itself could still mask the original error a few lines further down: the close-time chunk index persist and the lock release both do store I/O, which fails again exactly when the abort was a store failure. close() now takes aborting=True from __exit__ (and from __enter__'s failure path) when an exception is unwinding: only then are these errors logged instead of raised (the persisted index is only a cache and gets rebuilt; an unreleasable lock goes stale eventually). On a clean close a failing persist or lock release raises, as on master, so it is not silently ignored. The lock release and store close run in a finally block, so even a raising close — including the unflushed-chunks assertion — cannot leak the exclusive lock anymore.

Also: dropped buffer pieces are logged (debug level), the abort semantics are documented in docs/internals/packs.rst, and ChunkIndex.is_pending/F_PENDING were added to the .pyi stub.

Tests

• test_exception_unwind_drops_buffered_chunks — the base repro: asserts the original ValueError (not AssertionError) propagates, and that after reopening the buffered chunk is neither in the chunk index nor readable.
• test_exception_unwind_records_inflight_pack_drops_buffer — unwind with one pack in flight and more chunks buffered, including a chunk id that is in both: the stored pack's chunks stay recorded and readable, only the never-stored chunk dies. Fails without the join-before-drop ordering.
• test_reput_stored_chunk_survives_unwind — re-put of an already stored chunk, then abort: the chunk stays in the index and readable. Fails without the add() change.
• test_exception_unwind_survives_failing_index_persist / test_exception_unwind_survives_failing_lock_release / test_exception_unwind_survives_failing_inflight_store — the store dies mid-operation: the close-time persist / lock release / in-flight pack store fails, is logged, the original exception propagates, and the lock is still released (checked by reopening with an exclusive lock).
• test_clean_close_raises_on_failing_index_persist / test_clean_close_raises_on_failing_lock_release — on a clean close the same failures raise; the lock is still released before the persist error propagates.
• test_exception_unwind_does_not_rebuild_dropped_chunk_index / test_exception_unwind_with_inflight_pack_and_dropped_index — buffered chunks or an in-flight pack plus a dropped index: no chunk-index rebuild happens during unwind and the original exception survives.
• test_flush_after_invalidate_asserts — flush() with outstanding chunks after invalidate_chunk_index() is a caller bug and fails loudly.
• test_close_with_unflushed_chunks_asserts — documents that a clean exit with buffered chunks still trips the assertion, and that even that failing close releases the lock and closes the store.
• test_chunkindex_add_keeps_resolved_location — pins the new add() semantics at the ChunkIndex level.

Full test suite passes, ruff clean.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 94.23077% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.08%. Comparing base (9056ef4) to head (dedd851).
⚠️ Report is 36 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/repository.py	94.23%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10013      +/-   ##
==========================================
+ Coverage   87.86%   88.08%   +0.22%     
==========================================
  Files         103      103              
  Lines       18876    18942      +66     
  Branches     2915     2923       +8     
==========================================
+ Hits        16586    16686     +100     
+ Misses       1589     1565      -24     
+ Partials      701      691      -10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[mr-raj12]

I tested this on the PR head merged with current master. I found two blockers.

1. Abort after a re-put drops a chunk that is already stored

ChunkIndex.add() resets a resolved entry to F_PENDING. On abort, _drop_index_entries() deletes every pending entry, and close() then saves the index without that chunk.

def test_reput_stored_chunk_survives_unwind(tmp_path):
    location = os.fspath(tmp_path / "repo")
    with pytest.raises(ValueError, match="original error"):
        with Repository(location, exclusive=True, create=True) as repository:
            repository.put(H(0), fchunk(b"DATA"))
            repository.flush()
            repository.put(H(0), fchunk(b"DATA"))
            raise ValueError("original error")
    with Repository(location, exclusive=True) as repository:
        assert pdchunk(repository.get(H(0))) == b"DATA"

This test fails with ObjectNotFound, but the pack is present in the store. It passes if H(0) was flushed in an earlier session, so only chunks flushed in the same session are affected. Please keep the resolved location in add(), or restore the overwritten entries on drop, and add this test.

2. close() hides errors on a clean exit also

The new except Exception blocks around write_chunkindex_to_repo() and lock.release() run even when no exception is unwinding. I checked a clean with Repository(...) block:

• If the index write fails, master raises OSError. This branch only logs a warning.
• If the lock release fails, master raises OSError. This branch only logs a warning and leaves the exclusive lock behind.

Both are plain logger.warning calls, so the command still exits 0. Please swallow these errors only in the unwind path from __exit__.

Minor

• The no-index guard in _apply_outcome() also runs on the normal path. If I put 3 chunks, call invalidate_chunk_index() and then flush(), master raises KeyError. This branch returns normally, and get(H(0)) then gives ObjectNotFound. Please limit the guard to the abort path or document it.
• The tests don't cover the no-index return in _apply_outcome(), the except in discard(), or the except around lock.release(). close() sets self.lock = None either way, so assert repository.lock is None doesn't test the release.