Feature: Append to archive

[xeruf]

Have you checked borgbackup docs, FAQ, and open Github issues?

yes

Is this a BUG / ISSUE report or a QUESTION?

QUESTION

Describe the problem you're observing.

While collecting data for assembling my borg repository I have found folders that belong together.
Is there a way to amend an existing archive?

[ThomasWaldmann]

No, archives are not mutable after creation of them is finished.

There is borg recreate though which can do some sort of changes (but not add new files), but even that is not mutating an archive, but just reading it, creating a new one and then renaming the new over the old one.

[jdchristensen]

But maybe the more important advice is that the original poster should just create a new archive with the files included, and it will be roughly as fast as hypothetically adding those files to an existing archive, since borg does deduplication. They can then either delete the original archive, or keep it around until it gets pruned (since it will hardly use any additional space).

[xeruf]

The issue is that I want one backup for all files belonging to the project.
I would have to extract the archive, bring the files together and then create a new archive before deleting the old, which can create unnecessary wear on the partitions and eat time.

New suggestion:
How about a --extend flag for borg-create that can take an existing archive and includes all its contents in the new archive.
Then I could use that and delete the original archive afterwards.

[ThomasWaldmann]

Some notes about what could go wrong if we implemented something like borg create --extend ARCHIVE (new archive = existing archive's items + new paths, reusing the existing chunk references, without re-reading the old data) in borg 2:

1. Hardlink IDs can collide and give wrong file contents. This is the most important one. hlid = sha256(f"{ino}/{dev}"), with nothing else mixed in (fs.py). When borg extract meets a second item with an hlid it has already seen, it just links to the first file and ignores the second item's chunks (archive.py). Two ways to hit this in an extended archive:

• Reused inode: the filesystem gave the same inode number to a different hardlinked file after the source archive was made. The new file then gets restored as a link to an unrelated old file.
• Changed content: the same hardlinked file is in both the old part and the new paths, but its content changed in between. The newer version is silently thrown away.

The fix is to add a per-run salt to new hlids, or remap them all when merging.

2. A path can appear twice. The new paths may overlap what's already in the archive. Then list shows both entries, extract lets the later one overwrite the earlier one, mount shows only one, and diff gets confused. Extend needs a rule for this: reject, new wins, or old wins. Any rule except "reject" means the old item metadata chunks can't just be reused; they have to be read and filtered.

3. Item order breaks directory metadata. extract keeps a stack of directories. It sets a directory's mode and mtime as soon as an item outside that directory appears (extract_cmd.py), so it assumes a directory's contents come right after it. Appending items inside a directory that appeared earlier breaks that: the directory's mtime gets set, then new files are written into it and change it again. The same kind of assumption may exist in mount and diff too.

4. It's no longer a single point-in-time snapshot. Part of the archive is from T1 and part is from T2, and users expect an archive to be one snapshot. Archive-level metadata can only hold one value:

• time/start/end: prune keeps or drops the whole archive based on one timestamp. With T2, the stale T1 data looks newer than it is. With T1, the new data looks older than it is.
• hostname, username, cwd, command_line: these become partly false.
• Stats (nfiles, sizes): these must be recomputed for the combined archive, not copied from the source.

5. The two parts may have been made with different options. Excludes, --numeric-ids, --noacls/--noxattrs, chunker params and compression can all differ between the parts. Dedup doesn't care, but the archive records only one set of chunker_params, and other code relies on that value. For example, recreate uses it to decide whether to re-chunk.

6. Damage in the source archive gets copied. Reusing the old chunk references unchecked carries any missing or replaced chunks into the new archive.

7. It can't really be "in place". An archive's ID is the hash of its metadata, so an extended archive is always a new archive. Anything that refers to the old ID (scripts, tags, series position) has to be dealt with. It works like recreate: build a new archive, then delete the old one.

Not a problem: concurrency. compact takes an exclusive lock and deletes are soft. So another client can't remove the source archive's chunks while extend is reading them.

What a safe design needs: remap or salt hlids, decide what happens on path conflicts, rebuild the item stream in tree order, and set honest metadata (for example, record the source archive and its time).

The conflict and ordering requirements mean reading every item of the old archive. That's still much cheaper than the current workaround (borg mount the old archive, then borg create from the mountpoint plus the new paths), because no file data has to be read or chunked again.

[ThomasWaldmann]

Too many issues, closing this as not planned.