Question about borg2 (borgstore)

[callegar]

Hi, I understand that borg, with its version 2, shall switch to a new storage format, borgstore.
I also undrstand that with this change chuncks will not anymore be packed in files containing multiple chunks (I think they were called segments) but every chunk will be stored as an individual file. Is this correct?
I have a few questions about this:

• does it mean that it will be impossible/impractical to use non default (smaller) chunk sizes, since you will get too many individual files in the repo? Some backup tools seem to adopt very small chunk sizes (e.g., just 8k for bup) in the name of better deduplication. Also the original attic adopted a small chunk size by default for the same reason and in the restic tracker there is a benchmark (Tweaking the chunker block size targets restic/restic#1071 (comment)) showing that the delta size (and thus the repo size) can more than triple going from 4k chunks to the borg default.
• the new approach seems similar to the one adopted by duplicacy, apart from the fact that duplicacy performs a packing operation before the chunking, so there can be chunks containing more than one file. The average chunk size being the same, will borg2 create more chunk files in the repo than duplicacy?
• can this change create issues when backing up to certain backends? E.g., in the rustic docs, there are paragraphs about the need for an appropriate pack size since some backends have hard limits on the total number of files.
• are packs expected to reappear in some form with the current lack thereof being only temporary?
• would it be sensible to disable chunking at all for certain file types? For instance, certain file types internally using compression seem to never diff well even when only small changes are introduced in the actual content (think jpeg). For some others, you can expect to never have incremental updates (e.g. a collection of flac audio files from audio CDs). Would something like this let one use smaller chunks for files that do actually diff well?

[ThomasWaldmann]

borgstore is the name of the library used by borg2 / master branch that implements a key/value store (and supports different kinds of backends: fs, sftp, rclone, maybe also s3 directly soon).

Currently, each chunk is stored as a separate object into that kv store, but that may change in the future.

Answers:

• that depends on the filesystem. bad filesystems may run into problems. from the borgstore side, we try to avoid directory size problems by spreading all these files over 2 layers of nested directories (256 + 65536 directories total). the "many files" problems does not just come from using a small chunk size, it also comes naturally because one file is at least one chunk and there are usually files smaller than the chunk size anyway. the small chunk size of attic caused "out of memory" conditions for some users, that was the reason why it was significantly increased.
• i know that duplicacy does it that way, maybe that could be a future option for borg also.
• yes, of course we can only work within the boundaries/limitations of the backend storage. besides the overall files count, there can also be issues with performance or cost (e.g. for cloud storage billing by the amount of api calls).
• that might be. it's a big change though and it also has downsides and additional complexity, so don't expect it tomorrow. :-)
• iirc, mimetype specific chunking was already discussed in some ticket, mainly to improve performance and dedup efficiency, but we didn't implement it yet due to its implementation and configuration costs (we had a similar approach for compression for a while, but removed it again for simplicity). besides improving deduplication efficiency we also chunk because we somehow need to limit the amount of data we are dealing with at a time.

[callegar]

Thanks for all the information!