Multiple clients to one Borg repo

[NovaViper]

I just finished setting a borg repo on a raspberry pi. I'm wondering how I can have both my laptop and desktop going to that same repo. Both machines will be sharing the same data so I want to take advantage of the deduplication of the archives. Both computers run Linux also.

[ThomasWaldmann]

if you use a raspi (or any machine with not upgradable memory), avoid creating too big repos.

big repos use more resources (like memory) and if you run out of memory, it is game over on that system.

[jdchristensen]

But the server needs a lot less RAM than the client, so it shouldn't be too bad with the raspberry pi as a server. The RAM usage is described here. It would be better if it completely separated the server RAM usage from the client RAM usage, since it's very common that they are separate machines. With the default settings, I believe that 200GB of deduplicated data should require about 4MB of RAM for the server, plus some overhead of course. Have I got that right?

[ThomasWaldmann]

@jdchristensen guess that is correct for default chunker settings and only very big files (so that all chunks would be 2MiB+/-).

The problem is that if you have a huge amount of rather small files (like maildir or like source code), it creates a huge amount of chunks also, leading to a big repo index size. That can be easily 100..1000x as big (0.4..4GB rather than 4MB).

[jdchristensen]

@ThomasWaldmann That's a very good point about small files. I based my calculation on the FAQ, so I think the FAQ should also be updated to mention that small files increase the estimate. Taking my home folder as an example, it uses 18G, which would be 9000 chunks if all chunks were 2MB. But it contains 175,000 files, so there would be at least that many chunks (more since some files would certainly be split). So the FAQ estimate is off by about a factor of 20 in my case. (Even so, my home folder would probably only require around 8MB of RAM on the borg server, so for many types of backups, a raspberry pi can be fine.)

[ThomasWaldmann]

Yes, people just need to be aware of the limits. Problems come if you combine little RAM and lots of files.

Yeah, the FAQ / the part with the formulas might need some review / clarification.

In reality, it is even a bit more complicated due to the sparsity and growth of the hashtables and the temporary needs while transferring from the small current HT to the bigger new one when growing it (so there is a short spike more than doubling the memory usage). Usually not a problem, except if you are already at the memory limit.

[jdchristensen]

As a conservative upper bound, one could replace chunk_count ~= total_file_size / chunk_size with chunk_count ~= total_file_size / chunk_size + total_file_count. I think this should be a pretty good estimate for many file sizes.

[jdchristensen]

To answer the actual question, if you are using unencrypted repositories, you just point both clients at the same repo, and it works. For encrypted repos, it will depend on the type of encryption, and hopefully someone else can answer. (I think this should be explained in the FAQ item on this topic, which currently just gives warnings about doing this, but doesn't tell you how to do it.)

[ThomasWaldmann]

repokey just works (just give the passphrase).

keyfile needs locating the key on one system (~/.config/borg/keys/... usually) and copying it to the respective place on the other system. plus the passphrase, of course.

[NovaViper]

Thank you! I actually do have an encrypted repo with a keyfile, just got it working on both computers. 👍🏾

[callegar]

Can you please clarify if the forthcoming versions of borg (1.4 or 2 or maybe later) shall support backing up multiple (similar) clients to the same repository relaxing some of the difficulties that currently exist with borg 1.2? Thinking of the security implications and of the fact that you either need a huge cache on the clients or a very long time to fetch everything from the repository at the beginning of the backup. Thanks!

[ThomasWaldmann]

@callegar borg 1.4.x and 1.2.y (with y > 4) should be pretty compatible (the fork of 1.4-maint branch was at 1.2.7 IIRC).

For both of these, you should do the CVE-related upgrade steps from the top of the change log though.

About "relaxing the difficulties" (security implications, cache resync, etc.) - no change there in 1.2.x and 1.4.x. Bigger changes will come with borg2.