Release 2.0.0b7, beta quality, for testing only

[ThomasWaldmann]

borgbackup 2.0.0b7 beta release - do not use this on production backup repositories.

Long changelog:
https://borgbackup.readthedocs.io/en/2.0.0b7/changes.html#version-2-0-0b7-2023-09-14

If you use pip to install this, use: pip install "borgbackup==2.0.0b7"

Docs: https://borgbackup.readthedocs.io/en/2.0.0b7/

Feedback:

---

This discussion was created from the release Release 2.0.0b7, beta quality, for testing only.

[bket]

Tested on OpenBSD amd64 current. Builds OK, all tests pass, and so far no issues found with my use case.

Interestingly, several source files (*.c, *.h, *.pyx) are being packaged/installed (2.0.0b7 versus 2.0.0b6). I believe this is caused by 61e96eb. Is this intended?

--- 2.0.0b6
+++ 2.0.0b7
@@ -45,6 +45,10 @@ site-packages
 site-packages/borg/${MODPY_PYCACHE}version.${MODPY_PYC_MAGIC_TAG}pyc
 site-packages/borg/${MODPY_PYCACHE}xattr.${MODPY_PYC_MAGIC_TAG}${MODPY_PYOEXTENSION}
 site-packages/borg/${MODPY_PYCACHE}xattr.${MODPY_PYC_MAGIC_TAG}pyc
+site-packages/borg/_chunker.c
+site-packages/borg/_endian.h
+site-packages/borg/_hashindex.c
+site-packages/borg/_item.c
 site-packages/borg/_version.py
 site-packages/borg/archive.py
 site-packages/borg/archiver/
@@ -134,12 +138,24 @@ site-packages
 site-packages/borg/archiver/tar_cmds.py
 site-packages/borg/archiver/transfer_cmd.py
 site-packages/borg/cache.py
+site-packages/borg/cache_sync/
+site-packages/borg/cache_sync/cache_sync.c
+site-packages/borg/cache_sync/sysdep.h
+site-packages/borg/cache_sync/unpack.h
+site-packages/borg/cache_sync/unpack_define.h
+site-packages/borg/cache_sync/unpack_template.h
+site-packages/borg/checksums.c
 @so site-packages/borg/checksums.${MODPY_PYC_MAGIC_TAG}so
 site-packages/borg/checksums.pyi
+site-packages/borg/checksums.pyx
+site-packages/borg/chunker.c
 @so site-packages/borg/chunker.${MODPY_PYC_MAGIC_TAG}so
 site-packages/borg/chunker.pyi
+site-packages/borg/chunker.pyx
+site-packages/borg/compress.c
 @so site-packages/borg/compress.${MODPY_PYC_MAGIC_TAG}so
 site-packages/borg/compress.pyi
+site-packages/borg/compress.pyx
 site-packages/borg/constants.py
 site-packages/borg/crypto/
 site-packages/borg/crypto/__init__.py
@@ -155,11 +171,15 @@ site-packages
 site-packages/borg/crypto/file_integrity.py
 site-packages/borg/crypto/key.py
 site-packages/borg/crypto/keymanager.py
+site-packages/borg/crypto/low_level.c
 @so site-packages/borg/crypto/low_level.${MODPY_PYC_MAGIC_TAG}so
+site-packages/borg/crypto/low_level.pyx
 site-packages/borg/fuse.py
 site-packages/borg/fuse_impl.py
+site-packages/borg/hashindex.c
 @so site-packages/borg/hashindex.${MODPY_PYC_MAGIC_TAG}so
 site-packages/borg/hashindex.pyi
+site-packages/borg/hashindex.pyx
 site-packages/borg/helpers/
 site-packages/borg/helpers/__init__.py
 ${MODPY_COMMENT}site-packages/borg/helpers/${MODPY_PYCACHE}/
@@ -210,8 +230,10 @@ site-packages
 site-packages/borg/helpers/shellpattern.py
 site-packages/borg/helpers/time.py
 site-packages/borg/helpers/yes_no.py
+site-packages/borg/item.c
 @so site-packages/borg/item.${MODPY_PYC_MAGIC_TAG}so
 site-packages/borg/item.pyi
+site-packages/borg/item.pyx
 site-packages/borg/locking.py
 site-packages/borg/logger.py
 site-packages/borg/manifest.py
@@ -227,7 +249,19 @@ site-packages
 site-packages/borg/platform/${MODPY_PYCACHE}xattr.${MODPY_PYC_MAGIC_TAG}${MODPY_PYOEXTENSION}
 site-packages/borg/platform/${MODPY_PYCACHE}xattr.${MODPY_PYC_MAGIC_TAG}pyc
 site-packages/borg/platform/base.py
+site-packages/borg/platform/darwin.c
+site-packages/borg/platform/darwin.pyx
+site-packages/borg/platform/freebsd.c
+site-packages/borg/platform/freebsd.pyx
+site-packages/borg/platform/linux.c
+site-packages/borg/platform/linux.pyx
+site-packages/borg/platform/posix.c
 @so site-packages/borg/platform/posix.${MODPY_PYC_MAGIC_TAG}so
+site-packages/borg/platform/posix.pyx
+site-packages/borg/platform/syncfilerange.c
+site-packages/borg/platform/syncfilerange.pyx
+site-packages/borg/platform/windows.c
+site-packages/borg/platform/windows.pyx
 site-packages/borg/platform/xattr.py
 site-packages/borg/platformflags.py
 site-packages/borg/remote.py
@@ -283,6 +317,14 @@ site-packages
 site-packages/borg/testsuite/${MODPY_PYCACHE}patterns.${MODPY_PYC_MAGIC_TAG}pyc
 site-packages/borg/testsuite/${MODPY_PYCACHE}platform.${MODPY_PYC_MAGIC_TAG}${MODPY_PYOEXTENSION}
 site-packages/borg/testsuite/${MODPY_PYCACHE}platform.${MODPY_PYC_MAGIC_TAG}pyc
+site-packages/borg/testsuite/${MODPY_PYCACHE}platform_darwin.${MODPY_PYC_MAGIC_TAG}${MODPY_PYOEXTENSION}
+site-packages/borg/testsuite/${MODPY_PYCACHE}platform_darwin.${MODPY_PYC_MAGIC_TAG}pyc
+site-packages/borg/testsuite/${MODPY_PYCACHE}platform_freebsd.${MODPY_PYC_MAGIC_TAG}${MODPY_PYOEXTENSION}
+site-packages/borg/testsuite/${MODPY_PYCACHE}platform_freebsd.${MODPY_PYC_MAGIC_TAG}pyc
+site-packages/borg/testsuite/${MODPY_PYCACHE}platform_linux.${MODPY_PYC_MAGIC_TAG}${MODPY_PYOEXTENSION}
+site-packages/borg/testsuite/${MODPY_PYCACHE}platform_linux.${MODPY_PYC_MAGIC_TAG}pyc
+site-packages/borg/testsuite/${MODPY_PYCACHE}platform_posix.${MODPY_PYC_MAGIC_TAG}${MODPY_PYOEXTENSION}
+site-packages/borg/testsuite/${MODPY_PYCACHE}platform_posix.${MODPY_PYC_MAGIC_TAG}pyc
 site-packages/borg/testsuite/${MODPY_PYCACHE}remote.${MODPY_PYC_MAGIC_TAG}${MODPY_PYOEXTENSION}
 site-packages/borg/testsuite/${MODPY_PYCACHE}remote.${MODPY_PYC_MAGIC_TAG}pyc
 site-packages/borg/testsuite/${MODPY_PYCACHE}repoobj.${MODPY_PYC_MAGIC_TAG}${MODPY_PYOEXTENSION}
@@ -421,6 +463,10 @@ site-packages
 site-packages/borg/testsuite/nanorst.py
 site-packages/borg/testsuite/patterns.py
 site-packages/borg/testsuite/platform.py
+site-packages/borg/testsuite/platform_darwin.py
+site-packages/borg/testsuite/platform_freebsd.py
+site-packages/borg/testsuite/platform_linux.py
+site-packages/borg/testsuite/platform_posix.py
 site-packages/borg/testsuite/remote.py
 site-packages/borg/testsuite/repoobj.py
 site-packages/borg/testsuite/repository.py
@@ -431,6 +477,7 @@ site-packages
 site-packages/borg/version.py
 site-packages/borg/xattr.py
 site-packages/borgbackup-${MODPY_EGG_VERSION}.dist-info/
+site-packages/borgbackup-${MODPY_EGG_VERSION}.dist-info/AUTHORS
 site-packages/borgbackup-${MODPY_EGG_VERSION}.dist-info/LICENSE
 site-packages/borgbackup-${MODPY_EGG_VERSION}.dist-info/METADATA
 site-packages/borgbackup-${MODPY_EGG_VERSION}.dist-info/RECORD

[ThomasWaldmann]

Thanks for testing and review.

I guess the added source files are due to these lines:
61e96eb#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52L52

I didn't find a corresponding setting for pyproject.toml.

[ThomasWaldmann]

Found something: https://setuptools.pypa.io/en/stable/userguide/datafiles.html

Rather new setuptools seems to support this in pyproject.toml:

[tool.setuptools.packages.find]
where = ["src"]

[tool.setuptools.exclude-package-data]
mypkg = [".gitattributes"]

[awgcooper]

I've been using this for a month now on Debian 12 amd64, replicating two of my Borg 1.2x repos. Nothing to report other than there have been no problems and, subjectively speaking, it seems to be faster than 1.2x.

Dare I ask how close you think the final release may be?

[ThomasWaldmann]

Precise refcounting of chunks -> only "exists or not" and garbage collection, going away from transactions and log-like storage. That would simplify it a lot and could prepare for more parallelism in future. See the "breaking" issues for more about that.

[awgcooper]

Certainly does seem like a lot of work. Regarding this: #7377 (comment), you mention the need for a scalable fs. At a guess, which filesystems would and wouldn't work?

[ThomasWaldmann]

Depends a lot on the repo size and maybe also the size/count of input files (the latter could be worked around by not chunking individual files (as now), but the concatenated stream of all files' content). But guess older filesystems like *FAT and ext2/3 might struggle. Some issues (like too many files in a single directory) can be easily worked around.

[awgcooper]

What's your guesstimate for the maximum 'safe' repo size with Borg2, assuming you make these 'bigger changes'?

[ThomasWaldmann]

Can't say. The limit (if any) comes from the fs used.