Skip to content

WORKLOG.md

Latest commit

 

History

History
50 lines (41 loc) · 2.6 KB

WORKLOG.md

File metadata and controls

50 lines (41 loc) · 2.6 KB
Raw

CubbitSync — worklog

2026-08-12 — v2.0.0: real bidirectional sync

Done

  • Rewrote the app as a package (cubbitsync/) around a three-way reconciliation engine: local scan + remote listing + a SQLite baseline of the last agreement. Replaces v1's two rclone copy --ignore-existing shell scripts entirely.
  • Added rename detection (delete + create of identical content → server-side moveto), trash-on-delete with retention, conflict resolution with the losing version preserved, gitignore-style filters, safety caps, a plan-preview approval gate, persistent change history, and a headless --cli mode.
  • 66 unit tests, no third-party runner. build-deb.sh runs them before packaging.
  • Verified end-to-end with the real rclone binary: single-machine create / modify / rename / delete, two-machine convergence, conflict resolution, mass-deletion refusal. Read-only planning verified against a live bucket.

Decisions

  • Own engine rather than rclone bisync. Debian trixie ships rclone 1.60, whose bisync is the early beta: no --conflict-resolve, --resilient, --recover or --max-lock, and any interruption forces a full --resync. Not acceptable for a multi-gigabyte tree. The engine uses rclone purely as transport, so it works on the rclone that is actually installed.
  • Deletions default to a remote trash prefix, conflicts to newer-wins with the loser kept, first run to a dry-run preview behind an approval gate. All three are configurable.
  • --use-server-modtime on every listing. S3 keeps rclone's mtime in object metadata, which a LIST does not return — reading it costs one HEAD per object. Measured against a live bucket: >10 minutes for ~30k objects versus 5 seconds with server timestamps. Cross-side content comparison therefore uses the MD5 that S3 returns free in the ETag, never a local-vs-remote timestamp difference.

Findings worth remembering

  • Dependency and build directories dominate a typical sync root: on the tree this was developed against, the default filters removed roughly seven eighths of the local file count and a third of the remote objects. Filters are not cosmetic — preview before the first apply.
  • v1's scripts ran with --copy-links, so symlinked content already exists on the remote as regular objects. follow_symlinks therefore defaults to true; turning it off would make that content look locally deleted.

Next

  • Run Preview on the real pair and tune filters.txt before the first apply.
  • Consider a --resync-style "remote wins entirely" recovery action if a baseline is ever lost mid-transfer.