feat: WAL-based RocksDB replication with HTTP streaming and failover

[jgusler-CXDO]

This relates to #35.

Summary:

• Adds a primary/secondary replication system using RocksDB's WAL (Write-Ahead Log) streamed over HTTP
• Secondary bootstraps from a full checkpoint on startup, then streams incremental WAL frames
• Failover is triggered via POST /_tuwunel/replication/promote — no process restart needed
• All replication endpoints are protected by a shared secret token

Test plan:

• Ran two Docker containers (primary on :8008, secondary on :8009)
• Secondary bootstrapped from primary checkpoint at seq 281 and began streaming
• Stopped primary with docker stop (graceful SIGTERM)
• Promoted secondary via curl — responded {"status":"promoted"}
• All messages from before the failover were present on the promoted instance
• Measured RPO ~0 on planned failover, RTO = seconds

Relevant config options added:

• rocksdb_primary_url — URL of primary for WAL streaming
• rocksdb_replication_token — shared secret for endpoint auth
• rocksdb_replication_interval_ms — heartbeat interval (default 250ms)

[pschichtel]

this implements async replication, so some dataloss is to be expected after an expected failover (node failure, disk failure process crash, ...), right?

[jgusler-CXDO]

@pschichtel yes, that is correct. Under normal write load, RPO is determined just by network RTT.

[jgusler-CXDO]

Hey @x86pup. It seems the CI failures on this PR are all runner-side cache issues. I am seeing two issues:

1: failed to create locked file '/opt/rust/cargo/debian/x86_64-linux-gnu/git/db/rust-rocksdb-eed8465c83fc7d81/config.lock': File exists; class=Os (2); code=Locked (-14)

2: could not open '/opt/rust/cargo/debian/x86_64-linux-gnu/git/checkouts/ruma-8006605ea0e2ea25/30d063c/crates/ruma-events/src/poll/unstable_start/unstable_poll_answers_serde.rs' for writing: No such file or directory; class=Os (2)

Can you clean these up and re-run?

[jevolk]

Hey @x86pup. It seems the CI failures on this PR are all runner-side cache issues. I am seeing two issues:

1: failed to create locked file '/opt/rust/cargo/debian/x86_64-linux-gnu/git/db/rust-rocksdb-eed8465c83fc7d81/config.lock': File exists; class=Os (2); code=Locked (-14)

2: could not open '/opt/rust/cargo/debian/x86_64-linux-gnu/git/checkouts/ruma-8006605ea0e2ea25/30d063c/crates/ruma-events/src/poll/unstable_start/unstable_poll_answers_serde.rs' for writing: No such file or directory; class=Os (2)

Can you clean these up and re-run?

These docker flakes sometimes occur when CI is really busy, apologies! We'll be happy to rerun as necessary.

[jevolk]

I haven't had a chance to thoroughly review this yet since I'm currently away, but a few things stand out as suspicious.

Foremost it's not clear why WAL streaming is necessary. RocksDB already has internal mechanisms to synchronize primary and secondary; all that's missing is the promotion signalling. What is the basis for concerning ourselves with binary framing of rocksdb inner-workings at the user level? Is the rocksdb synchronization API being invoked here? Perhaps I missed it...

[jgusler-CXDO]

@jevolk Yes, you are correct, TryCatchUpWithPrimary() handles sync internally. Correct me if I'm wrong, but it requires the secondary instance to open the primary's database directory directly. This works when both instances share the same filesystem (same machine or NFS mount).

In our case, we have a cluster of physical servers where each server has its own local disk. We can't use NFS/shared storage in our infrastructure. So core2 has no direct filesystem access to core1's RocksDB directory.

That's the gap we're trying to fill by replicating the WAL and SST files over the network, so core2 can stay in sync with core1 without shared storage. Once core2 has a local copy of the data, TryCatchUpWithPrimary() could potentially still be used if we mirror the primary's files locally, or we apply the WAL batches ourselves.

Is there a mechanism in RocksDB you'd recommend for this case, or is shared storage assumed in your deployment model?

[jevolk]

@jevolk Yes, you are correct, TryCatchUpWithPrimary() handles sync internally. Correct me if I'm wrong, but it requires the secondary instance to open the primary's database directory directly. This works when both instances share the same filesystem (same machine or NFS mount).

In our case, we have a cluster of physical servers where each server has its own local disk. We can't use NFS/shared storage in our infrastructure. So core2 has no direct filesystem access to core1's RocksDB directory.

That's the gap we're trying to fill by replicating the WAL and SST files over the network, so core2 can stay in sync with core1 without shared storage. Once core2 has a local copy of the data, TryCatchUpWithPrimary() could potentially still be used if we mirror the primary's files locally, or we apply the WAL batches ourselves.

Is there a mechanism in RocksDB you'd recommend for this case, or is shared storage assumed in your deployment model?

Alright so this is not limited to shared filesystem mounts, that's rather exciting actually. Keep up the good work 👍

[jgusler-CXDO]

@jevolk Thanks! I see it has passed all checks, what's the next step here?

[x86pup]

It needs to be thoroughly reviewed here especially since the usage of AI is apparent, Jason is on vacation and will get to it soon. Thank you for ensuring CI passes to help this along.

[jgusler-CXDO]

Okay sounds good, thanks for letting me know!

[jevolk]

Thank you for your patience 🙏 I'm right around the corner now...

[jevolk]

I'll be revisiting this again at the top of the 1.6.1 dev cycle (start of next week). I only have a small number of re-organizations and applying CBOR (which is hugely simplifying) so this should go in pretty early on. Thank you again for your patience 🙏🏻

[jevolk]

Hi Jack, I took a second swipe at this and unfortunately I just haven't gotten it to where it needs to be. I should have taken notes to provide some details since there's way too much nuance to summarize here.

The tldr is that I have to revisit this after some higher priority tasks- either next week for 1.6.1 or on the backside of that release. Overall I think this feature has promise and we're very close now.

[jgusler-CXDO]

Hey, no worries! Take your time. Let me know if there's anything I can do to help.

[jgusler-CXDO]

Hey @jevolk, do you think this should also handle replicating media files?

[jevolk]

Hey @jevolk, do you think this should also handle replicating media files?

If we were to put media in RocksDB (and I have before) we would very likely be disabling the WAL for those columns and write-ops. We would need a different mode of transport. Now that we have S3 storage provider support, we have more possibilities for media backup. In fact I'm considering the ability to backup the database itself over an S3 connection- though that would be for "colder" storage and wouldn't replace this feature for "hot" failover of course 😅

[jevolk]

Hey I haven't forgotten about this. There are just a few open things I want to get right and I also have tied it to some database txn improvements we need. In any case I'm in a better position to integrate this now compared to last month due to workflow efficiency improvements so I would in fact hold my breath for this feature 😉

[jgusler-CXDO]

Hey, that’s great to hear! But also no worries, I found some issues during testing and this latest commit is, hopefully, the fix.

[jgusler-CXDO]

Hey @jevolk, any update on this PR?

[jevolk]

This PR still has enough distance between where it is and where it has to be that it's only feasible to merge it as part of a larger initiative when more database work is called for in a campaign for clustering and horizontal scaling. We can't just casually drop this in and then be forced to make that sprint immediately, as the the focus right now is basic protocol compliance and completeness heading toward a 2.0 version. That might be the point when we can have all the nice toys like this I hope...

[jgusler-CXDO]

@jevolk Sounds good and makes sense. Any idea for an ETA for 2.0?

[meovary150]

Thanks for the continued work on this.

I’ve been following this PR and the related hot-failover / clustering discussion for quite a while. This feature is very important for us because we are evaluating Tuwunel for a production deployment where reliability, failover, and future horizontal scaling matter a lot.

I understand from the latest comments that this may need to be part of a larger database / clustering effort, possibly around the 2.0 roadmap. Do you have any rough timeline or milestone for when this kind of failover / clustering support might become available?

Even a rough direction would be very helpful for our planning. Thanks again for all the work on Tuwunel.

[jevolk]

Here's a visual one could impose a rough trend-line on heading toward 2.0: