sync: SVS v3 large-group revision (mhash, PARTIAL, announce+pull) by Taranum01 · Pull Request #190 · named-data/ndnd

Taranum01 · 2026-06-29T17:01:51Z

Summary

Implements the SVS v3 large-group revision (approved by Tianyuan Yu): inline FULL/PARTIAL sync, membership hash (mhash), and announce+pull recovery via published full vectors at 32=sv/<version>.

Event	`size ≤ threshold`	`size > threshold`
Publication	Inline FULL	Inline PARTIAL
Periodic sync	Inline FULL	Announce + pull
`mhash` mismatch	Pull or sender announce	Pull or sender announce

Backward compatible: SyncVectorThreshold defaults to 0 (legacy mode: StateVector-only wire, no mhash/VectorType, no announce+pull recovery — matches pre-revision DV prefix-table SVS). Large-group modes activate when threshold > 0.

Spec: docs/svs-v3-revision.md

Main changes

TLV: MemberSetHash, VectorType, SvsDataRef on SvsData (std/ndn/svs/v3)
Core: ComputeMhash, PARTIAL encode, publish at 32=sv, announce-only Sync, pull + merge (std/sync)
Example: std/examples/svs/large-sync (--threshold)
Tests: unit + in-process multi-node integration (std/sync/*_test.go, see svs_mesh_test.go)

Open item

Mixed-version interoperability (revised vs plain SVS v3 peers in one group) — spec §8; not addressed in this PR.

Test plan

go test ./std/sync/...
GitHub e2e (make e2e, sprint topology)
NFD smoke: 3× large-sync --threshold=80 on /ndn/svs/{alice,bob,carol} with multicast on /ndn/svs

Implements the approved large-group SVS revision with backward-compatible defaults (SyncVectorThreshold=0). Adds mhash and PARTIAL/announce+pull paths, 32=sv publish/pull recovery, tests, and a large-sync example. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Stop blind pulls to unpublished 32=sv prefixes on inline FULL mhash mismatch (spec §5.6: merge inline first). Clone SvMap before encoding outside the mutex to avoid concurrent map mutation panics. Co-authored-by: Cursor <cursoragent@cursor.com>

DV prefix-table SVS uses default SyncVectorThreshold=0. Recovery was still forcing announce-only sync on mhash mismatch, breaking step-2 prefix propagation while router reachability looked fine. Gate large-group recovery behind threshold > 0 for true legacy behavior. Co-authored-by: Cursor <cursoragent@cursor.com>

DV prefix-table sync uses default threshold 0. Emit StateVector-only SvsData (no mhash/VectorType) so step-2 SVS matches pre-revision behavior on the wire while large-group features stay behind threshold > 0. Co-authored-by: Cursor <cursoragent@cursor.com>

Adds topo.min.conf (6 nodes) and make e2e-local for reliable local validation on Mac Docker/Colima. Full sprint topo remains make e2e for CI. Co-authored-by: Cursor <cursoragent@cursor.com>

Publish the revision spec alongside the std/sync implementation in PR named-data#190. Co-authored-by: Cursor <cursoragent@cursor.com>

tianyuan129 · 2026-07-03T23:37:59Z

@@ -0,0 +1,16 @@
+# Small topology for local e2e (6 nodes, minimal link delays).


Can we not introduce new topologies in e2e tests?

Removed in 79b79ee — topo.min.conf, make e2e-local, and the docker script were local Mac dev helpers only. CI continues to use topo.sprint.conf.

tianyuan129 · 2026-07-03T23:38:22Z

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash


Can you explain why we need this extra things?

Removed in 79b79ee for the same reason — not needed in the upstream repo.

tianyuan129 · 2026-07-03T23:39:37Z

@@ -0,0 +1,82 @@
+package face


I am completely lost, we are revising the SVS protocol. Why invent the multicast face here?

Fair point. Removed std/engine/face/multicast_face.go in 79b79ee. The in-process mesh tests now use a test-only multicast hub in svs_mesh_test.go (same coverage, no new production face type).

tianyuan129 · 2026-07-03T23:46:36Z

+		}
+		svsData = buildAnnounceSvsData(stateSnap, ref)
+	} else {
+		svsData = buildSvsDataForSend(


Should use arg struct to avoid long parameter list.

Done in 79b79ee — buildSvsDataForSend now takes a svsSendInput struct.

tianyuan129 · 2026-07-03T23:48:36Z

+			mtimeSnap,
+		)
+	}
+	logSyncSend(s, reason, svsData)


Is this a leftover from your debugging?

Yes — removed in 79b79ee. Deleted svs_debug.go and the logSyncSend / logSyncRecv calls on the send/recv path.

tianyuan129 · 2026-07-04T00:08:17Z

+	}
+	return exceedsSyncThreshold(threshold, inlineFullSize(state))
+}
+


I worry those helper functions are not helping with readability and mainainability.

Addressed in 79b79ee — merged svs_announce.go into svs_pull.go (announce+pull recovery) and svs_encode.go (inline FULL/PARTIAL build) to reduce file sprawl.

tianyuan129 · 2026-07-04T00:10:40Z

@@ -0,0 +1,89 @@
+package sync


I don't see it is necessary to have this file -- if this is only for debugging.

Removed svs_debug.go entirely in 79b79ee.

tianyuan129 · 2026-07-04T00:26:24Z

@@ -0,0 +1,157 @@
+package sync


Can you describe what this test is about?

Documented in 79b79ee — renamed to svs_mesh_test.go with a file-level comment. It is an in-process multi-node SVS harness (real engine + SvSync per node, test-only multicast hub) that exercises small-group sync, PARTIAL publication, and announce+pull recovery without mininet/NFD.

I don't see the necessity of this since we already have the e2e tests in CI

Remove local-only e2e topology and docker runner. Drop svs_debug and production multicast_face; keep mesh integration tests with a test-only hub in svs_mesh_test.go. Merge announce helpers into svs_pull, use svsSendInput for buildSvsDataForSend, and remove debug send logging. Co-authored-by: Cursor <cursoragent@cursor.com>

Taranum01 · 2026-07-04T16:54:51Z

Thanks for the review, @tianyuan129 — addressed in 79b79ee:

Removed local-only e2e topology / docker runner (topo.min.conf, e2e-local, e2e-local-docker.sh)
Removed production multicast_face; mesh integration tests use a test-only hub in svs_mesh_test.go
Removed svs_debug.go and debug send/recv logging
buildSvsDataForSend now takes a svsSendInput struct
Merged svs_announce.go into svs_pull.go + svs_encode.go

Also updated the PR description: SyncVectorThreshold=0 uses legacy StateVector-only wire (no mhash), which preserves DV prefix-table behavior and is what fixed e2e.

go test ./std/sync/... passes locally; CI should re-run on the new commit.

Remove dead large-sync debug flag and tuName helper, clarify threshold-0 comments, trim pull-path debug logging, rename announce tests to svs_pull_test, and test pullRefFromSyncDataWire with a signed Sync Data wire. Co-authored-by: Cursor <cursoragent@cursor.com>

Taranum01 · 2026-07-04T17:10:23Z

Pushed 60474b3 — minor style polish only (no logic changes): aligned `large-sync` with other SVS examples, clarified threshold-0 comments, trimmed pull-path debug logs, renamed `svs_announce_test.go` → `svs_pull_test.go`, and fixed `TestPullRefFromSyncDataWire` to use a real Sync Data wire.

Co-authored-by: Cursor <cursoragent@cursor.com>

Taranum01 · 2026-07-06T08:00:37Z

@tianyuan129 Pushed ffcc890 — fixes the test/lint failure (goimports column alignment in std/sync/svs_mesh_test.go). Could you re-run CI / re-review when convenient? No logic changes; only the field alignment in meshMulticastFace was touched.

tianyuan129 · 2026-07-08T22:05:51Z

@@ -0,0 +1,89 @@
+package main


If we did not change the Sync API, I do not see the necessity of having a separate example.

tianyuan129 · 2026-07-08T22:07:44Z

+
+	// SyncVectorThreshold is the max inline SvsData size (bytes).
+	// 0 = legacy mode (StateVector-only wire, no mhash or announce+pull recovery).
+	SyncVectorThreshold int


We don't need to be backward compatiable.

tianyuan129 · 2026-07-08T22:41:06Z


 		recvSv: make(chan svSyncRecvSvArgs, 128),

+		fullVectorPrefix: deriveFullVectorPrefix(opts.SyncDataName),


The later deriveFullVectorPrefix merely replaces the last name component with another keyword, so I ams thinking about making fullVectorPrefix as another configurable in parallel to SyncDataName.

tianyuan129 · 2026-07-08T22:44:09Z

+// buildLegacySvsData is the pre-revision SVS v3 wire format used when threshold is 0.
+func buildLegacySvsData(state SvMap[uint64]) *spec_svs.SvsData {
+	sv := state.Encode(func(seq uint64) uint64 { return seq })
+	return &spec_svs.SvsData{StateVector: sv}
+}
+
+// buildInlineSvsData constructs inline SvsData (mhash + VectorType + StateVector).
+func buildInlineSvsData(state SvMap[uint64], vectorType uint64, sv *spec_svs.StateVector) *spec_svs.SvsData {
+	return &spec_svs.SvsData{
+		MemberSetHash: ComputeMhash(state),
+		VectorType:    optional.Some(vectorType),
+		StateVector:   sv,
+	}
+}
+
+// inlineSvsDataSize returns the encoded byte length of SvsData content.
+func inlineSvsDataSize(data *spec_svs.SvsData) int {
+	return len(data.Encode().Join())
+}
+
+// exceedsSyncThreshold reports whether size is over the configured inline budget.
+// threshold 0 disables the large-group size limit (legacy mode).
+func exceedsSyncThreshold(threshold, size int) bool {
+	return threshold > 0 && size > threshold
+}
+
+// buildInlineFullFromState builds inline FULL SvsData for the complete local state.
+func buildInlineFullFromState(state SvMap[uint64]) *spec_svs.SvsData {
+	sv := state.Encode(func(seq uint64) uint64 { return seq })
+	return buildInlineSvsData(state, spec_svs.VectorTypeFull, sv)
+}
+
+// inlineFullSize returns encoded inline FULL SvsData size for threshold checks.
+func inlineFullSize(state SvMap[uint64]) int {
+	return inlineSvsDataSize(buildInlineFullFromState(state))
+}


Is there a reason that we must have those tiny helper functions?

I don't know....I prefer inline them for readability.

tianyuan129 · 2026-07-08T22:49:07Z

@@ -0,0 +1,157 @@
+package sync


I don't see the necessity of this since we already have the e2e tests in CI

tianyuan129 · 2026-07-08T22:52:11Z

+// membershipTuple is one (Name, BootstrapTime) pair in the sync group.
+type membershipTuple struct {
+	name enc.Name
+	boot uint64
+}


Making this a separate TLV structure

Tuple-T Tuple-L, [Name, Boot]

so that you can use the ndnd standard TLV codec to simplify code.

tianyuan129 · 2026-07-08T22:54:29Z

@@ -0,0 +1,157 @@
+package sync


We should put all svs related tests into one file.

Sync API for SvSync is unchanged; pure-sync suffices for svs examples.

Merge svs_encode_test.go, svs_mhash_test.go, svs_pull_test.go into the single svs_test.go. svsmap tests remain separate (testing the generic SvMap type, not the svs protocol). svsmat tests are dropped next.

The mesh-based end-to-end tests in svs_mesh_test.go are redundant with the GitHub e2e tests (sprint topology) and add an in-process hub plus test-only face type to the package. Remove them and the helper functions in svs_test.go that depended on the mesh.

Fold buildLegacySvsData, buildInlineSvsData, inlineSvsDataSize, exceedsSyncThreshold, buildInlineFullFromState and inlineFullSize into the call sites that actually use them. The inline FULL/PARTIAL SvsData structs are now visible at the build sites, matching the rest of the encode path.

Add SvSyncOpts.FullVectorPrefix next to SyncDataName. When set, it is used directly as the publish/serve prefix for retrievable FULL StateVector Data used by announce+pull recovery. When unset, the previous default (SyncDataName with trailing "32=svs" -> "32=sv") is preserved.

Introduce spec_svs.MembershipTuple (Tuple-T 0xcc with Name and BootstrapTime) in the v3 TLV definitions and use the standard TLV codec in ComputeMhash instead of inlining a bespoke 0xD4 TLV byte.

All Sync messages now carry mhash and VectorType. The legacy StateVector-only wire (and the threshold-0 paths that guarded it) is removed from encode, shouldUseAnnouncePull and handleMhashMismatch. SyncVectorThreshold defaults to 1200 bytes when 0 is supplied; the gating is preserved for switching between inline FULL / PARTIAL / announce+pull in large groups. The spec's §4.3 and §8 are updated to document the no-backward-compat posture.

The previous batch of commits touched svs_test.go and svs_mhash.go but left them non-canonical for gofmt (trailing blank line in svs_mhash.go) and goimports (third-party testify imports not separated from local named-data/ndnd imports in svs_test.go). Apply both.

Taranum01 changed the base branch from main to psvs June 29, 2026 17:07

Taranum Wasu and others added 6 commits June 29, 2026 22:43

fix lint: sort imports in svs_mhash_test.go

8540360

Co-authored-by: Cursor <cursoragent@cursor.com>

e2e: add minimal local topology and docker runner script

2b36d34

Adds topo.min.conf (6 nodes) and make e2e-local for reliable local validation on Mac Docker/Colima. Full sprint topo remains make e2e for CI. Co-authored-by: Cursor <cursoragent@cursor.com>

docs: add SVS v3 large-group revision specification

05623e2

Publish the revision spec alongside the std/sync implementation in PR named-data#190. Co-authored-by: Cursor <cursoragent@cursor.com>

tianyuan129 requested changes Jul 4, 2026

View reviewed changes

Taranum Wasu and others added 2 commits July 4, 2026 22:53

style: remove AI-generated comment from large-sync example

7323078

Co-authored-by: Cursor <cursoragent@cursor.com>

fix lint: align meshMulticastFace field columns

ffcc890

Co-authored-by: Cursor <cursoragent@cursor.com>

tianyuan129 requested changes Jul 8, 2026

View reviewed changes

Taranum Wasu added 8 commits July 9, 2026 12:52

examples: drop large-sync example

bdaecba

Sync API for SvSync is unchanged; pure-sync suffices for svs examples.

test: consolidate svs tests into svs_test.go

4c8926b

Merge svs_encode_test.go, svs_mhash_test.go, svs_pull_test.go into the single svs_test.go. svsmap tests remain separate (testing the generic SvMap type, not the svs protocol). svsmat tests are dropped next.

sync: encode membership tuple as standard TLV

e7e87bf

Introduce spec_svs.MembershipTuple (Tuple-T 0xcc with Name and BootstrapTime) in the v3 TLV definitions and use the standard TLV codec in ComputeMhash instead of inlining a bespoke 0xD4 TLV byte.

Taranum01 force-pushed the svs-exploration branch from ad4f144 to f0724bf Compare July 9, 2026 08:07

		@@ -0,0 +1,16 @@
		# Small topology for local e2e (6 nodes, minimal link delays).


		recvSv: make(chan svSyncRecvSvArgs, 128),

		fullVectorPrefix: deriveFullVectorPrefix(opts.SyncDataName),

Uh oh!

Conversation

Taranum01 commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Main changes

Open item

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Taranum01 commented Jul 4, 2026

Uh oh!

Taranum01 commented Jul 4, 2026

Uh oh!

Taranum01 commented Jul 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Taranum01 commented Jun 29, 2026 •

edited

Loading