Ugn grooming by lmbollen · Pull Request #1318 · bittide/bittide-hardware

lmbollen · 2026-06-03T10:31:36Z

What (what did you do)
Implements UGN grooming for the wire demo: restoring a freshly-booted bittide system to a stored, known-good latency profile (λ^safe) so the application's fixed schedule lands identically every boot, regardless of boot-time counter offsets.

The work is built bottom-up:

Generic weighted directed-graph library (Bittide.Graph.Weighted) with Bellman-Ford potentials / negative-cycle detection.
UGN grooming algorithm (Bittide.ClockControl.Ugn.Grooming) on top of the graph — computes the per-node relabel q and per-link frame corrections that map an observed boot onto λ^safe.
Two Wishbone peripherals: TimedReset (releases the application reset at a software-chosen local-counter cycle — the relabel point) and UgnCorrections (host→MU channel for per-link frame corrections).
Host/MU integration: the wire-demo driver reads the hardware UGNs over GDB, grooms this boot onto the stored λ^safe (Bellman-Ford), writes the corrections + reset-release timing; the management-unit firmware applies each correction to its elastic buffer as a single atomic Wishbone write. An app_counter readback lets the host verify on hardware that localCounter − app_counter == tReset per node.
Hedgehog property test for the relabel-restoration invariant, plus unit tests for the graph and TimedReset.

Why (context, issues, etc.)
After a boot, each node's UGNs carry arbitrary boot-time counter offsets and elastic buffers might settle at different occupancies, so a fixed application schedule can't be reused across boots. Grooming relabels every boot onto a single stored reference profile, absorbing the offsets via the timed-reset release and trimming the residual per-link drift (a few frames) inside the elastic buffer's safe range. This is the host-side groundwork for reproducible wire-demo scheduling.

Dear reviewer (anything you'd like the reviewer to pay close attention to?)
Commit-by-commit review is intended — the history is structured so each commit introduces one coherent piece and compiles on its own (graph → algorithm → peripherals → MU wiring → HITL utils → driver → firmware → readback → buffer tuning).

λ^safe is stored in the raw (non-canonicalized) gauge in the Driver.

AI disclaimer (heads-up for more than inline autocomplete)
I guided claude through the implementation of this PR based on the presentation offered by Sanjay regarding the method.

TODO

~~Cross-out~~ any that do not apply

Write (regression) test
Update documentation, including docs/
Link to existing issue

google-cla · 2026-06-03T10:31:52Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

A small directed, weighted graph backed by a Data.Map adjacency structure, with the Bellman-Ford shortest-path / negative-cycle machinery (potentials, relabel, negativeCycle, cycleSum). Node and weight types are generic. This is the reusable foundation for UGN grooming: deciding whether a set of UGNs is physically allowed reduces to "no negative cycle". Kept standalone (no UGN specifics) so it can be tested and reused independently.

UGNs are signed per-link clock/frame offsets (edge weights). They are physically allowed iff the sum around every directed cycle is nonnegative, and one boot can be relabeled onto another iff a difference-constraint system is feasible. Both reduce to the negative-cycle / potential machinery from Bittide.Graph.Weighted. Provides isAllowed and groomCorrection (Infeasible | Feasible{correction, frames}), using Integer weights to avoid overflow during relaxation. Pure and demo-agnostic.

A small Wishbone peripheral with one management-unit-writable register (release_cycle, default maxBound, with a description) that takes a local counter and holds its output reset asserted until localCounter > release_cycle, then deasserts (monotonic counter, so it stays deasserted). This is how the large boot-time counter offset (the grooming relabel q, far too big for the depth-bounded elastic buffer) is realised: gating a domain's reset on a chosen cycle sets that domain's counter origin. Generic over the Wishbone width; unit test verifies the reset deasserts exactly at the written cycle.

A small Wishbone scratch peripheral with two read/write registers: a vector of n signed per-link corrections (init 0) and a valid flag (init False). The host computes the per-link grooming frame corrections and writes them here over GDB; the management-unit CPU polls valid, reads the corrections and applies them to its elastic buffers. Plain bus-accessible scratch (no fabric writes), mirroring the CaptureUgn idiom.

… the user core Add both host-facing grooming peripherals to the generic management unit (NmuInternalBusses 3 -> 5): - UgnCorrections: the host writes per-link frame corrections for the MU CPU to apply. - TimedReset: the management unit chooses the local-counter cycle at which the application leaves reset (the boot-offset relabel). It lives here, in the layer around the user core, and exposes an `appReset` output. `core` routes `appReset` into the user core via a new `UserCoreCircuit` argument. The wire demo's user core uses it as the relabeled time base: an application counter that starts at release and drives the programmable mux (UserCoreBusses 3 -> 2, since the timed reset no longer sits on a user-core bus). The application counter stays in the user core; the soft-UGN user core ignores the reset.

Host-side helpers for the grooming flow, placed beside the existing HITL utilities: GDB-based component HAL accessors (readCurrentTime, writeReleaseCycle, writeCorrections, readHardwareUgns, readAppCounter) in Utils.Gdb, and the UgnEdge conversions (ugnEdges, hardwareUgnEdges) in Utils.Ugn.

After printing the captured UGNs, the management unit polls the UgnCorrections peripheral's valid flag, then applies each link's frame correction to its elastic buffer as a single atomic wishbone write with hardware ack (rather than one frame at a time). A per-frame loop is vulnerable to concurrent Callisto adjustments between iterations, which shrink the net data_count change; a single write limits the Callisto race window to at most one adjustment, keeping the error to ±1 frame. It self-reports a success line the host waits on. The firmware half of the grooming apply path; the corrections values and the reset-release timing are computed and written by the host.

…rdware Diagnostic instrumentation to pin down why the canonical-gauge grooming breaks the XOR chain partway (nodes 0-2 pass, 3-7 fail) despite the schedule math being equivalent to main's proven generateSchedule and the frame path being exercised by an earlier green run. - UserCore: expose the application counter via a read-only Wishbone register (new AppCounterReadback device, UserCoreBusses 2 -> 3). With the app running, localCounter - app_counter == tReset for a node iff its timed-reset relabel released at the intended (relabeled) cycle. - Driver: after the app runs, read each node's app_counter + localCounter and print loc-app vs tReset, so a divergence pinpoints which node's reset did not land. This is instrumentation, not the fix; it isolates whether the failure is the Bellman-Ford reset relabel not landing on-device vs the frame corrections.

…golden UGNs for FifoSize=6 midpoint

martijnbastiaan

We've run out of time on this one unfortunately. But here's all the feedback we've collected so far.

martijnbastiaan · 2026-06-16T14:32:54Z

+-- SPDX-License-Identifier: Apache-2.0
+
+{- |
+A small Wishbone peripheral that releases a reset at a software-chosen cycle.


This is a duplicate of ProgrammableMux. (Not exactly, but you know..)

martijnbastiaan · 2026-06-16T14:33:40Z

+{- |
+UGN grooming, expressed on top of "Bittide.Graph.Weighted".
+
+UGNs (@\lambda_{i->j}@) are signed per-link clock/frame offsets, i.e. edge weights


The LaTeX style documentation doesn't do it for me: it neither looks good in the source code, nor does it render to anything with Haddock. I'd prefer it if it either:

Renders properly in Haddock

Looks good in the source through use of unicode (unicode in documentation is a-ok).

martijnbastiaan · 2026-06-16T14:34:58Z

+that the UGNs changed too much. (Whether a set of UGNs is physically realisable at all —
+no negative cycle — is 'Bittide.Graph.Weighted.negativeCycle'.)
+
+The math (see the UGN grooming slides): with slack


What slides? IMO we should just drop the PDF in the repo, but we should ask @sanjaylall for permission.

martijnbastiaan · 2026-06-16T14:36:57Z

+-- SPDX-FileCopyrightText: 2026 Google LLC
+--
+-- SPDX-License-Identifier: Apache-2.0


The commit message's body is weird: it doesn't assume the reader is familiar with UGNs, but it does just drop jargon like "difference-constraint system is feasible". Who is your target audience?

The last paragraph can be removed, it doesn't belong in the commit message body.

martijnbastiaan · 2026-06-16T14:39:07Z

+This module provides 'groomCorrection': given a freshly measured @\lambda@ and a
+previously stored @\lambda^{safe} = \lambda^{obs} + \varepsilon@, it finds a relabeling
+@q@ and the frames to insert so the system is restored to @\lambda^{safe}@, or reports
+that the UGNs changed too much. (Whether a set of UGNs is physically realisable at all —
+no negative cycle — is 'Bittide.Graph.Weighted.negativeCycle'.)


This is way too short/jargony to be useful to the average digital designer. I think you should pull in more context on what UGN grooming is and why we're doing all this. Preferably write it yourself, not Claude. In my mind a good outline would be:

What is UGN / UGN pair

What differs between boots

Why is that bad

Why can't you just relabel

Explain that we found a trick (+\varepsilon) which happens to get a set of equations we can easily solve using well-known algorithms.

Mention that this is just a solution, but perhaps we'll find a "better" one in the future (e.g., one that can also remove frames).

martijnbastiaan · 2026-06-26T07:54:59Z

+{- | Golden UGN graph: a full set of per-link UGNs (@λ = local - remote@) captured from a
+passing CI boot of this rig, stored as a 'UgnEdge' list keyed by DNA-derived node id and
+port. This is the stored reference every boot is groomed onto.
+
+The UGNs carry their boot-time counter offsets; 'lambdaSafe' relabels them into a
+small, non-negative gauge before use.
+-}
+goldenUgns :: [UgnEdge]
+{- FOURMOLU_DISABLE -}
+goldenUgns = canonicalizeUgn


Not really a fan of this approach, given that bittide is still under development. Can it be replaced by a (nightly) test that just does two wire demos -- where the second one reuses the schedule from the first one?

martijnbastiaan · 2026-06-26T08:01:40Z

+          -- Relabel this boot onto the stored target latencies, using the grooming
+          -- utilities ("Bittide.Instances.Hitl.Utils.UgnGrooming"): build UGN
+          -- edges from the measured counters, groom them (Bellman-Ford) onto
+          -- λ^safe = the target latency on both directions of every chain hop, and turn
+          -- the result into per-node reset releases (which remove the boot-time counter
+          -- offsets) and per-link frame corrections (which raise each link's latency to
+          -- the target). The application then runs one FIXED schedule built from the
+          -- target latencies -- identical every run.
          currentTime <-


Suggested change

-- Relabel this boot onto the stored target latencies, using the grooming

-- utilities ("Bittide.Instances.Hitl.Utils.UgnGrooming"): build UGN

-- edges from the measured counters, groom them (Bellman-Ford) onto

-- λ^safe = the target latency on both directions of every chain hop, and turn

-- the result into per-node reset releases (which remove the boot-time counter

-- offsets) and per-link frame corrections (which raise each link's latency to

-- the target). The application then runs one FIXED schedule built from the

-- target latencies -- identical every run.

currentTime <-

-- Relabel this boot onto the stored target latencies, using the grooming

-- utilities ("Bittide.Instances.Hitl.Utils.UgnGrooming").

currentTime <-

martijnbastiaan · 2026-06-26T08:04:44Z

+            -- large constant. 'StartDelay' must exceed the GDB bookkeeping time, else a
+            -- node's reset could release before it is fully configured.


Suggested change

-- large constant. 'StartDelay' must exceed the GDB bookkeeping time, else a

-- node's reset could release before it is fully configured.

-- large constant.

This is already documented at StartDelay.

martijnbastiaan · 2026-06-26T08:06:16Z

+            -- This boot's measured UGN edges for the FULL graph (every node, every
+            -- link) -- the grooming is independent of the application and adjusts all
+            -- elastic buffers.
+            measuredEdges = hardwareUgnEdges ugnPairsTableV


Suggested change

-- This boot's measured UGN edges for the FULL graph (every node, every

-- link) -- the grooming is independent of the application and adjusts all

-- elastic buffers.

measuredEdges = hardwareUgnEdges ugnPairsTableV

measuredEdges = hardwareUgnEdges ugnPairsTableV

martijnbastiaan · 2026-06-26T08:07:18Z

+          -- Groom this boot's measured UGNs onto the stored 'lambdaSafe' (Bellman-Ford):
+          -- per-node reset offsets + per-link frame corrections for every edge.
+          -- 'appSchedule' is fixed and derived from the same 'lambdaSafe'.
+          relabelPlan <- case computeRelabel measuredEdges lambdaSafe of


Suggested change

relabelPlan <- case computeRelabel measuredEdges lambdaSafe of

This is already well-documented in the functions

lmbollen force-pushed the lucas/ugn-grooming branch 5 times, most recently from c4256c2 to adf98b1 Compare June 8, 2026 09:23

lmbollen force-pushed the lucas/ugn-grooming branch 3 times, most recently from 92f9673 to e723a27 Compare June 15, 2026 07:32

lmbollen added 7 commits June 15, 2026 10:51

Wire demo driver: add host-side UGN grooming via Bellman-Ford

5e6c996

lmbollen force-pushed the lucas/ugn-grooming branch from e723a27 to 9611e63 Compare June 15, 2026 11:34

lmbollen marked this pull request as ready for review June 15, 2026 11:50

lmbollen requested a review from hiddemoll June 15, 2026 11:50

lmbollen added 3 commits June 15, 2026 13:55

Wire demo: deepen elastic buffer to 64 entries; set margin and shift …

9f65e93

…golden UGNs for FifoSize=6 midpoint

lmbollen force-pushed the lucas/ugn-grooming branch from 9611e63 to 9f65e93 Compare June 15, 2026 11:55

Add ugn-grooming chapter to mdbook

6a701a8

martijnbastiaan requested changes Jun 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Ugn grooming - #1318

Ugn grooming#1318
lmbollen wants to merge 11 commits into
mainfrom
lucas/ugn-grooming

lmbollen commented Jun 3, 2026 •

edited

Loading

Uh oh!

google-cla Bot commented Jun 3, 2026

Uh oh!

martijnbastiaan left a comment

Uh oh!

martijnbastiaan Jun 16, 2026

Uh oh!

martijnbastiaan Jun 16, 2026

Uh oh!

martijnbastiaan Jun 16, 2026

Uh oh!

martijnbastiaan Jun 16, 2026

Uh oh!

martijnbastiaan Jun 16, 2026

Uh oh!

martijnbastiaan Jun 26, 2026

Uh oh!

martijnbastiaan Jun 26, 2026

Uh oh!

martijnbastiaan Jun 26, 2026

Uh oh!

martijnbastiaan Jun 26, 2026

Uh oh!

martijnbastiaan Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		-- large constant. 'StartDelay' must exceed the GDB bookkeeping time, else a
		-- node's reset could release before it is fully configured.

	-- large constant. 'StartDelay' must exceed the GDB bookkeeping time, else a
	-- node's reset could release before it is fully configured.
	-- large constant.

Uh oh!

Conversation

lmbollen commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

google-cla Bot commented Jun 3, 2026

Uh oh!

martijnbastiaan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lmbollen commented Jun 3, 2026 •

edited

Loading