Add example application of TCP-over-ringbuffers by lmbollen · Pull Request #1214 · bittide/bittide-hardware

lmbollen · 2026-03-03T13:18:37Z

What (what did you do)
This pull request adds the necessary library components plus a demo application that showcases asynchronous commmunication between nodes over a bittide network.

With these library components we can add a networking layer between nodes that allow them to arbitrarily exchange data in a reliable way by leveraging the following techniques:

Aligned ringbuffers
The hardware architecture allows CPUs to communicate through a ringbuffer extraction that is shared with its neighbour. Through the transmitRingbuffer and receiveRingbuffer implemented in #1208 we now have a way to share unidirectional memory regions with our link partners.

Packetless transmission header
Since we simply share a memory region with our neighbor to which we can write and they can read, there is no notion of packet transmission and arrival, the data will just end up in memory accessible to our neighbor.
To still allow us to transfer packets, we made a minimal header that allows our neighbor to reliably determine if there is valid data in its memory.

The packet header consists of a checksum, sequence counter, length and payload. We use the checksum to see if there are new packets, the length to determine how long the packet is and for which bytes to calculate the checksum and the sequence counter ensures that when the same packet is send twice, the checksum changes so we can distinguish it from a stale packet.

TCP based communication statemachine
To still deal with data corruption, we leverage the TCP statemachine as offered by the smoltcp package to take care of reliable data transmission.

LinkInterface
Each node will have a LinkInterface abstraction that contains everything necessary to facilitate this asynchronous communication channel. It allows the CPU to send and receive arbitrary data to/from its direct neighbor.

Why (context, issues, etc.)
To be able to perform asynchronous communication in-band over the bittide network. Having an arbitrary asynchronous communication layer allows us to distribute programs, schedules and relevant parameters over the network.

Dear reviewer (anything you'd like the reviewer to pay close attention to?)
I use MaybeUnInit to deal with the self-referential nature of the smoltcp elements in LinkInterface, is this the way?
Is the LinkInterface API (over)complete?
I'm not sure if this WireEncode / WireDecode approach is the way to got. I saw the other way required you to use certain types u32<LE> for example.

AI disclaimer (heads-up for more than inline autocomplete)
AI has been used throughout the engineering of different approaches to implement this layer and used to combine all aspects into the LinkInterface.

TODO

~~Cross-out~~ any that do not apply

Write (regression) test
Update documentation, including docs/
Link to existing issue

hydrolarus

Took a first look at the Rust code changes

hydrolarus · 2026-06-09T10:56:36Z

@@ -0,0 +1,394 @@
+#![no_std]
+#![cfg_attr(not(test), no_main)]
+#![feature(sync_unsafe_cell)]


Does this crate actually need this feature? It doesn't seem to be used here anywhere?

It's for the *LOGGER.get()

hydrolarus · 2026-06-09T13:17:09Z

+            return None;
+        }
+        self.rx_buffer.buffer.set_enable(false);
+        core::sync::atomic::fence(core::sync::atomic::Ordering::AcqRel);


Our core doesn't support any atomic operations (it lacks the A extension), are fences supported and do what they should?

So this was a point of discussion @rslawson and I had and I even made an issue about it (which turned out to not be necessary as there was no real problem) smoltcp-rs/smoltcp#1134

I was not sure how I'd have to verify it, but as far as I'm aware this fence prevents the compiler from moving stores / loads across the fence boundary, preserving the order of accesses. I'll add a comment :)

martijnbastiaan

Round 1! I've got more review comments, but I've already accumulated 30. Paired with the ones from @hydrolarus that should already be quite a lot to handle.

martijnbastiaan

Thanks for applying the suggestions! I thought I had more, but I don't remember now..

Increase MU processing element instruction and data memory from 16KB to 160KB (10 RAMB18E2 blocks each) to accommodate the larger smoltcp-demo firmware binary. Refactor the RingBuffer simulation test infrastructure to load firmware binaries by name instead of building PeConfig from Haskell. Add simSmolTcp for simulating the smoltcp ring buffer test. Support multiple TX/RX ring buffer pairs in the DUT circuit. Remove IO from simResultRingBuffer return type.

Update all internal doc references to use the new hyphenated ring-buffer naming convention.

Replace the pinned git smoltcp 0.11 dependency in bittide-hal with crates.io smoltcp 0.12. Add the medium-ip feature needed for point-to-point links. Remove workspace lints inherited section. Fix ring_buffer_test to use the renamed device instance fields (transmit_ring_buffer_0, receive_ring_buffer_0) after the ring buffer HAL was updated to support multiple ring buffers per device.

Move MAC address utilities from smoltcp.rs into smoltcp/mac.rs and convert the smoltcp module from a single file to a directory module. Update smoltcp_client to use the new import path. Also update smoltcp_client to remove workspace lints, add medium-ip feature, and fix format string and mutability warnings.

Log state changes in the link startup state machine to aid debugging link bring-up issues on hardware.

Add modules for TCP-over-ring-buffer communication between nodes: - net_state: UGN graph types (UgnEdge, UgnReport) for collecting network state across nodes - smoltcp/ring_buffer: smoltcp Device implementation over aligned ring buffers with CRC integrity and sequence numbers - smoltcp/link_protocol: wire format types for inter-node commands and UGN edge data - smoltcp/link_interface: high-level LinkInterface combining smoltcp TCP with ring buffer transport, providing typed send/recv over point-to-point links Add crc and zerocopy dependencies. Enable generic_const_exprs feature gate. Remove workspace lints inherited section. Add medium-ip feature to smoltcp.

Add three new firmware crates: - smoltcp-demo: HITL demo for soft UGN collection over TCP links, with ring buffer alignment, DNA exchange, and UGN report aggregation via a manager/subordinate protocol - ring_buffer_smoltcp_test: simulation test for TCP over ring buffers - tcp_simultaneous_open_test: simulation test for TCP simultaneous open (both sides connect at the same time)

One demo needs significantly larger memories, this way we dont have to do that for each demo

We almost never use the whole target so it's nice to have separate names for these

hydrolarus

Seems fine! Some form suggestions

hydrolarus · 2026-06-18T09:39:58Z

+    fn socket(&self) -> &tcp::Socket<'a> {
+        self.sockets.get::<tcp::Socket>(self.socket_handle)
+    }
+
+    fn socket_mut(&mut self) -> &mut tcp::Socket<'a> {
+        self.sockets.get_mut::<tcp::Socket>(self.socket_handle)
+    }


Would it make sense to use Deref<Target = tcp::Socket<'a>> and DerefMut to get rid of a lot of the accessor functions below? (like is_open() or is_active())

hydrolarus · 2026-06-18T09:43:00Z

+        self.socket_mut()
+            .recv_slice(buffer.as_bytes_mut())
+            .unwrap_or(0);


this last .unwrap_or(0) doesn't seem to have any effect, it's just ignored right after. If you want to ignore the result then _ = self.socket_mut()... is the more intentional way to express this.

hydrolarus · 2026-06-18T09:55:36Z

+        // Get direct pointer to hardware buffer (base pointer points to array of [u8; 8])
+        let hw_buffer_ptr = self.rx_buffer.buffer.base_ptr() as *const u8;
+
+        unsafe {


This is a really large unsafe block, I'm sure not all of this needs to be in there, but I can't see from a glance what exactly needs this. So I think this should be split up into smaller blocks where they are actually needed.

lmbollen force-pushed the lucas/ringbuffer-smoltcp branch from ba365ce to faef6c3 Compare March 6, 2026 14:04

lmbollen force-pushed the lucas/use-ringbuffers branch from 149cc19 to 291bec5 Compare March 6, 2026 14:10

lmbollen force-pushed the lucas/ringbuffer-smoltcp branch from faef6c3 to db929ce Compare March 6, 2026 14:23

lmbollen force-pushed the lucas/use-ringbuffers branch 2 times, most recently from 3e47f7b to 0f244e8 Compare March 9, 2026 08:58

lmbollen force-pushed the lucas/ringbuffer-smoltcp branch from db929ce to bea8fa4 Compare March 9, 2026 09:01

lmbollen force-pushed the lucas/use-ringbuffers branch from 0f244e8 to 1786069 Compare March 9, 2026 09:15

lmbollen force-pushed the lucas/ringbuffer-smoltcp branch 2 times, most recently from 65b988c to 93e7bb7 Compare March 9, 2026 13:05

lmbollen force-pushed the lucas/use-ringbuffers branch from d0eb688 to 1c7a78c Compare March 9, 2026 13:06

lmbollen force-pushed the lucas/ringbuffer-smoltcp branch from 929107c to 4e85b86 Compare March 9, 2026 13:16

lmbollen force-pushed the lucas/use-ringbuffers branch from 1c7a78c to 0f17d8c Compare March 9, 2026 13:22

lmbollen force-pushed the lucas/ringbuffer-smoltcp branch 18 times, most recently from 07d563c to 89888e9 Compare March 11, 2026 16:47

lmbollen force-pushed the lucas/ringbuffer-smoltcp branch 7 times, most recently from d73fb01 to 6dba7e2 Compare April 8, 2026 09:52

lmbollen force-pushed the lucas/use-ringbuffers branch 5 times, most recently from 0bbd504 to 78f03dc Compare April 15, 2026 11:58

lmbollen force-pushed the lucas/ringbuffer-smoltcp branch from 09e1b80 to 1760fca Compare April 15, 2026 12:12

lmbollen force-pushed the lucas/use-ringbuffers branch from 3db706c to 7fd9729 Compare April 15, 2026 12:20

lmbollen mentioned this pull request Jun 1, 2026

Ringbuffer smoltcp device implementation #1139

Closed

5 tasks

hydrolarus reviewed Jun 9, 2026

View reviewed changes

martijnbastiaan requested changes Jun 12, 2026

View reviewed changes

martijnbastiaan mentioned this pull request Jun 12, 2026

Move fincFdecTests to software #1327

Merged

3 tasks

martijnbastiaan approved these changes Jun 16, 2026

View reviewed changes

Comment thread bittide-instances/src/Bittide/Instances/Tests/RingBuffer.hs Outdated

Comment thread bittide-instances/tests/Wishbone/RingBuffer.hs

Comment thread firmware-binaries/sim-tests/tcp_simultaneous_open_test/src/main.rs Outdated

lmbollen added 10 commits June 18, 2026 10:45

Rename ringbuffer-alignment doc to ring-buffer-alignment

3b211a4

Update all internal doc references to use the new hyphenated ring-buffer naming convention.

Add debug logging to link_startup state transitions

e2df10d

Log state changes in the link startup state machine to aid debugging link bring-up issues on hardware.

Add hitl setup module to rust to describe hitl configuration

0c7fd4a

Allow configuring management unit configuration per demo

5eed24b

One demo needs significantly larger memories, this way we dont have to do that for each demo

separate targets into hwTargets and deviceInfos in Drivers

b673a7d

We almost never use the whole target so it's nice to have separate names for these

hydrolarus reviewed Jun 18, 2026

View reviewed changes

Uh oh!

Conversation

lmbollen commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

hydrolarus left a comment

Choose a reason for hiding this comment

Uh oh!

hydrolarus Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

lmbollen Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hydrolarus Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

lmbollen Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

martijnbastiaan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

martijnbastiaan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hydrolarus left a comment

Choose a reason for hiding this comment

Uh oh!

hydrolarus Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

hydrolarus Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

hydrolarus Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lmbollen commented Mar 3, 2026 •

edited

Loading