Flush returning "Receive: netlink receive: recvmsg: no buffer space available"

[sontumzmdr]

I observed that when we combine multiple messages and call Flush, the Flush may fail (with error "Receive: netlink receive: recvmsg: no buffer space available") if the message payload crosses netlink buffer size.
Currently Flush() doesn't handle it? is application supposed to handle it? is there any API exposed from which application can know if the netlink message size is exceeded?

[stapelberg]

Which git commit version of google/nftables are you using?

I’m asking because @nickgarlis did some work in this area recently, and I’m wondering if you’re reporting behavior based on that, or based on an older version before that.

[sontumzmdr]

I tried using both v0.3 (github.com/google/nftables v0.3.0) and as well latest master (github.com/google/nftables v0.0.0-20220607152305-1f0380f5c76e), both has same behavior, Flush fails if message payload size is big.

[nickgarlis]

@sontumzmdr Could you share the exact message you're sending (ideally a code snippet) ? I could have a look later today.

[sontumzmdr]

@nickgarlis I wrote a test code to simulate the issue. (the rule content of the file is not relevant here; I just put some payloads for test)

code: test.go file

package main

import (
	"encoding/binary"
	"fmt"
	"log"
	"net"
	"os"
	"strconv"

	"github.com/google/nftables"
	"github.com/google/nftables/expr"
	"golang.org/x/sys/unix"
)

func main() {
	// Get rule count from command line argument
	ruleCount := 5 // Default value
	if len(os.Args) > 1 {
		count, err := strconv.Atoi(os.Args[1])
		if err != nil {
			log.Fatalf("Invalid rule count argument: %v. Usage: %s <rule_count>", err, os.Args[0])
		}
		if count < 1 {
			log.Fatalf("Rule count must be at least 1")
		}
		ruleCount = count
	}

	fmt.Printf("Creating %d rules...\n", ruleCount)

	conn, err := nftables.New()
	if err != nil {
		log.Fatalf("Failed to create nftables connection: %v", err)
	}

	// Create table
	table := conn.AddTable(&nftables.Table{
		Family: nftables.TableFamilyINet,
		Name:   "test_custom_chain",
	})

	// Create custom chain
	chain := conn.AddChain(&nftables.Chain{
		Name:     "my_rules",
		Table:    table,
		Type:     nftables.ChainTypeFilter,
		Hooknum:  nftables.ChainHookInput,
		Priority: nftables.ChainPriorityFilter,
	})

	// Create accept verdict chain
	acceptChain := conn.AddChain(&nftables.Chain{
		Name:  "accept",
		Table: table,
	})

	// Add accept rule to accept chain
	conn.AddRule(&nftables.Rule{
		Table: table,
		Chain: acceptChain,
		Exprs: []expr.Any{
			&expr.Verdict{
				Kind: expr.VerdictAccept,
			},
		},
	})

	// Define base IP ranges and ports (will be incremented for each rule)
	type ruleSpec struct {
		saddrStart net.IP
		saddrEnd   net.IP
		daddrStart net.IP
		daddrEnd   net.IP
		dportStart uint16
		dportEnd   uint16
	}

	// Add multiple rules based on count
	for i := 0; i < ruleCount; i++ {
		// Increment IP addresses and ports for each rule
		// Use simpler incremental values to avoid overflow
		spec := ruleSpec{
			saddrStart: net.ParseIP(fmt.Sprintf("1111:1234:5678:9abc:def0::%x", i+1)),
			saddrEnd:   net.ParseIP(fmt.Sprintf("1111:4321:8765:cba9:fed:ffff:%x:ffff", i)),
			daddrStart: net.ParseIP(fmt.Sprintf("1170:1234:5678:9abc:def0::%x", i+1)),
			daddrEnd:   net.ParseIP(fmt.Sprintf("1170:4321:8765:cba9:fed:ffff:%x:ffff", i)),
			dportStart: uint16(11309 + i*2),
			dportEnd:   uint16(11310 + i*2),
		}

		// Add rule: ip6 saddr <range> ip6 daddr <range> th dport <range> jump accept
		conn.AddRule(&nftables.Rule{
			Table: table,
			Chain: chain,
			Exprs: []expr.Any{
				// Load IPv6 source address
				&expr.Payload{
					DestRegister: 1,
					Base:         expr.PayloadBaseNetworkHeader,
					Offset:       8,  // IPv6 source address offset
					Len:          16, // IPv6 address length
				},
				// Check if saddr >= saddrStart
				&expr.Cmp{
					Op:       expr.CmpOpGte,
					Register: 1,
					Data:     spec.saddrStart.To16(),
				},
				// Check if saddr <= saddrEnd
				&expr.Cmp{
					Op:       expr.CmpOpLte,
					Register: 1,
					Data:     spec.saddrEnd.To16(),
				},

				// Load IPv6 destination address
				&expr.Payload{
					DestRegister: 1,
					Base:         expr.PayloadBaseNetworkHeader,
					Offset:       24, // IPv6 destination address offset
					Len:          16, // IPv6 address length
				},
				// Check if daddr >= daddrStart
				&expr.Cmp{
					Op:       expr.CmpOpGte,
					Register: 1,
					Data:     spec.daddrStart.To16(),
				},
				// Check if daddr <= daddrEnd
				&expr.Cmp{
					Op:       expr.CmpOpLte,
					Register: 1,
					Data:     spec.daddrEnd.To16(),
				},

				// Load layer 4 protocol
				&expr.Meta{
					Key:      expr.MetaKeyL4PROTO,
					Register: 1,
				},
				// Check if protocol is TCP or UDP (for port matching)
				&expr.Cmp{
					Op:       expr.CmpOpEq,
					Register: 1,
					Data:     []byte{unix.IPPROTO_TCP}, // Change to IPPROTO_UDP if needed
				},

				// Load destination port
				&expr.Payload{
					DestRegister: 1,
					Base:         expr.PayloadBaseTransportHeader,
					Offset:       2, // Destination port offset
					Len:          2, // Port length
				},
				// Check if dport >= dportStart
				&expr.Cmp{
					Op:       expr.CmpOpGte,
					Register: 1,
					Data:     binaryBigEndian(spec.dportStart),
				},
				// Check if dport <= dportEnd
				&expr.Cmp{
					Op:       expr.CmpOpLte,
					Register: 1,
					Data:     binaryBigEndian(spec.dportEnd),
				},

				// Jump to accept chain
				&expr.Verdict{
					Kind:  expr.VerdictJump,
					Chain: "accept",
				},
			},
		})

		fmt.Printf("Added rule %d: ip6 saddr %s-%s ip6 daddr %s-%s th dport %d-%d jump accept\n",
			i+1, spec.saddrStart, spec.saddrEnd, spec.daddrStart, spec.daddrEnd, spec.dportStart, spec.dportEnd)
	}

	// Flush the changes
	if err := conn.Flush(); err != nil {
		log.Fatalf("Failed to flush: %v", err)
	}

	fmt.Printf("\nSuccessfully created custom chain with %d range-based rules\n", ruleCount)
}

func binaryBigEndian(v uint16) []byte {
	b := make([]byte, 2)
	binary.BigEndian.PutUint16(b, v)
	return b
}

If I give count like 50 or 100 it works but when I try with 500 it fails.
root [ /home/sontu ]# ./test 500
....
....
Added rule 500: ip6 saddr 1111:1234:5678:9abc:def0::1f4-1111:4321:8765:cba9:fed:ffff:1f3:ffff ip6 daddr 1170:1234:5678:9abc:def0::1f4-1170:4321:8765:cba9:fed:ffff:1f3:ffff th dport 12307-12308 jump accept
2025/12/18 15:01:42 Failed to flush: SendMessages: netlink send-messages: sendmsg: message too long
root [ /home/sontu ]#

[nickgarlis]

@sontumzmdr I am not seeing an issue when I use the latest version (df1ed79) with 500 or even 2000 rules.

Perhaps it's something unique to your system ? What's the output of uname -a and getconf PAGESIZE ?

It also seems that the error you received from your test is different to the one in the issue title. Did the error message change after upgrading to the latest version ?

[sontumzmdr]

@nickgarlis I am using Mariner, below are the details

root [ /home/sontu ]# uname -a
Linux sontu-vm 5.15.186.1-1.cm2 #1 SMP Tue Jul 29 02:54:43 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root [ /home/sontu ]#
root [ /home/sontu ]# getconf PAGESIZE
4096
root [ /home/sontu ]#

My application code is little complicated; hence I wrote the test code to simulate the issue, but as you mentioned the error is different here (behavior is same in latest master and as well in 0.3 version).
But the underneath issue is same, when the number of messages is big I hit the issue.
I have seen the Flush() code, I don't see we are doing any message size check and breaking the message. So eventually I am suspecting we will hit the issue (can vary upon system to system).

[nickgarlis]

It seems that you're using a relatively older kernel. I do intend to have a look once I have time but a more immediate fix for you might be to upgrade to a more recent version. If you find that upgrading fixes the problem, it'd be helpful to report back.

[nickgarlis]

@sontumzmdr I've now tested with 5.15.186 and I was not able to reproduce the issue.

Could you remove the print statements, test with strace and paste the output ?

# strace -e trace=setsockopt,getsockopt,sendmsg -s 0 ./test 500

Also, just to make sure... You're running the exact same test as the one pasted above right ? I am asking because the buffer logic is different if you initialize nftables using WithSockOptions as the netlink buffers will not be automatically enlarged.

[sontumzmdr]

@nickgarlis I have removed the print statements from the code. The code I am using is the exact same what I pasted.
Below is the output with strace option.

Working case with count 100:

root [ /home/sontu ]# strace -e trace=setsockopt,getsockopt,sendmsg -s 0 ./test 100
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
getsockopt(3, SOL_SOCKET, SO_TYPE, [3], [4]) = 0
sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[...], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 67548
+++ exited with 0 +++
root [ /home/sontu ]#

Non-working case with count 300 (recvmsg: no buffer space available, what I reported in the heading) :

root [ /home/sontu ]# strace -e trace=setsockopt,getsockopt,sendmsg -s 0 ./test 300
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388949, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388949, si_uid=0} ---
getsockopt(3, SOL_SOCKET, SO_TYPE, [3], [4]) = 0
sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[...], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 201948
2025/12/20 05:51:09 Failed to flush: Receive: netlink receive: recvmsg: no buffer space available
+++ exited with 1 +++
root [ /home/sontu ]#

Non-working case with count 500 (sendmsg: message too long) :

root [ /home/sontu ]# strace -e trace=setsockopt,getsockopt,sendmsg -s 0 ./test 500
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=389030, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=389030, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=389030, si_uid=0} ---
getsockopt(3, SOL_SOCKET, SO_TYPE, [3], [4]) = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=389030, si_uid=0} ---
2025/12/20 05:51:22 Failed to flush: SendMessages: netlink send-messages: sendmsg: message too long
+++ exited with 1 +++
root [ /home/sontu ]#

[nickgarlis]

@sontumzmdr

I would have expected to see something like this in your strace output

# Get send buffer size
getsockopt(7, SOL_SOCKET, SO_SNDBUF, [212992], [4]) = 0
# Increase send buffer size
setsockopt(7, SOL_SOCKET, SO_SNDBUFFORCE, [458868], 4) = 0
# Get read buffer size
getsockopt(7, SOL_SOCKET, SO_RCVBUF, [212992], [4]) = 0
# Increase read buffer size
setsockopt(7, SOL_SOCKET, SO_RCVBUFFORCE, [8388608], 4) = 0

which is a result of enlargeWriteBuffer and enlargeReadBuffer.

For some reason the functions appear to not work properly for you ? Perhaps you could add some print statements in there to see what's happening.

[sontumzmdr]

@nickgarlis I realized I was using an old main branch which doesn't have the enlargeWriteBuffer and enlargeReadBuffer methods. After pointing to latest version, I don't see the issue anymore.

I have a follow up question, is there any practical limit on size of the write/read socket buffer ? for my use case the ruleset could be very huge (~100K (4 tuple nft rules) at worst), so I am thinking if I should add the batching logic at my application or let enlargeWriteBuffer and enlargeReadBuffer takes care of it, which could be in few MBs ?

Thanks for your help.

[nickgarlis]

@sontumzmdr

I realized I was using an old main branch which doesn't have the enlargeWriteBuffer and enlargeReadBuffer methods. After pointing to latest version, I don't see the issue anymore.

That would explain it. Thanks for the follow-up.

I have a follow up question, is there any practical limit on size of the write/read socket buffer ? for my use case the ruleset could be very huge (~100K (4 tuple nft rules) at worst), so I am thinking if I should add the batching logic at my application or let enlargeWriteBuffer and enlargeReadBuffer takes care of it, which could be in few MBs ?

I believe the only hard limit is kernel memory when the buffers are adjusted automatically. That said, I’m not sure how far it’s safe to push this in practice, so it’s probably a good idea to remain mindful of buffer sizes.

It doesn't sound like your worst case would produce a message that's dangerously big though.

Some additional points to consider:

• If I am not mistaken, creating a lot of rules is not "the nft way" and sets are encouraged instead. Not that there is anything wrong with it but it might not perform as well.
• If you break your transaction into two smaller ones: one for creating the sets and one for applying the rules which reference them, you can reduce the size of each individual transaction.
• From your test case, I've noticed that you load the IPv6 payload by use of the expr.Payload expression without matching the protocol first (ipv4 or ipv6). Given that your table is of type inet this might be necessary and it is what nft does too. A better way to do it would be to do something like this

[]expr.Any {
    // Load L3 proto
    & expr.Meta {
            Key: expr.MetaKeyNFPROTO,
            Register: 1,
        },
        // check if L3 proto is ipv6
        & expr.Cmp {
            Op: expr.CmpOpEq,
            Register: 1,
            Data: [] byte {
                uint8(unix.NFPROTO_IPV6)
            },
        },
        // Load IPv6 source address
        & expr.Payload {
            DestRegister: 1,
            Base: expr.PayloadBaseNetworkHeader,
            Offset: 8, // IPv6 source address offset
            Len: 16, // IPv6 address length
        },
        // Check if saddr >= saddrStart
        & expr.Cmp {
            Op: expr.CmpOpGte,
            Register: 1,
            Data: spec.saddrStart.To16(),
        },
        // Check if saddr <= saddrEnd
        & expr.Cmp {
            Op: expr.CmpOpLte,
            Register: 1,
            Data: spec.saddrEnd.To16(),
        },
       // Continue with the rest of the exprs
}

[sontumzmdr]

@nickgarlis we initially explored the idea of verdict map (same as set) - where all combinations of 4 tuples are added as element and for each element, we are doing a specific jump to take corresponding element action like logging.
However, we encountered a big bottleneck in vmap (set) where if the number of elements increases gradually the time to load the vmap is exponential (if elements are in 100K range, it will take minutes). The behavior is similar with "nft -f" option (with pre-populated config) as well.

You can ignore my test code, this is just for testing the message size issue I created quickly.

[nickgarlis]

@sontumzmdr Thanks for the clarification! I don’t really have a specific suggestion based on what you shared, but since it doesn’t seem to be a buffer space issue anymore, maybe we can close the original issue?