Vector CPU Utilisation

[prayerx]

Hi, im trying to understand limit of vector. I read scaling documents and i should be easily see 10MB/s for each CPU core.
Im testing on my lab with a python script sending syslog and listening on vector end with socket source without transforms and sinking to blackhole.
I can reach around 70MB/s without packet drops with having 16 cpu cores, after that it starts to drop packets. When i monitor each cpu core with htop, 5 cores are working (changing which core is active) with %50 utilisation.
I'm expecting from vector to use all CPU cores to %100 (or close) before it start dropping packets. Is my expectation wrong? If not what should be the issue here?

[jszwedko]

Hi @prayerx ,

There are a few factors that can limit throughput including the concurrency model of any components involved (sources, sinks, transforms) as well as back pressure from sinks if they can't send data fast enough. In your case, you mention it is just socket source to blackhole sink. Are you sending data via UDP, TCP, or over a Unix socket? Since you said syslog traffic, I'm guessing UDP in which case there are a few relevant issues that may be limiting throughput:

• Improve Vector's throughput for UDP protocols #8518
• Excess stat calls #12451
• performance: UDP recvmmsg instead of recvfrom #12452

Importantly the socket source, for UDP traffic, is single threaded.

[prayerx]

Hi @jszwedko, thanks for your detailed answer. You guessed right, its UDP since I believed it'll perform better but maybe i'm wrong. The example threads didn't help a lot but your last sentence is very important.
I made the same test with a system with 4 CPU cores and I'm still getting same values as testing with a 16 cores system. Which alligns my utilisation problem and your last sence that UDP Socket source is single threaded.
Is there any way to check other sources and Vector's behavior about threading? I checked syslog source but there is no information about being single threaded on udp socket. It'll be very helpful to know when scaling vector for different scenarios and deciding which source to use.

[jszwedko]

Is there any way to check other sources and Vector's behavior about threading? I checked syslog source but there is no information about being single threaded on udp socket. It'll be very helpful to know when scaling vector for different scenarios and deciding which source to use.

Unfortunately there isn't much in the way of documentation on the threading model for each source. In general sources try to handle input concurrently (e.g. TCP streams in the socket source, kafka topics in the kafka source, HTTP requests in the http source, etc.), but UDP is a bit of an outlier in that everything comes into one queue and is processed sequentially.

[Katarina1134]

Coming back to this thread because @prayerx's question about threading models across sources is still relevant and worth a more complete answer.

Vector's threading model by source type (rough guide):

• socket (UDP): single-threaded — all packets enter one queue, processed sequentially. This is why you see 5 cores at 50% instead of all 16 at 100%.
• socket (TCP): concurrent per-connection — each TCP stream gets its own task
• kafka: single async task per partition consumer
• http: concurrent per-request
• kubernetes_logs: single file-watching loop with async reads
• syslog (UDP): same limitation as UDP socket

For your specific case — high-throughput UDP syslog — the practical options are:

1. Run multiple Vector instances on the same host, each bound to a different port
Load-balance UDP traffic across ports at the sender or with a hardware load balancer. Each Vector instance gets its own process and CPU affinity. Crude but effective.

2. Switch to TCP syslog
TCP socket source handles connections concurrently. If your senders support TCP syslog (RFC 6587), you'll immediately see better CPU utilization across cores.

3. Use a UDP fan-out proxy in front of Vector
Tools like stunnel or a custom socat setup can receive UDP on one port and fan out to multiple Vector instances via TCP. Adds a hop but removes the single-threaded bottleneck.

On the broader question of CPU utilization

The 70 MB/s ceiling on 16 cores (effectively using ~5 cores at 50%) is consistent with a single-threaded receive loop hitting its per-core limit. The remaining cores are idle because there's no work to distribute to them — the UDP queue is the only ingestion path.

For what it's worth, this single-threaded ingestion ceiling is one of the design constraints we ran into when building WarpParse. We ended up using a shared thread pool for ingestion rather than per-source single threads — which lets the process saturate available cores regardless of source type. Happy to share more detail on the tradeoffs if useful.

[splitice]

I wrote a batched UDP recvmmsg PR a while ago. Its quite alot faster for UDP.

No multi threading though.