[feat] Emitter Pools - #105
Merged
Merged
Conversation
Contributor
Author
|
I think this PR is significant enough to bump the minor version (or make this the 1.0 release). It should be backwards compatible with the old implementation, and even with the default configuration, should perform better and be safer than the pooled implementation. |
Contributor
Author
|
@arkgil PTAL! |
This was referenced Jul 27, 2025
arkgil
reviewed
Jul 27, 2025
During testing at Whatnot, we found that when emitting around 50k metrics per second, the current pool implementation caused the calling processes to stall while waiting for `:gen_udp` to write data. This stalling caused significant degradation to our application, and eventually caused an outage. To address this, we came up with the concept of using a pool of emitters. An emitter is a process that contains a socket (either UDP or domain), receives metrics and writes them to a socket. This architecture frees up the calling processes so they can write a metric and not wait on a result. Emitters also allow us to combine metrics so they fill up a UDP packet. The end result, according to our testing, moved us from 50k to 780k metrics per second.
Adds the emit percentage as a metric. Also added tests
Contributor
Collaborator
|
It is, we’re successfully running this in production for more than a year now at 5x the scale Steve originally built it for. We’ll get this shipped soon.
…On Wed, Mar 11, 2026, at 15:58, Andrea Leopardi wrote:
*whatyouhide* left a comment (beam-telemetry/telemetry_metrics_statsd#105) <#105 (comment)>
@arkgil <https://github.com/arkgil> @scohen <https://github.com/scohen> is this something we're still pursuing?
—
Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACYSVEJ22UMV3TT5MCO2QPT4QF5HVAVCNFSM6AAAAACWOMXON6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMZZHAYTSMBSHA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
arkgil
reviewed
Mar 16, 2026
Contributor
Author
To be fair, I tested it at a million metrics per second, we're not there yet ;) |
arkgil
approved these changes
Mar 20, 2026
| :ok | ||
|
|
||
| {:error, :econnrefused} -> | ||
| # TODO: Carrying over socket lib behavior, this seems like something we'd like to know about. |
Collaborator
There was a problem hiding this comment.
@scohen would you mind removing the comment and the one below? they don't add much without the context of the original socket PR
Normalize errors during initialization emitters via our helpers; also add an option for starting a non-linked emitter process, which prevents a crash on startup from taking down the test.
Contributor
Author
|
@arkgil all green! |
Collaborator
|
🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During testing at Whatnot, we found that when emitting around 50k metrics per second, the current pool implementation caused the calling processes to stall while waiting for
:gen_udpto write data. This stalling caused significant degradation to our application, and eventually caused an outage.To address this, we came up with the concept of using a pool of emitters. An emitter is a process that contains a socket (either UDP or domain), receives metrics and writes them to a socket. This architecture frees up the calling processes so they can write a metric and not wait on a result. Emitters also allow us to combine metrics so they fill up a UDP packet. The end result, according to our testing, moved us from 50k to 1MM metrics per second.