Prometheus metrics exporter

[ArviFoxy]

Prometheus is a metrics collection and monitoring system. I'd like to use it with "roc-recv" and "roc-send" it to monitor my streaming setup - for latency, jitter, dropped packets, error correction stats, etc. This would have some advantages:

• I have 5 devices using ROC and collecting the CSV metric export is getting clunky.
• Prometheus supports histograms for distributions.

I'm going to implement this in a fork (using prometheus-cpp, optional dep). Once it's ready: if this is something you'd like to see in Roc I'd be happy to upstream.

[Btw thanks a lot for this project, it's powering my living room speaker system!]

[ArviFoxy]

https://github.com/ArviFoxy/roc-toolkit/tree/prometheus

It adds some boilerplate like:

    auto& jh_family = prometheus::BuildHistogram()
                          .Name("roc_recv_jitter_seconds")
                          .Help("Current instantaneous jitter distribution in seconds")
                          .Register(*registry);
    curr_jitter_histogram_ =
        &jh_family.Add({ }, metrics::generate_logspace_buckets(config.prometheus.jitter));

#ifdef ROC_TARGET_PROMETHEUS
    curr_jitter_histogram_->Observe((double)metrics_.curr_jitter / 1e9);
    curr_envelope_gauge_->Set((double)metrics_.curr_envelope / 1e9);
    mean_jitter_gauge_->Set((double)metrics_.mean_jitter / 1e9);
#endif

The cost should be very small- it's just an atomic op and a separate thread that collects metrics whenever polled. The results look great:

[ArviFoxy]

• FEC histograms to distinguish between burst loss and random loss.

[baranovmv]

Hi @ArviFoxy

This is a great effort! Thank you for posting it here

How do you see your intentions on this branch: is this an experiment, quick-ad-hoc-to-get-stuff done, or are you about to get it into master at some point?

In any case, I believe this might be useful for many, so I think it'd be great if you could cook up a PR at some point. I'm missing such functionality my home setup: roc-recv occasionally hiccups, but it's quite rare (once or twice a day), so it'd be difficult to reproduce efficiently to trace the issue. I suspect it's the wifi AP that freezes the traffic for more than 200ms of my configured latency.

More on your code:

At the moment we have --dump cli parameter that switches on verbose dumps of internal metrics and events into a text file. This is intended to be used in conjunction with a tool similar to logplotter -- a quick and dirty solution to plot various runtime metrics in a great volume effectively, producing oscilloscope like plots.

But its main purpose and readiness is quite different from prometheus: it's suited to debug internal algorithms like Frequency Estimator or Latency Tuner, not to monitor a production system 24/7. So it'd be very cool to have both tools on board, and make them get data from a uniform, cross-platform mechanism.

Let me throw an idea in here: can we enable prometheus, logplotter, any other ad-hoc-utils use a uniform API roc-toolkit could provide. A variant that comes to mind:

at the moment in develop branch one can find such methods:

void LinkMeter::dump_(const packet::Packet& packet) {
    const audio::JitterMetrics& jit_metrics = jitter_meter_.metrics();

    dbgio::CsvEntry e;
    e.type = 'm';
    e.n_fields = 5;
    e.fields[0] = packet.udp()->queue_timestamp;
    e.fields[1] = packet.rtp()->stream_timestamp;
    e.fields[2] = jit_metrics.curr_jitter;
    e.fields[3] = jit_metrics.peak_jitter;
    e.fields[4] = jit_metrics.curr_envelope;

    dumper_->write(e);
}

Later dumper gets these entries from a queue and dumps them into a file.

What I'm suggesting:

1. dumper and prometheus bindings are moved into cli tools. CsvEntry and prometheus::gauge are triggered from a dumping thread in a cli tool. roc-toolkit is a framework, and use cases can vary. prometheus might be useful to monitor roc-recv instance, but not necessary in some other use cases.
2. Structures similar to CsvEntry are transmitted towards a subscriber in the aforementioned dumping thread by means of a (possibly lock-free) queue, if requested by api. Something like roc::dump_subscribe("m", callback); or similar, not much of architecture...

[ArviFoxy]

The current code is fully functional and fast but the architecture needs some cleanup before merging.

Let me explain how prometheus-cpp works for context:

• It uses a pull model, not push.
• The initialization path adds metrics to a Registry (a map<string, Metric>). The metric is just some atomic state, for example for a gauge it's std::atomic<double> value_{0.0}; (https://github.com/jupp0r/prometheus-cpp/blob/master/core/include/prometheus/gauge.h )(histograms are a bit more complex)
• On the fast path calling Observe modifies that atomic state.
• The puller threads can read metrics on demand, by going over the registry and reading the atomic state. (There is an extremely tiny change that metrics won't be consistent with each other, like a "total' counter that got incremented but "missing" didn't yet. In practice that doesn't matter.)
• The exporter runs an HTTP server that pulls the metrics whenever called and formats them in a simple text format.

*Histograms:

• Observe on the fast path has to find the bucket to increment (either linear or binary search), but it's fast enough to not matter and can be turned off.
• The prometheus-cpp implementation unfortunately uses a mutex (https://github.com/jupp0r/prometheus-cpp/blob/master/core/include/prometheus/histogram.h). That's probably to make "ObserveMultiple" atomic. We only need "Observe" so it could be made lock free.

On your suggestions:

1. "dumper and prometheus bindings are moved into cli tools." - I agree (except if we just wanted to reuse the basics in B below).
2. I'd suggest using the pull model instead to avoid queues. It can be triggered by the consumer only when needed, no problems with queues filling up, and is how Prometheus works (triggered by a GET HTTP request).

So I'm thinking the implementation options are:

A. (imo the best) "Roll your own" and reimplement what prometheus-cpp does:

• Add Gauge, Counter, Histogram to roc.
• Fully lockfree without the histogram mutex.
• API: we have a choice between a "Registry" that's a map<string, Metric>, structs of metrics with a visit() method, or the dump_ method but called in the pulling thread. It's only a syntax choice.
• We might give up on metric families as I haven't used them.

B. Make prometheus-cpp a required dependency and the main way of storing metrics.

• The CSV export is easy to do from prometheus-cpp.
• Make sure the names are backwards compatible with CSV :)
• Fix the global Registry hack and properly pass it to all code emitting metrics.
• The mutex could sometimes suspend a fast path thread.. Although in practice I see no issues in my setup.

C. There's also OpenTelemetry for a common standard but tbh it looks very overcomplicated.

[ArviFoxy]

I'd implement A and send a pull request if the pull design looks good to you. @baranovmv

[baranovmv]

Hi @ArviFoxy , sorry for being away for so long.

I'd suggest using the pull model instead to avoid queues.

I think we can't avoid queues as the worker threads often must be hard real time: either it a sound card or a timer, they generate events and metrics we are going to collect at their own pace.

Add Gauge, Counter, Histogram to roc.

I'd like to see the part inside src/internal_modules and src/public_api as general as possible and seal protheus-aware code and CsvDumper in let's say src/tools/utils/metrics. By doing so we can avoid #ifdef (which I believe will make it more code-style complient).

This would inevitably lead to an abstract interface that both protheus module (class) and CsvDumper both should implement.

[ArviFoxy]

I think we can't avoid queues as the worker threads often must be hard real time: either it a sound card or a timer, they generate events and metrics we are going to collect at their own pace.

We don't even need queues to achieve that - the worker thread only needs to do an atomic op update the metric (cheaper than pushing to a queue or firing events). The poller thread can collect metrics from memory whenever it wants to.

I think we agree pretty much, I'll share a pull request once I have a bit of time and it should be clearer.

d like to see the part inside src/internal_modules and src/public_api as general as possible and seal protheus-aware code and CsvDumper in let's say src/tools/utils/metrics. By doing so we can avoid #ifdef (which I believe will make it more code-style complient).
This would inevitably lead to an abstract interface that both protheus module (class) and CsvDumper both should implement.

Sounds good! That's all I need to know.

[baranovmv]

We don't even need queues to achieve that - the worker thread only needs to do an atomic op update the metric

Yes, it will work, and most of the time metrics doesn't require strict sampling. However I believe all of us including yourself will be happy to be able to use this mechanism for debug purposes as a substitute to a CSV file logging, to which we refer when we debug some hard-to-locate or hard-to-reproduce nasty issues. Or both. In that case it might be useful to dump substantial amount of data, and a queue will support it.

Also it worth noting that we already have a mpsc queue in the codebase here src/internal_modules/roc_core/mpsc_queue.h. But I'm not quite sure it fits your needs precisely.

[ArviFoxy]

I think these are two very different use cases:

1. CSV file logging of every single sample for debugging.
2. Aggregated metrics used for 24/7 monitoring.

(2) can be done cheaply and lock-free by sampling and run on the whole time.
(1) is heavier and you probably want to only run it for debugging. This would need the queue.

IMO this might just need two different mechanisms.