Skip to content

enhancement(aggregate transform): Add Distribution as an aggregation mode - #23994

Open
jlambatl wants to merge 18 commits into
vectordotdev:masterfrom
jlambatl:transform_aggregate_distribution
Open

enhancement(aggregate transform): Add Distribution as an aggregation mode#23994
jlambatl wants to merge 18 commits into
vectordotdev:masterfrom
jlambatl:transform_aggregate_distribution

Conversation

@jlambatl

@jlambatl jlambatl commented Oct 13, 2025 •
edited by pront
Loading

Copy link
Copy Markdown
Contributor

Summary

This change adds a new AggregateMode distribution that aggregates individual metrics into a distribution. This doesn't reduce the accuracy of the metrics; instead, it reduces the volume of payloads being emitted from Vector by reducing the overhead of duplicate tags and other fields in exchange for a single larger payload.

This pattern suits real-time analytics databases, where histograms and other statistical functions can be calculated on the fly. It reduces the need to manually codify histogram buckets, which can become an onerous task on systems dealing with large metric volumes.

The distribution is sorted in the current implementation. I have been unsure whether that is a good idea, but I'm happy to change that based on your feedback.

closes: #24001

Vector configuration

[transforms.my_transform_id]
type = "aggregate"
inputs = [ "my-source-or-transform-id" ]
interval_ms = 10_000
mode = "distribution"

How did you test this PR?

I have added many different unit tests that cover the primary aggregation types, histograms, distributions, etc.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user-facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

Notes

N/A

@jlambatl
jlambatl requested review from a team as code owners October 13, 2025 04:04
@github-actions github-actions Bot added domain:transforms Anything related to Vector's transform components domain:external docs Anything related to Vector's external, public documentation labels Oct 13, 2025
@jlambatl jlambatl changed the title enhancement(transform aggregate): Add Distribution as an aggregation mode enhancement(aggregate): Add Distribution as an aggregation mode Oct 13, 2025
pront
pront reviewed Oct 13, 2025
View reviewed changes
Comment thread src/transforms/aggregate.rs Outdated
Comment thread src/transforms/aggregate.rs Outdated
Comment thread src/transforms/aggregate.rs Outdated
@jlambatl
Update rustdoc lines, changelog error with nonewline on checkdoc and …
8d7c5a0 
…improve the documentation that is generated for the distribution aggregation mode in the configuration parameter
@jlambatl

jlambatl commented Oct 13, 2025

Copy link
Copy Markdown
Contributor Author

I've pushed up a commit that should resolve the comments/feedback provided. It's hard to find the balance between being too verbose and not providing enough clarity.

@pront

pront commented Oct 14, 2025

Copy link
Copy Markdown
Member

Starting to look good, will wait for a review from @vectordotdev/documentation.

@rtrieu rtrieu self-assigned this Oct 14, 2025
rtrieu
rtrieu previously approved these changes Oct 14, 2025
View reviewed changes
@pront pront self-assigned this Oct 20, 2025
pront
pront reviewed Oct 24, 2025
View reviewed changes
Comment thread src/transforms/aggregate.rs Outdated
@pront pront added the meta:awaiting author Pull requests that are awaiting their author. label Oct 24, 2025
@jlambatl

jlambatl commented Oct 28, 2025

Copy link
Copy Markdown
Contributor Author

@pront The update looks good to me; it's also nice to see the improved tests. Thankyou.

@pront
pront enabled auto-merge October 29, 2025 13:35
@pront pront removed the meta:awaiting author Pull requests that are awaiting their author. label Oct 29, 2025
auto-merge was automatically disabled November 5, 2025 22:49

Head branch was pushed to by a user without write access

@jlambatl

jlambatl commented Nov 5, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@pront I've fixed the Clippy error that was failing in CI and pushed up the changes.

@pront
pront enabled auto-merge November 6, 2025 15:40
@jlambatl jlambatl changed the title enhancement(aggregate): Add Distribution as an aggregation mode enhancement(aggregate transform): Add Distribution as an aggregation mode Feb 16, 2026
@jlambatl

jlambatl commented Feb 25, 2026

Copy link
Copy Markdown
Contributor Author

@vectordotdev/vector Is there anything outstanding with this PR?

auto-merge was automatically disabled April 15, 2026 01:37

Head branch was pushed to by a user without write access

@jlambatl

jlambatl commented Apr 15, 2026

Copy link
Copy Markdown
Contributor Author

I have resolved the merge conflicts...

@pront pront added the transform:aggregate Anything `aggregate` transform related label May 24, 2026
@jlambatl
Merge upstream/master into aggregate distribution branch
482c0bc
@jlambatl
enhancement(aggregate transform): Port Distribution mode to the aggre…
94d23a8 
…gate module

Re-apply the `distribution` aggregation mode on top of the split
`transforms/aggregate/` module layout and the new event-time aggregation.

- Add `AggregationMode::Distribution` / `InnerMode::Distribution` and handle
  it in both the system-time and event-time paths via a shared
  `distribution_from_entries` helper.
- Pass incremental metrics through unchanged, consistent with the other
  modes that aggregate absolute metrics (Latest, Diff, Max, Min, Mean, Stdev).
- Only store absolute metrics that can yield samples (gauge, set,
  distribution, aggregated histogram). Other absolute types are ignored, so
  an empty distribution is never emitted and no event-time bucket is created.
- Port the existing Distribution tests and add coverage for incremental
  passthrough, unsupported absolute types and event-time bucketing.
- Update the aggregate component docs.
@github-actions github-actions Bot added the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label Oct 28, 2026
@jlambatl
jlambatl requested review from pront and rtrieu October 28, 2026 11:50

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pront pront Awaiting requested review from pront
@rtrieu rtrieu Awaiting requested review from rtrieu

At least 1 approving review is required to merge this pull request.

Assignees

@pront pront

@rtrieu rtrieu

Labels

docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain:external docs Anything related to Vector's external, public documentation domain:transforms Anything related to Vector's transform components transform:aggregate Anything `aggregate` transform related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

vector gauge to distribution aggregation mode

3 participants

@jlambatl @pront @rtrieu