From 9e641964d0767ab4db6710805b31675bfba1248e Mon Sep 17 00:00:00 2001 From: stefans-elastic Date: Wed, 22 Jul 2026 16:37:25 +0300 Subject: [PATCH 1/5] fix: make Azure billing usage and forecast windows configurable --- ...-billing-configurable-lookback-window.yaml | 50 ++++++++++++++ .../metricbeat-metricset-azure-billing.md | 9 +++ .../metricbeat/metricbeat-module-azure.md | 7 ++ x-pack/metricbeat/metricbeat.reference.yml | 7 ++ .../module/azure/_meta/config.reference.yml | 7 ++ .../metricbeat/module/azure/_meta/config.yml | 5 ++ .../module/azure/billing/_meta/docs.md | 9 +++ .../module/azure/billing/billing.go | 69 ++++++++++++------- .../module/azure/billing/billing_test.go | 65 ++++++++++++++++- .../module/azure/billing/client_test.go | 4 +- .../module/azure/billing/data_test.go | 20 +++--- x-pack/metricbeat/module/azure/config.go | 6 ++ .../metricbeat/modules.d/azure.yml.disabled | 5 ++ 13 files changed, 226 insertions(+), 37 deletions(-) create mode 100644 changelog/fragments/1784719246-azure-billing-configurable-lookback-window.yaml diff --git a/changelog/fragments/1784719246-azure-billing-configurable-lookback-window.yaml b/changelog/fragments/1784719246-azure-billing-configurable-lookback-window.yaml new file mode 100644 index 000000000000..8ba40a06bfcd --- /dev/null +++ b/changelog/fragments/1784719246-azure-billing-configurable-lookback-window.yaml @@ -0,0 +1,50 @@ +# REQUIRED +# Kind can be one of: +# - breaking-change: a change to previously-documented behavior +# - deprecation: functionality that is being removed in a later release +# - bug-fix: fixes a problem in a previous version +# - enhancement: extends functionality but does not break or fix existing behavior +# - feature: new functionality +# - known-issue: problems that we are aware of in a given version +# - security: impacts on the security of a product or a user’s deployment. +# - upgrade: important information for someone upgrading from a prior version +# - other: does not fit into any of the other categories +kind: bug-fix + +# REQUIRED for all kinds +# Change summary; a 80ish characters long description of the change. +summary: Add configurable usage lookback and forecast windows to the Azure billing metricset + +# REQUIRED for breaking-change, deprecation, known-issue +# Long description; in case the summary is not enough to describe the change +# this field accommodate a description without length limits. +description: | + Adds `billing_usage_lookback` and `billing_forecast_window` duration config options to the + Azure billing metricset. `billing_usage_lookback` (default `24h`) allows re-querying multiple + previous days of usage data so that Azure's documented up-to-72h cost corrections are captured. + `billing_forecast_window` (default `720h`) controls the length of the forecast period. Defaults + preserve the existing behaviour. + +# REQUIRED for breaking-change, deprecation, known-issue +# impact: + +# REQUIRED for breaking-change, deprecation, known-issue +# action: + +# REQUIRED for all kinds +# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc. +component: metricbeat + +# AUTOMATED +# OPTIONAL to manually add other PR URLs +# PR URL: A link the PR that added the changeset. +# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added. +# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number. +# Please provide it if you are adding a fragment for a different PR. +# pr: https://github.com/owner/repo/1234 + +# AUTOMATED +# OPTIONAL to manually add other issue URLs +# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of). +# If not present is automatically filled by the tooling with the issue linked to the PR number. +# issue: https://github.com/owner/repo/1234 diff --git a/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md b/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md index ebb2ce5bd08e..4327252f73e1 100644 --- a/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md +++ b/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md @@ -39,6 +39,15 @@ This metricset allows users to retrieve usage details and forecast information o If none of the 2 options are entered then the subscription ID will be used as scope. +`billing_usage_lookback` +: (*duration*, default: `24h`) How far back to query usage data on each fetch. Azure Cost Management + data is documented to be updated for up to 72 hours after a billing period closes, so increasing + this to `72h` ensures that late cost corrections from Azure are re-queried and ingested. + +`billing_forecast_window` +: (*duration*, default: `720h`) The length of the forecast period, counted forward from the forecast + start date (reference time minus 2 days). Defaults to 30 days (`720h`). + ## Fields [_fields] For a description of each field in the metricset, see the [exported fields](/reference/metricbeat/exported-fields-azure.md) section. diff --git a/docs/reference/metricbeat/metricbeat-module-azure.md b/docs/reference/metricbeat/metricbeat-module-azure.md index 21f825c071a9..562db0bccc70 100644 --- a/docs/reference/metricbeat/metricbeat-module-azure.md +++ b/docs/reference/metricbeat/metricbeat-module-azure.md @@ -258,6 +258,13 @@ metricbeat.modules: client_secret: '${AZURE_CLIENT_SECRET:""}' tenant_id: '${AZURE_TENANT_ID:""}' subscription_id: '${AZURE_SUBSCRIPTION_ID:""}' + # billing_usage_lookback controls how far back usage data is queried on each fetch. + # Defaults to 24h (previous full day). Azure Cost Management data can be updated for up + # to 72 hours after a billing period closes, so setting this to 72h ensures late + # corrections are captured. + #billing_usage_lookback: 72h + # billing_forecast_window controls the length of the forecast period. Defaults to 720h (30 days). + #billing_forecast_window: 720h - module: azure metricsets: diff --git a/x-pack/metricbeat/metricbeat.reference.yml b/x-pack/metricbeat/metricbeat.reference.yml index 41c951418227..0ff1c7d49389 100644 --- a/x-pack/metricbeat/metricbeat.reference.yml +++ b/x-pack/metricbeat/metricbeat.reference.yml @@ -400,6 +400,13 @@ metricbeat.modules: client_secret: '${AZURE_CLIENT_SECRET:""}' tenant_id: '${AZURE_TENANT_ID:""}' subscription_id: '${AZURE_SUBSCRIPTION_ID:""}' + # billing_usage_lookback controls how far back usage data is queried on each fetch. + # Defaults to 24h (previous full day). Azure Cost Management data can be updated for up + # to 72 hours after a billing period closes, so setting this to 72h ensures late + # corrections are captured. + #billing_usage_lookback: 72h + # billing_forecast_window controls the length of the forecast period. Defaults to 720h (30 days). + #billing_forecast_window: 720h - module: azure metricsets: diff --git a/x-pack/metricbeat/module/azure/_meta/config.reference.yml b/x-pack/metricbeat/module/azure/_meta/config.reference.yml index b469a846edf3..b00893b8cb89 100644 --- a/x-pack/metricbeat/module/azure/_meta/config.reference.yml +++ b/x-pack/metricbeat/module/azure/_meta/config.reference.yml @@ -92,6 +92,13 @@ client_secret: '${AZURE_CLIENT_SECRET:""}' tenant_id: '${AZURE_TENANT_ID:""}' subscription_id: '${AZURE_SUBSCRIPTION_ID:""}' + # billing_usage_lookback controls how far back usage data is queried on each fetch. + # Defaults to 24h (previous full day). Azure Cost Management data can be updated for up + # to 72 hours after a billing period closes, so setting this to 72h ensures late + # corrections are captured. + #billing_usage_lookback: 72h + # billing_forecast_window controls the length of the forecast period. Defaults to 720h (30 days). + #billing_forecast_window: 720h - module: azure metricsets: diff --git a/x-pack/metricbeat/module/azure/_meta/config.yml b/x-pack/metricbeat/module/azure/_meta/config.yml index 434329c4b4bd..3cec2b74210b 100644 --- a/x-pack/metricbeat/module/azure/_meta/config.yml +++ b/x-pack/metricbeat/module/azure/_meta/config.yml @@ -101,6 +101,11 @@ # tenant_id: '${AZURE_TENANT_ID:""}' # subscription_id: '${AZURE_SUBSCRIPTION_ID:""}' # refresh_list_interval: 600s +# # How far back to query usage data on each fetch (default 24h). Azure Cost Management data +# # can be updated for up to 72 hours, so set this to 72h to capture late cost corrections. +# billing_usage_lookback: 72h +# # Length of the forecast period (default 720h / 30 days). +# billing_forecast_window: 720h #- module: azure # metricsets: diff --git a/x-pack/metricbeat/module/azure/billing/_meta/docs.md b/x-pack/metricbeat/module/azure/billing/_meta/docs.md index 5191e1585d26..24bcad9ffc85 100644 --- a/x-pack/metricbeat/module/azure/billing/_meta/docs.md +++ b/x-pack/metricbeat/module/azure/billing/_meta/docs.md @@ -26,3 +26,12 @@ This metricset allows users to retrieve usage details and forecast information o : (*string*) Retrieve usage details based on the billing account ID scope. If none of the 2 options are entered then the subscription ID will be used as scope. + +`billing_usage_lookback` +: (*duration*, default: `24h`) How far back to query usage data on each fetch. Azure Cost Management + data is documented to be updated for up to 72 hours after a billing period closes, so increasing + this to `72h` ensures that late cost corrections from Azure are re-queried and ingested. + +`billing_forecast_window` +: (*duration*, default: `720h`) The length of the forecast period, counted forward from the forecast + start date (reference time minus 2 days). Defaults to 30 days (`720h`). diff --git a/x-pack/metricbeat/module/azure/billing/billing.go b/x-pack/metricbeat/module/azure/billing/billing.go index 93bad5d87afc..17f091cdfbdb 100644 --- a/x-pack/metricbeat/module/azure/billing/billing.go +++ b/x-pack/metricbeat/module/azure/billing/billing.go @@ -16,6 +16,16 @@ import ( "github.com/elastic/elastic-agent-libs/logp" ) +const ( + // defaultUsageLookback is the usage query window when billing_usage_lookback is unset. + // It matches the original hardcoded behaviour: query the single previous full UTC day. + defaultUsageLookback = 24 * time.Hour + + // defaultForecastWindow is the forecast query window when billing_forecast_window is unset. + // It matches the original hardcoded behaviour: 30 days forward from the forecast start date. + defaultForecastWindow = 30 * 24 * time.Hour +) + // init registers the MetricSet with the central registry as soon as the program // starts. The New function will be called later to instantiate an instance of // the MetricSet for each host defined in the module's configuration. After the @@ -42,9 +52,7 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) { if err != nil { return nil, fmt.Errorf("error unpack raw module config using UnpackConfig: %w", err) } - if err != nil { - return nil, err - } + applyBillingDefaults(&config) // instantiate monitor client billingClient, err := NewClient(config, base.Logger()) if err != nil { @@ -57,6 +65,17 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) { }, nil } +// applyBillingDefaults fills in default values for billing-specific duration +// config fields that were not explicitly set by the user. +func applyBillingDefaults(cfg *azure.Config) { + if cfg.BillingUsageLookback == 0 { + cfg.BillingUsageLookback = defaultUsageLookback + } + if cfg.BillingForecastWindow == 0 { + cfg.BillingForecastWindow = defaultForecastWindow + } +} + // TimeIntervalOptions represents the options used to retrieve the billing data. type TimeIntervalOptions struct { // Usage details start time (UTC). @@ -76,8 +95,8 @@ func (m *MetricSet) Fetch(report mb.ReporterV2) error { // reference time used to calculate usage and forecast time intervals. referenceTime := time.Now() - usageStart, usageEnd := usageIntervalFrom(referenceTime) - forecastStart, forecastEnd := forecastIntervalFrom(referenceTime) + usageStart, usageEnd := usageIntervalFrom(referenceTime, m.client.Config.BillingUsageLookback) + forecastStart, forecastEnd := forecastIntervalFrom(referenceTime, m.client.Config.BillingForecastWindow) timeIntervalOptions := TimeIntervalOptions{ usageStart: usageStart, @@ -110,31 +129,35 @@ func (m *MetricSet) Fetch(report mb.ReporterV2) error { return nil } -// usageIntervalFrom returns the start/end times (UTC) of the usage period given the `reference` time. +// usageIntervalFrom returns the start/end times (UTC) of the usage period given the +// reference time and a lookback duration. // -// Currently, the usage period is the start/end time (00:00:00->23:59:59 UTC) of the day before the reference time. +// The usage period ends at start-of-today minus one second (i.e. yesterday 23:59:59 UTC) +// and starts lookback duration before that. // -// For example, if the reference time is 2007-01-09 09:41:00Z, the usage period is: +// For example, with reference 2007-01-09 09:41:00Z and a 24h lookback, the usage period is: // // 2007-01-08 00:00:00Z -> 2007-01-08 23:59:59Z -func usageIntervalFrom(reference time.Time) (time.Time, time.Time) { - beginningOfDay := reference.UTC().Truncate(24 * time.Hour).Add((-24) * time.Hour) - endOfDay := beginningOfDay.Add(time.Hour * 24).Add(time.Second * (-1)) - return beginningOfDay, endOfDay +// +// With a 72h lookback it covers the three previous full days: +// +// 2007-01-06 00:00:00Z -> 2007-01-08 23:59:59Z +func usageIntervalFrom(reference time.Time, lookback time.Duration) (time.Time, time.Time) { + startOfToday := reference.UTC().Truncate(24 * time.Hour) + return startOfToday.Add(-lookback), startOfToday.Add(-time.Second) } -// forecastIntervalFrom returns the start/end times (UTC) of the forecast period, given the `reference` time. +// forecastIntervalFrom returns the start/end times (UTC) of the forecast period given the +// reference time and a window duration. // -// Currently, the forecast period is the start/end times (00:00:00->23:59:59 UTC) of the current month relative to the -// reference time. +// The forecast period always starts at reference minus 2 days (00:00:00 UTC) and extends +// forward for the given window duration. // -// For example, if the reference time is 2007-01-09 09:41:00Z, the forecast period is: -// The forecast data is fetched from current day - 2 and for next 30 days. +// For example, with reference 2007-01-09 09:41:00Z and a 30-day (720h) window: // -// 2007-01-07T00:00:00Z -> 2007-02-05:59:59Z -func forecastIntervalFrom(reference time.Time) (time.Time, time.Time) { - referenceUTC := reference.UTC().Truncate(24 * time.Hour).Add((-48) * time.Hour) - forecastStartDate := time.Date(referenceUTC.Year(), referenceUTC.Month(), referenceUTC.Day(), 0, 0, 0, 0, time.UTC) - forecastEndDate := forecastStartDate.AddDate(0, 0, 0).Add(-1*time.Second).AddDate(0, 0, 30) - return forecastStartDate, forecastEndDate +// 2007-01-07T00:00:00Z -> 2007-02-05T23:59:59Z +func forecastIntervalFrom(reference time.Time, window time.Duration) (time.Time, time.Time) { + forecastStart := reference.UTC().Truncate(24 * time.Hour).Add(-48 * time.Hour) + forecastEnd := forecastStart.Add(window).Add(-time.Second) + return forecastStart, forecastEnd } diff --git a/x-pack/metricbeat/module/azure/billing/billing_test.go b/x-pack/metricbeat/module/azure/billing/billing_test.go index 24ad87ca35ff..ef009231e714 100644 --- a/x-pack/metricbeat/module/azure/billing/billing_test.go +++ b/x-pack/metricbeat/module/azure/billing/billing_test.go @@ -11,6 +11,8 @@ import ( "time" "github.com/stretchr/testify/assert" + + "github.com/elastic/beats/v7/x-pack/metricbeat/module/azure" ) func TestUsagePeriodFrom(t *testing.T) { @@ -22,7 +24,23 @@ func TestUsagePeriodFrom(t *testing.T) { expectedEndTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-08 23:59:59") assert.NoError(t, err) - actualStartTime, actualEndTime := usageIntervalFrom(referenceTime) + actualStartTime, actualEndTime := usageIntervalFrom(referenceTime, 24*time.Hour) + + assert.Equal(t, expectedStartTime, actualStartTime) + assert.Equal(t, expectedEndTime, actualEndTime) + }) + + t.Run("widens the window when lookback is 72h", func(t *testing.T) { + referenceTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-09 09:41:00") + assert.NoError(t, err) + // 72h lookback from start-of-today (2007-01-09 00:00:00) → 2007-01-06 00:00:00 + // end always = start-of-today minus one second → 2007-01-08 23:59:59 + expectedStartTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-06 00:00:00") + assert.NoError(t, err) + expectedEndTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-08 23:59:59") + assert.NoError(t, err) + + actualStartTime, actualEndTime := usageIntervalFrom(referenceTime, 72*time.Hour) assert.Equal(t, expectedStartTime, actualStartTime) assert.Equal(t, expectedEndTime, actualEndTime) @@ -39,9 +57,52 @@ func TestForecastPeriodFrom(t *testing.T) { expectedEndTime, err := time.Parse("2006-01-02 15:04:05", "2007-02-05 23:59:59") assert.NoError(t, err) - actualStartTime, actualEndTime := forecastIntervalFrom(referenceTime) + actualStartTime, actualEndTime := forecastIntervalFrom(referenceTime, 30*24*time.Hour) + + assert.Equal(t, expectedStartTime, actualStartTime) + assert.Equal(t, expectedEndTime, actualEndTime) + }) + + t.Run("uses the configured window duration", func(t *testing.T) { + referenceTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-09 09:41:00") + assert.NoError(t, err) + + // forecast always starts at reference - 2 days; end = start + window - 1s + // with a 7-day window: 2007-01-07 00:00:00 + 7d - 1s = 2007-01-13 23:59:59 + expectedStartTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-07 00:00:00") + assert.NoError(t, err) + expectedEndTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-13 23:59:59") + assert.NoError(t, err) + + actualStartTime, actualEndTime := forecastIntervalFrom(referenceTime, 7*24*time.Hour) assert.Equal(t, expectedStartTime, actualStartTime) assert.Equal(t, expectedEndTime, actualEndTime) }) } + +func TestApplyBillingDefaults(t *testing.T) { + t.Run("sets usage lookback and forecast window when both are unset", func(t *testing.T) { + cfg := azure.Config{} + applyBillingDefaults(&cfg) + + assert.Equal(t, defaultUsageLookback, cfg.BillingUsageLookback) + assert.Equal(t, defaultForecastWindow, cfg.BillingForecastWindow) + }) + + t.Run("does not override an explicitly configured usage lookback", func(t *testing.T) { + cfg := azure.Config{BillingUsageLookback: 72 * time.Hour} + applyBillingDefaults(&cfg) + + assert.Equal(t, 72*time.Hour, cfg.BillingUsageLookback) + assert.Equal(t, defaultForecastWindow, cfg.BillingForecastWindow) + }) + + t.Run("does not override an explicitly configured forecast window", func(t *testing.T) { + cfg := azure.Config{BillingForecastWindow: 14 * 24 * time.Hour} + applyBillingDefaults(&cfg) + + assert.Equal(t, defaultUsageLookback, cfg.BillingUsageLookback) + assert.Equal(t, 14*24*time.Hour, cfg.BillingForecastWindow) + }) +} diff --git a/x-pack/metricbeat/module/azure/billing/client_test.go b/x-pack/metricbeat/module/azure/billing/client_test.go index fba73277671e..59fcf24b9d74 100644 --- a/x-pack/metricbeat/module/azure/billing/client_test.go +++ b/x-pack/metricbeat/module/azure/billing/client_test.go @@ -26,8 +26,8 @@ var ( ) func TestClient(t *testing.T) { - usageStart, usageEnd := usageIntervalFrom(time.Now()) - forecastStart, forecastEnd := forecastIntervalFrom(time.Now()) + usageStart, usageEnd := usageIntervalFrom(time.Now(), defaultUsageLookback) + forecastStart, forecastEnd := forecastIntervalFrom(time.Now(), defaultForecastWindow) opts := TimeIntervalOptions{ usageStart: usageStart, usageEnd: usageEnd, diff --git a/x-pack/metricbeat/module/azure/billing/data_test.go b/x-pack/metricbeat/module/azure/billing/data_test.go index 1b2075345c61..f8366069aeec 100644 --- a/x-pack/metricbeat/module/azure/billing/data_test.go +++ b/x-pack/metricbeat/module/azure/billing/data_test.go @@ -67,7 +67,7 @@ func TestEventMapping(t *testing.T) { // this unusual format. actualUsageDate := float64(20200807) forecastUsageDate := float64(20200808) - rows := [][]interface{}{ + rows := [][]any{ {actualCost, actualUsageDate, "Actual", "USD"}, {forecastCost, forecastUsageDate, "Forecast", "USD"}, } @@ -92,8 +92,8 @@ func TestEventMapping(t *testing.T) { // // Run the tests // - usageStart, usageEnd := usageIntervalFrom(time.Now()) - forecastStart, forecastEnd := forecastIntervalFrom(time.Now()) + usageStart, usageEnd := usageIntervalFrom(time.Now(), defaultUsageLookback) + forecastStart, forecastEnd := forecastIntervalFrom(time.Now(), defaultForecastWindow) opts := TimeIntervalOptions{ usageStart: usageStart, usageEnd: usageEnd, @@ -103,7 +103,7 @@ func TestEventMapping(t *testing.T) { events, err := EventsMapping("sub", usage, opts, logger) assert.NoError(t, err) - assert.Equal(t, 3, len(events)) + assert.Len(t, events, 3) // // Check the results @@ -120,7 +120,7 @@ func TestEventMapping(t *testing.T) { val3, _ := event.MetricSetFields.GetValue("department_name") assert.Equal(t, val3, &name) tags, _ := event.ModuleFields.GetValue("resource.tags") - assert.Equal(t, tags, map[string]*string{tagName: &tagValue}) + assert.Equal(t, map[string]*string{tagName: &tagValue}, tags) } else { // @@ -131,7 +131,7 @@ func TestEventMapping(t *testing.T) { isActual, _ := event.MetricSetFields.HasKey("actual_cost") if isActual { cost, _ := event.MetricSetFields.GetValue("actual_cost") - assert.Equal(t, actualCost, cost) + assert.InDelta(t, actualCost, cost, 0.0001) dt, _ := time.Parse("2006-01-02", "2020-08-07") usageDate, _ := event.MetricSetFields.GetValue("usage_date") assert.Equal(t, usageDate, dt) @@ -141,7 +141,7 @@ func TestEventMapping(t *testing.T) { isForecast, _ := event.MetricSetFields.HasKey("forecast_cost") if isForecast { cost, _ := event.MetricSetFields.GetValue("forecast_cost") - assert.Equal(t, forecastCost, cost) + assert.InDelta(t, forecastCost, cost, 0.0001) dt, _ := time.Parse("2006-01-02", "2020-08-08") usageDate, _ := event.MetricSetFields.GetValue("usage_date") assert.Equal(t, usageDate, dt) @@ -207,7 +207,7 @@ func TestGetEventsFromQueryResult(t *testing.T) { }) t.Run("wrong number of elements in a row", func(t *testing.T) { - rows := [][]interface{}{ + rows := [][]any{ {float64(1), float64(2), "Actual", "USD", "UnexpectedValue"}, } queryResult := armcostmanagement.QueryResult{ @@ -223,7 +223,7 @@ func TestGetEventsFromQueryResult(t *testing.T) { }) t.Run("drop rows with a wrong type", func(t *testing.T) { - rows := [][]interface{}{ + rows := [][]any{ {float64(1), float64(20220818), "Actual", "USD"}, // good row, this will be mapped as event {42, float64(20220818), "Actual", "USD"}, // wrong cost type {float64(1), 20220818, "Actual", "USD"}, // wrong usage date type @@ -238,7 +238,7 @@ func TestGetEventsFromQueryResult(t *testing.T) { } events, err := getEventsFromQueryResult(queryResult, subscriptionID, logger) - assert.Equal(t, 1, len(events)) + assert.Len(t, events, 1) assert.NoError(t, err) }) } diff --git a/x-pack/metricbeat/module/azure/config.go b/x-pack/metricbeat/module/azure/config.go index 6ca6d9b836de..f296993b46fe 100644 --- a/x-pack/metricbeat/module/azure/config.go +++ b/x-pack/metricbeat/module/azure/config.go @@ -49,6 +49,12 @@ type Config struct { // specific to billing BillingScopeDepartment string `config:"billing_scope_department"` // retrieve usage details from department scope BillingScopeAccountId string `config:"billing_scope_account_id"` // retrieve usage details from billing account ID scope + // BillingUsageLookback controls how far back (as a duration) the billing metricset + // queries usage data. Defaults to 24h (previous full day). Increase to 72h to capture + // Azure's documented up-to-72h cost data updates. + BillingUsageLookback time.Duration `config:"billing_usage_lookback"` + // BillingForecastWindow controls the length of the forecast period. Defaults to 720h (30 days). + BillingForecastWindow time.Duration `config:"billing_forecast_window"` // Use BatchApi for metric values collection EnableBatchApi bool `config:"enable_batch_api"` // defaults to false // DefaultTimeGrain sets the default time interval when the resource config diff --git a/x-pack/metricbeat/modules.d/azure.yml.disabled b/x-pack/metricbeat/modules.d/azure.yml.disabled index 8005c0e74c85..55c0699458b6 100644 --- a/x-pack/metricbeat/modules.d/azure.yml.disabled +++ b/x-pack/metricbeat/modules.d/azure.yml.disabled @@ -104,6 +104,11 @@ # tenant_id: '${AZURE_TENANT_ID:""}' # subscription_id: '${AZURE_SUBSCRIPTION_ID:""}' # refresh_list_interval: 600s +# # How far back to query usage data on each fetch (default 24h). Azure Cost Management data +# # can be updated for up to 72 hours, so set this to 72h to capture late cost corrections. +# billing_usage_lookback: 72h +# # Length of the forecast period (default 720h / 30 days). +# billing_forecast_window: 720h #- module: azure # metricsets: From 119be81001fb72920c0a6a505ddf5ff52c523506 Mon Sep 17 00:00:00 2001 From: stefans-elastic Date: Thu, 23 Jul 2026 17:13:57 +0300 Subject: [PATCH 2/5] Fix Azure billing usage date filter Use range comparisons for the Azure Consumption usage details filter so configurable multi-day lookback windows work for legacy billing scopes as well as the startDate/endDate path. Add a focused unit test that verifies the generated usage details filter and request dates for a multi-day window. Assisted-By: Cursor Co-authored-by: Cursor --- .../metricbeat/module/azure/billing/client.go | 2 +- .../module/azure/billing/client_test.go | 30 +++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/x-pack/metricbeat/module/azure/billing/client.go b/x-pack/metricbeat/module/azure/billing/client.go index d0c0acf95e42..4550a45c0e81 100644 --- a/x-pack/metricbeat/module/azure/billing/client.go +++ b/x-pack/metricbeat/module/azure/billing/client.go @@ -71,7 +71,7 @@ func (client *Client) GetMetrics(timeOpts TimeIntervalOptions) (Usage, error) { Infow("Getting usage details for scope") filter := fmt.Sprintf( - "properties/usageStart eq '%s' and properties/usageEnd eq '%s'", + "properties/usageStart ge '%s' and properties/usageEnd le '%s'", timeOpts.usageStart.Format(time.RFC3339Nano), timeOpts.usageEnd.Format(time.RFC3339Nano), ) diff --git a/x-pack/metricbeat/module/azure/billing/client_test.go b/x-pack/metricbeat/module/azure/billing/client_test.go index 59fcf24b9d74..fd73c43a920d 100644 --- a/x-pack/metricbeat/module/azure/billing/client_test.go +++ b/x-pack/metricbeat/module/azure/billing/client_test.go @@ -65,3 +65,33 @@ func TestClient(t *testing.T) { m.AssertExpectations(t) }) } + +func TestClientUsesRangeFilterForUsageDetails(t *testing.T) { + opts := TimeIntervalOptions{ + usageStart: time.Date(2026, 7, 19, 0, 0, 0, 0, time.UTC), + usageEnd: time.Date(2026, 7, 21, 23, 59, 59, 0, time.UTC), + forecastStart: time.Date(2026, 7, 19, 0, 0, 0, 0, time.UTC), + forecastEnd: time.Date(2026, 8, 17, 23, 59, 59, 0, time.UTC), + } + + client := NewMockClient(logptest.NewTestingLogger(t, "")) + client.Config = azure.Config{SubscriptionId: "sub"} + m := &MockService{} + + expectedFilter := "properties/usageStart ge '2026-07-19T00:00:00Z' and properties/usageEnd le '2026-07-21T23:59:59Z'" + m.On( + "GetUsageDetails", + "subscriptions/sub", + "properties/meterDetails", + expectedFilter, + armconsumption.MetrictypeActualCostMetricType, + "2026-07-19", + "2026-07-21", + ).Return(armconsumption.UsageDetailsListResult{}, nil) + m.On("GetForecast", "subscriptions/sub", opts.forecastStart, opts.forecastEnd).Return(armcostmanagement.QueryResult{}, nil) + client.BillingService = m + + _, err := client.GetMetrics(opts) + assert.NoError(t, err) + m.AssertExpectations(t) +} From 228edcb216943de69d0d50863ae13a3683233663 Mon Sep 17 00:00:00 2001 From: stefans-elastic Date: Wed, 29 Jul 2026 11:29:07 +0300 Subject: [PATCH 3/5] Update docs/reference/metricbeat/metricbeat-metricset-azure-billing.md Co-authored-by: Visha Angelova <91186315+vishaangelova@users.noreply.github.com> --- docs/reference/metricbeat/metricbeat-metricset-azure-billing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md b/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md index 4327252f73e1..9d492f7fb8c5 100644 --- a/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md +++ b/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md @@ -39,7 +39,7 @@ This metricset allows users to retrieve usage details and forecast information o If none of the 2 options are entered then the subscription ID will be used as scope. -`billing_usage_lookback` +`billing_usage_lookback` {applies_to}`stack: ga 9.3+` : (*duration*, default: `24h`) How far back to query usage data on each fetch. Azure Cost Management data is documented to be updated for up to 72 hours after a billing period closes, so increasing this to `72h` ensures that late cost corrections from Azure are re-queried and ingested. From a531756986077c7c79a7735d910459ab815e2870 Mon Sep 17 00:00:00 2001 From: stefans-elastic Date: Wed, 29 Jul 2026 11:29:18 +0300 Subject: [PATCH 4/5] Update docs/reference/metricbeat/metricbeat-metricset-azure-billing.md Co-authored-by: Visha Angelova <91186315+vishaangelova@users.noreply.github.com> --- docs/reference/metricbeat/metricbeat-metricset-azure-billing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md b/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md index 9d492f7fb8c5..b16234290297 100644 --- a/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md +++ b/docs/reference/metricbeat/metricbeat-metricset-azure-billing.md @@ -44,7 +44,7 @@ If none of the 2 options are entered then the subscription ID will be used as sc data is documented to be updated for up to 72 hours after a billing period closes, so increasing this to `72h` ensures that late cost corrections from Azure are re-queried and ingested. -`billing_forecast_window` +`billing_forecast_window` {applies_to}`stack: ga 9.3+` : (*duration*, default: `720h`) The length of the forecast period, counted forward from the forecast start date (reference time minus 2 days). Defaults to 30 days (`720h`). From 939d95d922a55ba0993cef55e0dd430e544f982a Mon Sep 17 00:00:00 2001 From: stefans-elastic Date: Wed, 29 Jul 2026 11:46:55 +0300 Subject: [PATCH 5/5] docs: sync azure billing applies_to annotations to generated source The `{applies_to}` version annotations for `billing_usage_lookback` and `billing_forecast_window` were applied directly to the generated file `docs/reference/metricbeat/metricbeat-metricset-azure-billing.md` when accepting review suggestions. That file is produced from the metricset's `_meta/docs.md` by `metricbeat/scripts/mage/docs_collector.go`, so the `check-no-changes` CI target regenerated it and failed on the drift. Apply the same annotations to the source `_meta/docs.md` so regeneration reproduces the committed output. Assisted-By: Cursor Co-authored-by: Cursor --- x-pack/metricbeat/module/azure/billing/_meta/docs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/x-pack/metricbeat/module/azure/billing/_meta/docs.md b/x-pack/metricbeat/module/azure/billing/_meta/docs.md index 24bcad9ffc85..257278e4b9df 100644 --- a/x-pack/metricbeat/module/azure/billing/_meta/docs.md +++ b/x-pack/metricbeat/module/azure/billing/_meta/docs.md @@ -27,11 +27,11 @@ This metricset allows users to retrieve usage details and forecast information o If none of the 2 options are entered then the subscription ID will be used as scope. -`billing_usage_lookback` +`billing_usage_lookback` {applies_to}`stack: ga 9.3+` : (*duration*, default: `24h`) How far back to query usage data on each fetch. Azure Cost Management data is documented to be updated for up to 72 hours after a billing period closes, so increasing this to `72h` ensures that late cost corrections from Azure are re-queried and ingested. -`billing_forecast_window` +`billing_forecast_window` {applies_to}`stack: ga 9.3+` : (*duration*, default: `720h`) The length of the forecast period, counted forward from the forecast start date (reference time minus 2 days). Defaults to 30 days (`720h`).