Skip to content

[lustre] Client stats silently missing on FSx for Lustre — bundled lctl 2.15.x does not support "get_param -y" #24475

Description

@kazu0716

Agent / integration: Datadog Agent 7.80.2, lustre check 1.5.0 (latest; checked CHANGELOG — no related fix)

Environment: Amazon FSx for Lustre (PERSISTENT_2) clients on Ubuntu 22.04 — lustre-client-utils 2.15.6-1fsx28 (lctl 2.15.6), SageMaker HyperPod fleet

What happens
With node_type: client, only lustre.device.health and lustre.net.* (lnetctl-based) are emitted. All stats-based metrics (llite / mdc / osc stats, curated params, enable_extra_params) are silently missing.

The check invokes lctl get_param -ny <param>, but -y (YAML output) was only added in Lustre 2.16. The lctl shipped with the current FSx for Lustre client (2.15.x) rejects it:

$ lctl get_param --help
usage: get_param [-F|n|-N|-R] <param_path1 param_path2 ...>   # no -y

Agent debug log — every stats param fails with rc=4, logged at DEBUG only, check status stays [OK]:

(check.py:258) Running command: ['sudo', '/usr/sbin/lctl', 'get_param', '-ny', 'llite.<fs>-ffff....kbytesfree']
(check.py:263) Command [...] exited with returncode 4. Captured stderr: get_param: invalid option -- 'y'

Why it matters
FSx for Lustre is presumably the largest managed-Lustre install base, and its newest client packages are 2.15-based. For all these users the integration's client-side metrics (and most of the Lustre - Overview dashboard) are empty, with no visible error — we only found this by running datadog-agent check lustre -l debug during an incident investigation.

Suggested fix

  1. Detect -y support once at init (or on first rc=4) and fall back to lctl get_param -n with plain-text parsing, or
  2. At minimum, surface stats-collection failures as a check WARNING / service check instead of DEBUG, so the data gap is visible.

Repro: any FSx for Lustre client with lustre-client-utils 2.15.x → enable lustre check with node_type: clientdatadog-agent check lustre -l debug.

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions