Currently, prmon provides excellent support for NVIDIA GPU monitoring through the nvidiamon component, utilizing both the NVIDIA Management Library (NVML) and nvidia-smi as a fallback. However, as High Energy Physics (HEP) computing sites increasingly adopt AMD hardware (such as the AMD Instinct™ MI series accelerators), there is a growing need to monitor AMD GPU resource consumption alongside CPU and memory metrics.
I propose the addition of a new monitoring component (e.g., amdgpu-mon or extending the existing monitoring logic) to support AMD GPUs.
Proposed Implementation Details:
Primary Monitoring Path (ROCm SMI Library): Similar to how nvidiamon uses libnvidia-ml.so, the AMD implementation should preferentially use the ROCm SMI (System Management Interface) library (librocm_smi64.so).
This would allow for low-overhead, direct access to metrics like GPU utilization, VRAM usage (FB memory), and power consumption.
Symbols should be loaded dynamically via dlopen at runtime to ensure prmon remains portable and doesn't require AMD drivers installed at build time.
Fallback Monitoring Path (rocm-smi / amdgpu-smi): If the library is not available, prmon should fall back to parsing the output of the CLI tools provided by the AMD stack.
Metrics to Track (Parity with NVIDIA): To maintain consistency in the output files (prmon.txt and prmon.json), we should aim to collect:
gpufbmem: Current GPU VRAM usage.
gpusmpct: GPU compute unit utilization percentage.
gpumempct: GPU memory controller utilization percentage.
gpupower: Current power draw (if supported by the hardware).
Hardware Information: The get_hardware_info method should be updated to capture AMD-specific details like the device name (e.g., "AMD Instinct MI210") and the number of available cards.
Context: As many WLCG/HSF sites are transitioning towards heterogeneous computing environments, having a unified tool that can monitor both NVIDIA and AMD accelerators is critical for accurate cost and performance evaluation of HEP workflows.
Additional Considerations:
Should we consider renaming nvidiamon to a more generic gpumon and use a plugin architecture for different vendors?
How should we handle multi-GPU environments where cards from different vendors might (theoretically) exist?
Currently, prmon provides excellent support for NVIDIA GPU monitoring through the nvidiamon component, utilizing both the NVIDIA Management Library (NVML) and nvidia-smi as a fallback. However, as High Energy Physics (HEP) computing sites increasingly adopt AMD hardware (such as the AMD Instinct™ MI series accelerators), there is a growing need to monitor AMD GPU resource consumption alongside CPU and memory metrics.
I propose the addition of a new monitoring component (e.g., amdgpu-mon or extending the existing monitoring logic) to support AMD GPUs.
Proposed Implementation Details:
Primary Monitoring Path (ROCm SMI Library): Similar to how nvidiamon uses libnvidia-ml.so, the AMD implementation should preferentially use the ROCm SMI (System Management Interface) library (librocm_smi64.so).
This would allow for low-overhead, direct access to metrics like GPU utilization, VRAM usage (FB memory), and power consumption.
Symbols should be loaded dynamically via dlopen at runtime to ensure prmon remains portable and doesn't require AMD drivers installed at build time.
Fallback Monitoring Path (rocm-smi / amdgpu-smi): If the library is not available, prmon should fall back to parsing the output of the CLI tools provided by the AMD stack.
Metrics to Track (Parity with NVIDIA): To maintain consistency in the output files (prmon.txt and prmon.json), we should aim to collect:
gpufbmem: Current GPU VRAM usage.
gpusmpct: GPU compute unit utilization percentage.
gpumempct: GPU memory controller utilization percentage.
gpupower: Current power draw (if supported by the hardware).
Hardware Information: The get_hardware_info method should be updated to capture AMD-specific details like the device name (e.g., "AMD Instinct MI210") and the number of available cards.
Context: As many WLCG/HSF sites are transitioning towards heterogeneous computing environments, having a unified tool that can monitor both NVIDIA and AMD accelerators is critical for accurate cost and performance evaluation of HEP workflows.
Additional Considerations:
Should we consider renaming nvidiamon to a more generic gpumon and use a plugin architecture for different vendors?
How should we handle multi-GPU environments where cards from different vendors might (theoretically) exist?