Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions doc/bmc/leakage_detection_hld.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Due to the excessive heat generated by the equipment, traditional air-cooling me
## 2. Requirements
1. Monitoring the liquid colling leakge detection sensor, and alarm accordingly.
2. For platform that doesn't support liquid cooling at all, there should be no further performance overheading.
3. Support controlled per-sensor leak-test injection for platform validation without allowing a test-only leak to trigger host power control.

## 3. Detection and alarm flow
The leak alarm process is straightforward. The platform API first acquires the status of the leak detection sensors. Then, thermalctld has a thread that calling the API and in turn notifies system health monitor, who ultimately sends out a gNMI event.
Expand Down Expand Up @@ -65,6 +66,21 @@ Class LeakageSensor(sensor_base):

```

### Leak-test API

Platforms that support controlled leak simulation shall expose an optional `LeakageSensorTestBase` instance from `LiquidCoolingBase.get_leak_sensor_test()`. Platforms that do not support it return `None`.

| Method | Description |
|--------|-------------|
| `is_leak_test_supported()` | Return whether controlled leak-test injection is supported |
| `set_test_leak(sensor_name, enable)` | Enable or disable test injection for one sensor |
| `is_test_leak_enabled(sensor_name)` | Return whether test injection is enabled for one sensor |
| `clear_test_leaks()` | Clear test injection on every sensor |

`sonic-mgmt` obtains this interface through a privileged test helper on the DUT; it is not exposed through the normal production CLI. Test state must be cleared at boot and test-interface use must be audited.

On platforms with hardware test bits, enabling a test may cause `is_leak()` to return `True` because the hardware leak indication is forced. `thermalctld` uses `is_test_leak_enabled()` to classify that indication as test-injected and excludes it from the physical result used for policy enforcement. If the underlying physical state is not independently observable while a hardware test bit is enabled, test injection is restricted to controlled validation where a concurrent real leak can be ruled out.

## 5. Thermal control daemon
A new object `LiquidCoolingUpdater` will be added to the Thermal Control daemon that dedicated monitoring the liquid device status.

Expand All @@ -78,6 +94,8 @@ enable_liquid_cooling: true,
liquid_cooling_update_interval: 0.5
```

The thread records physical leak state and optional test-injection state separately. It aggregates physical leaks for policy enforcement and test injections for validation visibility. A test injection must not increase, suppress, or otherwise change the physical aggregate used for enforcement.

Once the leakage event has been detected, the thread will write it to state db to notify the system health monitor. Meanwhile, the syslog error message will be printed out.
"Liquid cooling leakge has been detected on sensor{}"

Expand All @@ -104,6 +122,12 @@ key = LIQUID_COOLING_DEVICE|leakage_sensors{X}
name = STR ; sensor name
leaking = STR ; 1 or 0 to indicate leakage status
leak_status = STR ; Covert leaking 1/0 to yes/no, default as N/A to prevent sensor not readable
test_leak = STR ; Enabled or Disabled

key = SYSTEM_LEAK_STATUS|system
; field = value
device_leak_status = STR ; physical aggregate used for policy enforcement
test_device_leak_status = STR ; test-injection aggregate; validation only
```

## 6. system health monitor
Expand Down Expand Up @@ -154,3 +178,6 @@ A mock testing should be created to demonstrate the functionality of this implem
2. state db is rightly updated
3. GNMI event had been sent out
4. `show platform leakage status` command output is correct
5. Test injection is available only when `LiquidCoolingBase.get_leak_sensor_test()` returns a supported test interface.
6. A test-only critical aggregate is recorded and audited but never invokes graceful shutdown, power-off, power-cycle, GNOI, or a platform power-control API.
7. A physical critical leak still invokes the configured policy action.
27 changes: 24 additions & 3 deletions doc/bmc/sonicBMC/pmon-bmc-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,10 @@ On an Event
- SKIP if `system_leak_policy` is `disabled` in LEAK_CONTROL_POLICY [2.3.1 Config commands](#231-config-commands)
- Read system_minor_leak_action from LEAK_CONTROL_POLICY; ==> dispatch_action(system_minor_leak_action)

- if TEST_SYSTEM_LEAK_EVENT (test_device_leak_status == CRITICAL or MINOR in SYSTEM_LEAK_STATUS)
- Syslog/audit the configured action that would apply to an equivalent physical leak.
- Do not dispatch an action and do not invoke GNOI, graceful shutdown, power-off, power-cycle, or a platform power-control API.

- if MINOR External-Rack-Mgr Alert event
- SKIP if `rack_mgr_leak_policy` is `disabled` in LEAK_CONTROL_POLICY [2.3.1 Config commands](#231-config-commands)
- Read rack_mgr_minor_alert_action from LEAK_CONTROL_POLICY; ==> dispatch_action(rack_mgr_minor_alert_action)
Expand Down Expand Up @@ -432,12 +436,12 @@ There is a thread to check the leak sensors and store it in the LIQUID_COOLING_I
```
Loop on this logic
(i) Check system leak sensors using platform API
-- store the result in LIQUID_COOLING_INFO table
-- store physical leak state and optional test-injection state separately in LIQUID_COOLING_INFO table
-- Per-sensor leak_severity can be CRITICAL or MINOR (sensor-level assessment)

```

The main thermalctld daemon will run the sonic thermal policy based on the number and severity of leak sensors with leak
The main thermalctld daemon aggregates physical and test-injection leak states independently. Only the physical aggregate is used for SONiC leak policy enforcement; test-only results are used for validation visibility and audit.

```
- Subscribe to LIQUID_COOLING_INFO to check if there is any change in leak sensor status
Expand All @@ -455,7 +459,8 @@ The main thermalctld daemon will run the sonic thermal policy based on the numbe
- Additional considerations, the timers can be configured per leak sensor profile.
- MAX-T secs defined before which a MINOR leak can be considered CRITICAL.

- Update the system SYSTEM_LEAK_STATUS table with the severity of leak. This will be used in bmcctld process.
- Update `device_leak_status` with physical-leak severity. This is the only leak severity used by bmcctld.
- Update `test_device_leak_status` with test-injection severity. This is used for validation visibility and audit only.

```

Expand All @@ -467,6 +472,7 @@ key = LIQUID_COOLING_INFO|leakage_sensors{X} ; leak data
; field = value
name = STR ; sensor name
leaking = STR ; Yes or No to indicate leak status
test_leak = STR ; Enabled or Disabled
leak_sensor_status = STR ; Is Leak sensor good or faulty.
type = STR ; leak sensor type
location = STR ; leak sensor location
Expand All @@ -480,6 +486,7 @@ max_minor_duration_sec = integer ; MAX-T se
key = SYSTEM_LEAK_STATUS|system ; system bmc leak status in STATE DB
; field = value
device_leak_status = "status" ; CRITICAL_SYSTEM_LEAK/MINOR_SYSTEM_LEAK (system aggregate level)
test_device_leak_status = STR ; CRITICAL, MINOR, or None; test injection only
timestamp = STR ; timestamp when this status is recorded.
```

Expand Down Expand Up @@ -515,6 +522,19 @@ This base class is already defined in sonic-platform-common, additional new plat

The get_leak_severity() API call has to determine whether any condition indicates a severe leak detected from a leak sensor.This decision could be based on:

#### LeakageSensorTestBase

`LeakageSensorTestBase` is an optional platform test interface used by `sonic-mgmt` and platform validation tooling. It is not exposed through the normal production CLI.

| Method | Description |
|---------|----------|
| is_leak_test_supported() | Return whether this platform supports controlled leak-test injection |
| set_test_leak(sensor_name, enable) | Enable or disable test injection for one sensor |
| is_test_leak_enabled(sensor_name) | Return whether test injection is enabled for one sensor |
| clear_test_leaks() | Clear test injection on every sensor |

Platforms that do not implement this interface do not support test injection. Test state is cleared at boot and interface use is audited. On platforms with hardware test bits, enabling a test may cause `is_leak()` to return `True` because the hardware leak indication is forced. `thermalctld` uses `is_test_leak_enabled()` to classify that indication as test-injected and excludes it from the physical result used for policy enforcement. If the underlying physical state is not independently observable while a hardware test bit is enabled, test injection is restricted to controlled validation where a concurrent real leak can be ruled out.

(i) Zone criticality:
The physical location of the leak sensor. For example, if a spot leak sensor is located near critical components (ASIC/CPU) and the measured value (e.g. resistance or other leak-detection metric) crosses a platform-defined threshold, it should be treated as a critical leak.

Expand All @@ -540,6 +560,7 @@ This base class is already defined in sonic-platform-common.
| get_leak_sensor(index) | Y | Get per-leak-sensor status |
| get_leak_sensor_status() | Y | Get all leak sensor status |
| get_all_leak_sensors() | Y | Get list of all leak sensors |
| get_leak_sensor_test() | New | Return the optional `LeakageSensorTestBase` instance, or `None` when leak-test injection is unsupported |


#### ModuleBase
Expand Down