[ISSUE #9698] Add scriptable cluster health checks

[btlqql]

Which Issue(s) This PR Fixes

• Fixes [Feature] How can healthcheck be conducted for the current project in docker？ #9698

Brief Description

Problem

RocketMQ distributions do not provide a health-check command with reliable process exit codes. Container users currently have to install tools such as nc or nmap and can only test whether a port is open, or build a producer/consumer probe that creates traffic and resources.

Root Cause

The existing mqadmin brokerStatus command is designed for interactive diagnostics: it prints runtime data, does not validate a cluster as a whole, and MQAdminStartup catches command failures without exposing a health-specific exit-code contract. There is also no NameServer-only probe for an empty bootstrap cluster.

Changes

• add mqadmin clusterHealth for direct-broker, NameServer-only, selected-cluster, and all-cluster checks
• discover brokers through NameServer and probe broker runtime RPCs concurrently with a bounded overall deadline
• support master-only checks and an optional brokerActive requirement
• provide deterministic text and JSON reports with per-broker latency, version, status, and failure details
• add mqhealthcheck and mqhealthcheck.cmd distribution launchers with stable exit codes: 0 healthy, 1 unhealthy, 2 usage or execution error
• keep checks read-only: no topics, messages, producers, or consumers are created
• add focused coverage for discovery failures, malformed responses, direct checks, filtering, master/slave selection, active-state validation, deadlines, formatting, lifecycle cleanup, and exit codes

Impact

Container images and orchestration systems can use the bundled command directly in Docker HEALTHCHECK or readiness scripts. Operators can distinguish a live RPC service from an open TCP port, inspect machine-readable details, and check an empty NameServer independently from broker availability. Existing mqadmin commands and behavior are unchanged.

Example uses:

• mqhealthcheck -n 127.0.0.1:9876 -s
• mqhealthcheck -b 127.0.0.1:10911
• mqhealthcheck -n 127.0.0.1:9876 -c DefaultCluster -m -a -f json

How Did You Test This Change?

JDK 8 (Temurin 8.0.482):

• focused health-check tests: 54 tests, 0 failures, 0 errors, 0 skipped
• full rocketmq-tools suite: 201 tests, 0 failures, 0 errors, 5 existing skips; BUILD SUCCESS
• tools reactor package build: 7 modules; BUILD SUCCESS
• Checkstyle: 0 violations

Commands:

mvn -pl tools test -Dtest=ClusterHealthRequestTest,ClusterHealthReportTest,ClusterHealthCheckerTest,ClusterHealthSubCommandTest,MQHealthCheckStartupTest -Dspotbugs.skip=true -Djacoco.skip=true

mvn -pl tools test -Dspotbugs.skip=true -Djacoco.skip=true

mvn -pl tools -am package -DskipTests -Dspotbugs.skip=true -Djacoco.skip=true

[RockteMQ-AI]

Review by github-manager-bot

Summary

This PR adds a scriptable mqadmin clusterHealth command that performs broker and NameServer health checks with reliable exit codes — addressing a real gap for container-based deployments that currently lack a built-in health probe.

Findings

• [Info] ClusterHealthChecker.probeAll() — The overall deadline via ScheduledExecutorService.shutdownNow() is a good pattern. One consideration: if a broker probe future is cancelled mid-flight, the underlying invokeSync on the remoting layer may not immediately respect the interrupt. This is acceptable for a health check (best-effort), but worth documenting that the timeout is a soft upper bound.

• [Info] ClusterHealthChecker.selectTargets() — The deduplication via identity string (cluster + broker + brokerId + address) is correct and handles edge cases like the same address appearing under different broker names. Good defensive null/empty checks throughout.

• [Info] ClusterHealthReport.toJson() — The JSON output is manually constructed (no Jackson/Gson dependency). This is a reasonable choice for the tools module to avoid pulling in extra dependencies. Just ensure special characters in broker names/addresses are properly escaped if they ever contain quotes or backslashes.

• [Info] ClusterHealthSubCommand — The exit code contract (0 = healthy, 1 = unhealthy, 2 = usage error) is clean and follows standard Unix conventions. The --format json option is excellent for automation.

• [Info] mqhealthcheck shell script — Follows the same pattern as other mq* scripts. Consistent and clean.

Positive Observations

• Well-structured: Clear separation between request, checker, report, and CLI layers.
• Good test coverage: Tests cover concurrency, timeout, malformed metadata, masters-only filtering, and output formatting.
• Backward compatible: Only adds new functionality, no existing APIs changed.
• Concurrency safety: Bounded thread pool, overall deadline, per-broker exception isolation.

Suggestions

• Consider adding a --verbose flag that includes per-broker latency in the text output for interactive debugging.
• The ClusterHealthRequest.validate() method could benefit from a check that parallelism > 0 (currently only checks >= 1 via timeoutMillis >= 0).

Overall this is a solid addition to the operations tooling. 👍

---

Automated review by github-manager-bot