-
Notifications
You must be signed in to change notification settings - Fork 190
feat: Extend AKS skill to perform cluster reviews #2076
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
johnbilliris (johnbilliris)
wants to merge
27
commits into
microsoft:main
Choose a base branch
from
johnbilliris:azure-kubernetes-review
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 11 commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
24252ed
Introduce AKS review skill
johnbilliris b94151d
Fix for tests/skills.json validation failure
johnbilliris 63914d8
Further fix for tests/skills.json validation failure
johnbilliris 39b9284
Apply suggestions from code review
johnbilliris 3ba2854
Apply suggestions from code review
johnbilliris 8bf45c9
Changes from review and feedback
johnbilliris 70e0144
Merge branch 'microsoft:main' into azure-kubernetes-review
johnbilliris f15fb09
Changes to fix charCount and tests
johnbilliris d684565
Merge branch 'microsoft:main' into azure-kubernetes-review
johnbilliris 241f078
azure-kubernetes Day 2 description and trigger moved into reference file
johnbilliris 4674889
fix: shrink azure-kubernetes description to fit Copilot CLI char budget
johnbilliris 3e97d69
Three issues fixed
johnbilliris dade136
Merge branch 'main' into azure-kubernetes-review
johnbilliris 789e8c3
refresh azure-kubernetes trigger snapshots
johnbilliris f1de12a
Merge remote-tracking branch 'upstream/main' into azure-kubernetes-re…
johnbilliris 60a7d58
Merge branch 'microsoft:main' into azure-kubernetes-review
johnbilliris 2720e79
Merge branch 'microsoft:main' into azure-kubernetes-review
johnbilliris dd5f348
Fix for Skill Evaluations
johnbilliris 6d9f192
Merge branch 'microsoft:main' into azure-kubernetes-review
johnbilliris 64fab45
Fixed azure-compute test snapshot
johnbilliris 9aa715f
Merge main into this branch
johnbilliris c49cb3b
Moved report template to correct folder
johnbilliris 0554371
(Manually) Revert azure-compute skill
johnbilliris 38de1ce
Fixes based off PR comments
johnbilliris 91489b0
Reduce char count due to budget
johnbilliris 533e7eb
Merge branch 'azure-kubernetes-review' into main
johnbilliris 71d7eb5
Merge pull request #2 from johnbilliris/main
johnbilliris File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
256 changes: 256 additions & 0 deletions
256
plugin/skills/azure-kubernetes/assets/aks-audit-report-template.md
Large diffs are not rendered by default.
Oops, something went wrong.
204 changes: 204 additions & 0 deletions
204
plugin/skills/azure-kubernetes/references/aks-checklist-matrix.md
Large diffs are not rendered by default.
Oops, something went wrong.
223 changes: 223 additions & 0 deletions
223
plugin/skills/azure-kubernetes/references/aks-container-best-practices.md
Large diffs are not rendered by default.
Oops, something went wrong.
139 changes: 139 additions & 0 deletions
139
plugin/skills/azure-kubernetes/references/aks-diagnostics-detectors.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,139 @@ | ||
| # AKS Diagnostics Detectors Reference | ||
|
|
||
| This reference documents the AKS built-in diagnostics detectors available through Azure's "Diagnose and solve problems" capability. These detectors provide runtime risk alerts and health assessments that supplement the static checklist matrix. | ||
|
|
||
| ## Extraction Methods (Priority Order) | ||
|
|
||
| ### 1. AKS MCP Server (Primary) | ||
|
|
||
| Use the AKS MCP tools to extract diagnostics data programmatically: | ||
|
|
||
| 1. **List all available detectors**: | ||
| - Tool: `mcp_aks_mcp_list_detectors` | ||
| - Parameters: `cluster_resource_id` (full ARM resource ID of the managed cluster, e.g. `/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ContainerService/managedClusters/<cluster>`) | ||
| - Returns: List of all detector names, descriptions, and categories. | ||
|
|
||
| 2. **Run detectors by category**: | ||
| - Tool: `mcp_aks_mcp_run_detectors_by_category` | ||
| - Parameters: `cluster_resource_id`, `category` | ||
| - Run once per category listed below. | ||
|
|
||
| 3. **Run a specific detector** (for deeper investigation): | ||
| - Tool: `mcp_aks_mcp_run_detector` | ||
| - Parameters: `cluster_resource_id`, `detector_name` | ||
|
|
||
| > 💡 **Tip:** A live `mcp_aks_mcp_list_detectors` call against a representative AKS cluster returned ~119 detectors across 14 categories (8 core + 6 supplemental). Treat the cluster's live inventory as authoritative; the tables below are reference baselines. | ||
|
johnbilliris marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### 2. Azure REST API (Fallback) | ||
|
|
||
| If MCP tools are unavailable, use the AKS Diagnostics REST API via Azure CLI: | ||
|
|
||
| ```bash | ||
| # List all detectors | ||
| az rest --method get \ | ||
| --url "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroup>/providers/Microsoft.ContainerService/managedClusters/<clusterName>/detectors?api-version=2025-10-01" | ||
|
|
||
| # Run a specific detector | ||
| az rest --method get \ | ||
| --url "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroup>/providers/Microsoft.ContainerService/managedClusters/<clusterName>/detectors/<detectorName>?api-version=2025-10-01" | ||
| ``` | ||
|
|
||
| ### 3. Azure AppLens / Diagnostics API (Alternative) | ||
|
|
||
| ```bash | ||
| # List available detectors via the diagnostics provider | ||
| az rest --method get \ | ||
| --url "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroup>/providers/Microsoft.ContainerService/managedClusters/<clusterName>/providers/Microsoft.ResourceHealth/diagnostics?api-version=2025-10-01" | ||
| ``` | ||
|
|
||
| ## Detector Categories | ||
|
|
||
| The detector inventory has two layers of category handling: | ||
|
|
||
| 1. **Core audit categories**: the eight categories that must always be queried during an audit. | ||
| 2. **Supplemental live categories**: additional categories that may appear in the live detector inventory for a specific cluster and should be incorporated when present. | ||
|
|
||
| ### Core Audit Categories | ||
|
|
||
| | Category | Audit Relevance | Pillar Alignment | | ||
| | --- | --- | --- | | ||
| | Best Practices | Direct alignment with checklist controls | All pillars | | ||
| | Cluster and Control Plane Availability and Performance | API server health, etcd latency, control plane SLA | Reliability, Performance Efficiency | | ||
| | Connectivity Issues | DNS, networking, load balancer, ingress, egress | Security, Reliability | | ||
| | Create, Upgrade, Delete and Scale | Upgrade readiness, scaling failures, provisioning issues | Reliability, Operational Excellence, Performance Efficiency | | ||
| | Deprecations | Deprecated APIs, features, or configurations requiring action | Operational Excellence | | ||
| | Identity and Security | RBAC, Microsoft Entra ID, secrets, network policies, Defender findings | Security | | ||
| | Node Health | Node readiness, resource pressure, OS/kernel issues | Reliability, Performance Efficiency | | ||
| | Storage | PV/PVC issues, CSI driver health, disk attach failures | Reliability | | ||
|
|
||
| ### Supplemental Live Categories | ||
|
|
||
| The live detector catalog can also return additional categories. These are not a replacement for the core eight. They should be enumerated from `mcp_aks_mcp_list_detectors` and incorporated into the report when present. | ||
|
|
||
| | Category | Typical Content | Handling Guidance | | ||
| | --- | --- | --- | | ||
| | Risk Alerts | Cross-category risk summaries such as availability, reliability, and support-eligibility alerts (e.g. `riskalerts-availability`, `riskalerts-reliability`, `riskalerts-support-eligibility`) | Always capture. Treat as high-value supplemental evidence and include all Critical and Warning results in the Risk Alerts table. | | ||
| | Control Plane | Focused control plane detectors such as scheduler-specific analysis (e.g. `KubeScheduler`) | Capture and cross-reference to reliability and performance checks where applicable. | | ||
| | Keystone | Service-specific platform diagnostics surfaced by the AKS diagnostics backend (e.g. `keystone`) | Capture as supplemental evidence. If no checklist item maps cleanly, keep in the diagnostics section only. | | ||
| | Workflow Nodes | Guided troubleshooting nodes rather than direct compliance detectors (e.g. `operation-troubleshooter`, Kusto spoke providers) | Record availability and use them to drive deeper investigation, but do not score them as standalone checklist controls unless they emit concrete findings. | | ||
| | zWorkflowDetectors | Reconciliation workflow detectors (e.g. `ReconcileManagedCluster`, `ReconcileNodePool`) used for orchestration rather than direct compliance scoring | Record availability. Run only when investigating a specific reconcile/operation failure; do not score as standalone checklist controls. | | ||
| | Analysis - L2 Support Topic | Topic-level analysis aggregators that fan out to multiple underlying detectors (e.g. `st-l2-analysis-connectivity`, `st-l2-analysis-crud`, `st-l2-analysis-storage`) | Run when category-level execution is needed for the matching topic. Treat outputs as cross-references to the relevant core category rather than as new checklist controls. | | ||
| | Analysis - L3 Support Topic | Scenario-specific analysis nodes for targeted investigations (e.g. `st-l3-analysis-cluster-upgrade`, `st-l3-node-high-cpu-mem`, `st-l3-analysis-creating-nodepool`) | Use on demand to drill into a specific failure scenario surfaced by core detectors or risk alerts. Capture findings as supplemental evidence linked to the impacted checklist item. | | ||
|
|
||
| ### Category Execution Rules | ||
|
|
||
| 1. Query all eight core audit categories on every audit. | ||
| 2. Enumerate supplemental live categories from the detector inventory on every audit. | ||
| 3. If a supplemental category supports category-level execution, run it. | ||
| 4. If category-level execution is not supported, run the underlying detectors individually where possible. | ||
| 5. If a supplemental category is inventory-only (for example a workflow or overview node), record it as supplemental metadata with handling notes. | ||
| 6. Do not let supplemental-category coverage replace or weaken the requirement to query the core eight. | ||
|
|
||
| ## Severity Levels | ||
|
|
||
| Detector findings use the following severity levels: | ||
|
|
||
| | Severity | Meaning | Audit Impact | | ||
| | --- | --- | --- | | ||
| | Critical | Active issue causing service impact or significant risk | Must be addressed; may override checklist status to `Does not meet` | | ||
| | Warning | Potential risk or degraded posture detected | Should be addressed; may affect checklist status | | ||
| | Info | Informational finding with no immediate risk | Record as evidence; no status change required | | ||
| | Healthy | Detector ran and no issues detected | Positive evidence for checklist items | | ||
| | None | Detector returned no severity (informational or not applicable) | Record as supplemental context; verify the detector executed correctly | | ||
|
|
||
| ## Mapping Detectors to Checklist Items | ||
|
|
||
| When a detector finding relates to an existing checklist item (from the checklist matrix), cross-reference it: | ||
|
|
||
| | Detector Category | Likely Checklist Alignment | | ||
| | --- | --- | | ||
| | Best Practices | `AKS-OP-*`, `AKS-DEV-*`, `AKSC-*`, `COST-*`, `CTR-*`, `OPS-04` | | ||
| | Cluster and Control Plane Availability and Performance | `REL-01` through `REL-08`, `PERF-01` through `PERF-05`, `OPS-03` | | ||
| | Connectivity Issues | `SEC-03`, `SEC-04`, `SEC-09`, `AKS-NET-*`, `AKSC-NET-*` | | ||
| | Create, Upgrade, Delete and Scale | `AKS-CSEC-01`, `AKS-CSEC-02`, `OPS-01`, `OPS-02` | | ||
| | Deprecations | `AKS-CSEC-01`, `OPS-01` | | ||
| | Identity and Security | `SEC-01` through `SEC-10`, `AKS-ID-*`, `OPS-05` | | ||
| | Node Health | `REL-04`, `REL-05`, `REL-06`, `PERF-01`, `PERF-03` | | ||
| | Storage | `AKS-STO-01`, `AKS-STO-02`, `AKSC-BCDR-03` | | ||
|
|
||
| ### Cross-Reference Rules | ||
|
|
||
| 1. If a detector finding directly validates or contradicts a checklist item, update that item's `Evidence Summary` and `Comments` with the detector output. | ||
| 2. If a detector finding has no matching checklist item, include it only in the AKS Diagnostics Findings section (not in the main Detailed Analysis table). | ||
| 3. If a Critical detector finding contradicts a checklist item's `Meets` status, re-evaluate and downgrade the status with explanation. | ||
| 4. Detector findings supplement but do not replace validation command evidence for checklist items. | ||
| 5. Supplemental categories such as `Risk Alerts`, `Control Plane`, and `Keystone` can influence checklist status when they contain concrete detector findings. | ||
| 6. `Workflow Nodes`, `zWorkflowDetectors`, and category-overview detectors (e.g. `aks-category-*`) are supporting evidence and orchestration aids; they should not be scored as direct control checks unless a concrete detector result is also captured. | ||
| 7. `Analysis - L2/L3 Support Topic` detectors are scenario aggregators; map their findings back to the matching core category rather than treating them as new controls. | ||
|
|
||
| ## Interpretation Guidelines | ||
|
|
||
| - **Risk Alerts**: The AKS diagnostics API exposes a concrete `Risk Alerts` category that aggregates cross-category risk summaries (see Supplemental Live Categories above). This is distinct from the general concept of risk-level findings, which are any `Critical` or `Warning` results returned by detectors in any category. Capture the `Risk Alerts` category results in the Risk Alerts table; cross-reference individual Critical/Warning findings from other categories to their respective checklist items. | ||
| - **Troubleshooting Results**: The full set of detector outputs organized first by the eight core audit categories, then by any supplemental live categories. Even `Healthy` results provide positive evidence for the audit. | ||
| - **Detector Descriptions**: Each detector includes a description of what it checks. Record this in the evidence to explain what was assessed. | ||
| - **Time Sensitivity**: Detector results are point-in-time snapshots. Record the timestamp of execution in the report. | ||
| - **Supplemental Category Semantics**: A live category can represent a concrete detector bucket, a summary view, or a troubleshooting workflow. Preserve the returned detector type in the report so readers understand whether the item is scored evidence or supporting context. | ||
|
|
||
| ## Guardrails / Safety | ||
|
|
||
| - Do not skip execution of any detector. ALL detectors MUST be run. | ||
42 changes: 42 additions & 0 deletions
42
plugin/skills/azure-kubernetes/references/aks-review-container-assessment.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Container Assessment Workflow | ||
|
|
||
| ## Step 5: Run Container Best Practices Assessment | ||
|
|
||
| See [AKS Container Best Practices](./aks-container-best-practices.md) for complete checks, commands, and scoring. | ||
|
|
||
| ### Namespace Setup | ||
|
|
||
| Separate system namespaces (`kube-system`, `kube-node-lease`, `kube-public`, `gatekeeper-system`) from application namespaces for distinct reporting. | ||
|
|
||
| ### Check Categories | ||
|
|
||
| Execute all validation commands for: | ||
|
|
||
| | Category | Check IDs | Scope | | ||
| |----------|-----------|-------| | ||
| | Image Hygiene | `CTR-IMG-*` | Tag pinning, pull policies, private registries, ACR | | ||
| | Security Context | `CTR-SEC-*` | runAsNonRoot, readOnlyRootFilesystem, privileges, capabilities, seccomp | | ||
| | Resource Management | `CTR-RES-*` | CPU/memory requests/limits, QoS distribution, limit-to-request ratios | | ||
| | Health Probes & Lifecycle | `CTR-PROBE-*`, `CTR-LIFE-*` | Readiness/liveness/startup probes, preStop hooks, rolling updates | | ||
| | Pod Security | `CTR-POD-*` | Service account tokens, hostNetwork/PID/IPC, hostPath, PSS labels | | ||
| | Config Hygiene | `CTR-CFG-*` | Inline secrets, ConfigMap/Secret usage, naked pods, ports | | ||
| | Network | `CTR-NET-*` | NetworkPolicy coverage, default-deny, service types | | ||
| | Supply Chain | `CTR-ACR-*` | Vulnerability scanning, content trust, retention, webhooks | | ||
|
|
||
| ### Counting Methodology | ||
|
|
||
| Count every **running container instance** (not unique specs) — includes scaled replicas and injected sidecars. Separately report sidecar/proxy container counts. State methodology explicitly in report. | ||
|
|
||
| ### Sidecar-Aware Compliance | ||
|
|
||
| Where sidecars materially affect metrics (e.g., `runAsNonRoot`, readiness probes), report both sidecar-inclusive and application-only percentages. | ||
|
|
||
| ### Scoring | ||
|
|
||
| - Compute per-namespace and cluster-wide statistics | ||
| - Thresholds: ≥80% = Meets, 50-79% = Partially meets, <50% = Does not meet | ||
| - Cross-reference with checklist matrix overlap mapping | ||
|
|
||
| ### Fallback | ||
|
|
||
| If `kubectl` unavailable, use `az aks command invoke`. If also unavailable, mark `CTR-*` checks `Not assessed` except platform-level checks (`CTR-IMG-06`, `CTR-ACR-*`) assessable via `az` CLI. |
25 changes: 25 additions & 0 deletions
25
plugin/skills/azure-kubernetes/references/aks-review-diagnostics.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Diagnostics Workflow | ||
|
|
||
| ## Step 3: Run Diagnostics Detectors | ||
|
|
||
| See [AKS Diagnostics Detectors Reference](./aks-diagnostics-detectors.md) for extraction methods and interpretation rules. | ||
|
|
||
| ### Primary Method: AKS MCP Server | ||
|
|
||
| 1. Call `mcp_aks_mcp_list_detectors` to enumerate available detectors. | ||
| 2. **Core audit categories** (always query all 8): `Best Practices`, `Cluster and Control Plane Availability and Performance`, `Connectivity Issues`, `Create, Upgrade, Delete and Scale`, `Deprecations`, `Identity and Security`, `Node Health`, `Storage`. | ||
| 3. Call `mcp_aks_mcp_run_detectors_by_category` for each core category. | ||
| 4. Enumerate **supplemental categories** from live catalog (e.g., `Risk Alerts`, `Control Plane`, `Keystone`, `Workflow Nodes`). | ||
| 5. For supplemental categories: prefer category-level execution; if unsupported, run individual detectors via `mcp_aks_mcp_run_detector` and record why. | ||
| 6. For Critical/Warning findings needing deeper investigation, call `mcp_aks_mcp_run_detector` with specific detector name. | ||
|
johnbilliris marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### Fallback Method | ||
|
|
||
| If MCP unavailable, use `az rest` calls to AKS Diagnostics REST API (see detectors reference). | ||
|
|
||
| ### Post-Processing | ||
|
|
||
| - Record extraction timestamp (UTC and local time). | ||
| - Classify findings by severity: `Critical`, `Warning`, `Info`, `Healthy/None`. | ||
| - Cross-reference findings to checklist items using detectors reference mapping. | ||
| - If a Critical finding contradicts a checklist item's `Meets` status, downgrade and record justification. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.