Add: New azure-data-collection-rules skill for Azure Monitor DCR authoring (reopens #2575) - #3088
Closed
Ivan Varnitski (ivkhrul) wants to merge 3 commits into
Closed
Conversation
…oring Adds a skill that guides authoring, editing, and validating Azure Monitor Data Collection Rules (DCRs), including: - DCR JSON schema and structure guidance - KQL ingestion-time transformation patterns (7-stage pipeline) - Stream declarations and destination routing - Direct ingestion DCR authoring - Custom table creation - Client-side and ingestion-side processors - Token-optimized reference files (all under 2000-token limit) Covers single-stage and multi-stage transformation DCRs, Log Ingestion API direct ingestion, and custom log tables.
- Security: Remove AppSecret, promote CLI/Federated auth (RickWinter) - Fix PS 5.1 array-unrolling and multi-stage routing check - Add bash script equivalents for cross-platform support (tmeschter) - Add version.json + SKILL.md restructure for CI (jongio/tmeschter) - Remove [Parameter(Mandatory)] from all scripts - Add supported-tables.json reference data - Add Vally integration tests (evals/) with 9 stimuli (JasonYeMSFT) - Fix blank-line nits in kql-transforms.md, limits.md (JasonYeMSFT)
- Remove .gitignore (no Python/presentation files in upstream skill) - send-logs.sh: use mktemp + trap cleanup for temp file (security) - eval.yaml: remove duplicate YAML frontmatter, adopt standard format with name/description/environment - README.md: fix directory name (azure-data-collection-rules), fix file tree to match actual structure, add bash scripts, remove non-existent slash command - validate-dcr.sh: full parity with PowerShell version (limits, column validation, routing rules, transform cross-refs, Direct DCR name constraints)
Ivan Varnitski (ivkhrul)
requested review from
Rick Winter (RickWinter) and
Tom Meschter (tmeschter)
as code owners
August 18, 2026 17:13
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new azure-data-collection-rules skill under plugin/skills/ to guide authoring/validation/deployment of Azure Monitor Data Collection Rules (DCRs), including direct ingestion (Log Ingestion API) and multi-stage transformations, with bundled helper scripts, examples, and Vally eval coverage.
Changes:
- Introduces the
azure-data-collection-rulesskill entrypoint (SKILL.md), extensive reference docs, and example DCR JSON templates. - Adds cross-platform helper scripts (PowerShell + Bash) for validating, deploying, retrieving DCRs, managing Log Analytics tables, and sending logs.
- Adds Vally eval suite (
evals/azure-data-collection-rules/) to test routing and output correctness.
Reviewed changes
Copilot reviewed 40 out of 40 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| plugin/skills/azure-data-collection-rules/version.json | Skill-level versioning config (NBGV). |
| plugin/skills/azure-data-collection-rules/SKILL.md | Skill frontmatter + quick reference + procedure entrypoint. |
| plugin/skills/azure-data-collection-rules/README.md | Human-facing overview and repository layout for the skill. |
| plugin/skills/azure-data-collection-rules/scripts/validate-dcr.sh | Bash offline validator for DCR JSON structure/routing/limits. |
| plugin/skills/azure-data-collection-rules/scripts/validate-dcr.ps1 | PowerShell offline validator for DCR JSON structure/routing/limits. |
| plugin/skills/azure-data-collection-rules/scripts/send-logs.sh | Bash sender for Log Ingestion API sample payloads. |
| plugin/skills/azure-data-collection-rules/scripts/send-logs.ps1 | PowerShell sender for Log Ingestion API sample payloads. |
| plugin/skills/azure-data-collection-rules/scripts/put-dcr.sh | Bash deploy/update helper for DCRs using az rest. |
| plugin/skills/azure-data-collection-rules/scripts/put-dcr.ps1 | PowerShell deploy/update helper for DCRs using Invoke-AzRestMethod. |
| plugin/skills/azure-data-collection-rules/scripts/get-dcr.sh | Bash helper to retrieve and summarize an existing DCR. |
| plugin/skills/azure-data-collection-rules/scripts/get-dcr.ps1 | PowerShell helper to retrieve and summarize an existing DCR. |
| plugin/skills/azure-data-collection-rules/scripts/get-table-schema.sh | Bash helper to fetch Log Analytics table schema via az rest. |
| plugin/skills/azure-data-collection-rules/scripts/get-table-schema.ps1 | PowerShell helper to fetch Log Analytics table schema via Invoke-AzRestMethod. |
| plugin/skills/azure-data-collection-rules/scripts/create-custom-table.sh | Bash helper to create/update custom Log Analytics tables via az rest. |
| plugin/skills/azure-data-collection-rules/scripts/create-custom-table.ps1 | PowerShell helper to create/update custom Log Analytics tables via Invoke-AzRestMethod. |
| plugin/skills/azure-data-collection-rules/references/procedure.md | Step-by-step workflow for DCR design/authoring/validation/deployment. |
| plugin/skills/azure-data-collection-rules/references/dcr-kinds.md | Kind selection guidance and capability matrix. |
| plugin/skills/azure-data-collection-rules/references/dcr-schema.md | Schema reference (properties, constraints, context rules). |
| plugin/skills/azure-data-collection-rules/references/destination-routing.md | Stream-to-table routing rules and decision logic. |
| plugin/skills/azure-data-collection-rules/references/stream-declarations.md | Guidance for defining custom stream schemas. |
| plugin/skills/azure-data-collection-rules/references/processors-headers.md | Processor header reference and stage availability. |
| plugin/skills/azure-data-collection-rules/references/processors-operations.md | Processor operation reference (filter/map/parse/aggregate/enrich/KQL). |
| plugin/skills/azure-data-collection-rules/references/processor-heuristics-filters.md | Heuristics for choosing native filters vs processors. |
| plugin/skills/azure-data-collection-rules/references/processor-heuristics-transforms.md | Intent-to-transform mapping heuristics. |
| plugin/skills/azure-data-collection-rules/references/processor-heuristics-staging.md | Staging/cost optimization guidance for multi-stage pipelines. |
| plugin/skills/azure-data-collection-rules/references/kql-transforms.md | Ingestion-time KQL transform patterns and constraints. |
| plugin/skills/azure-data-collection-rules/references/la-tables.md | Reference for standard/custom Log Analytics tables and plans. |
| plugin/skills/azure-data-collection-rules/references/supported-tables.md | Doc list of standard tables that accept custom streams. |
| plugin/skills/azure-data-collection-rules/references/supported-tables.json | Machine-readable list used by validators for routing checks. |
| plugin/skills/azure-data-collection-rules/references/direct-ingestion.md | Direct ingestion (Logs Ingestion API) guidance, auth, and structure. |
| plugin/skills/azure-data-collection-rules/references/decision-guide.md | Scenario-to-approach routing cheat-sheet. |
| plugin/skills/azure-data-collection-rules/references/limits.md | Consolidated service limits/constraints used by guidance/validators. |
| plugin/skills/azure-data-collection-rules/examples/syslog-filter-drop.json | Example DCR: syslog with native filtering + client-side column drop. |
| plugin/skills/azure-data-collection-rules/examples/custom-json-log.json | Example DCR: logFiles JSON parsing + ingestion KQL. |
| plugin/skills/azure-data-collection-rules/examples/perf-counter-aggregation.json | Example DCR: perf counters with client-side aggregation. |
| plugin/skills/azure-data-collection-rules/examples/windows-events-split.json | Example DCR: Windows events with split routing. |
| plugin/skills/azure-data-collection-rules/examples/direct-ingestion-custom-table.json | Example DCR: direct ingestion payload mapped to a custom table. |
| evals/azure-data-collection-rules/README.md | Vally eval suite documentation and coverage map. |
| evals/azure-data-collection-rules/eval.yaml | Vally eval definitions (routing + correctness graders). |
| evals/azure-data-collection-rules/fixtures/sample-logs.json | Sample payload fixture used by evals/context. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+33
to
+43
| # Validate JSON array | ||
| if ! jq empty "$DATA_FILE" 2>/dev/null; then | ||
| echo "ERROR: Invalid JSON in $DATA_FILE" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| RECORD_COUNT=$(jq 'length' "$DATA_FILE") | ||
| if [ "$RECORD_COUNT" -eq 0 ]; then | ||
| echo "ERROR: Data file contains no records" >&2 | ||
| exit 1 | ||
| fi |
Comment on lines
+42
to
+55
| $data = Get-Content -Path $DataFilePath -Raw | ||
|
|
||
| # Validate JSON array (wrap in @() to handle PS 5.1 single-element unrolling) | ||
| try { | ||
| $parsed = $data | ConvertFrom-Json -ErrorAction Stop | ||
| $parsed = @($parsed) | ||
| if ($parsed.Count -eq 0) { | ||
| Write-Error "Data file contains no records" | ||
| exit 1 | ||
| } | ||
| } catch { | ||
| Write-Error "Invalid JSON: $_" | ||
| exit 1 | ||
| } |
Comment on lines
+50
to
+54
| # Collect declared custom streams | ||
| $declaredStreams = @() | ||
| if ($props.streamDeclarations) { | ||
| $declaredStreams = $props.streamDeclarations.PSObject.Properties.Name | ||
| } |
Comment on lines
+47
to
+66
| # Check required sections | ||
| HAS_DATASOURCES=$(jq "${PROPS}.dataSources != null" "$DcrFilePath") | ||
| HAS_DESTINATIONS=$(jq "${PROPS}.destinations != null" "$DcrFilePath") | ||
| HAS_DATAFLOWS=$(jq "${PROPS}.dataFlows != null" "$DcrFilePath") | ||
|
|
||
| if [ "$IS_DIRECT" = false ] && [ "$HAS_DATASOURCES" = "false" ]; then | ||
| ERRORS+=("Missing 'dataSources' section (required for non-Direct DCRs)") | ||
| fi | ||
| if [ "$IS_DIRECT" = true ] && [ "$HAS_DATASOURCES" = "true" ]; then | ||
| WARNINGS+=("Direct DCR should not have a 'dataSources' section") | ||
| fi | ||
| if [ "$HAS_DESTINATIONS" = "false" ]; then | ||
| ERRORS+=("Missing 'destinations' section") | ||
| fi | ||
| if [ "$HAS_DATAFLOWS" = "false" ]; then | ||
| ERRORS+=("Missing 'dataFlows' section") | ||
| fi | ||
|
|
||
| # Collect declared custom streams | ||
| DECLARED_STREAMS=$(jq -r "${PROPS}.streamDeclarations // {} | keys[]" "$DcrFilePath" 2>/dev/null) |
Comment on lines
+35
to
+39
| # Validate JSON | ||
| if ! jq empty "$DCR_FILE" 2>/dev/null; then | ||
| echo "ERROR: Invalid JSON in $DCR_FILE" >&2 | ||
| exit 1 | ||
| fi |
Comment on lines
+42
to
+44
| # Write full JSON to file | ||
| echo "$RESPONSE" | jq '.' > "$OUTPUT_PATH" | ||
|
|
Comment on lines
+39
to
+44
| echo "Table: $(echo "$RESPONSE" | jq -r '.properties.schema.name')" | ||
| echo "Plan: $(echo "$RESPONSE" | jq -r '.properties.plan')" | ||
| echo "Retention: $(echo "$RESPONSE" | jq -r '.properties.retentionInDays') days" | ||
| echo "" | ||
| echo "Columns:" | ||
| echo "$RESPONSE" | jq -r '.properties.schema.columns[] | " \(.name) (\(.type))"' |
Comment on lines
+47
to
+52
| # Verify TimeGenerated column exists | ||
| HAS_TG=$(jq '[.columns[] | select(.name == "TimeGenerated" and .type == "datetime")] | length' "$SCHEMA_FILE") | ||
| if [ "$HAS_TG" -eq 0 ]; then | ||
| echo "ERROR: Schema must include a 'TimeGenerated' column of type 'datetime'" >&2 | ||
| exit 1 | ||
| fi |
Comment on lines
+42
to
+45
| | `mcp_azure_mcp_ser_monitor` | Query Azure Monitor resources, DCRs, and diagnostics | | ||
| | `mcp_azure_mcp_ser_subscription_list` | List available subscriptions | | ||
| | `mcp_azure_mcp_ser_group_list` | List resource groups in subscription | | ||
| | `mcp_azure_mcp_ser_group_resource_list` | List resources in a resource group | |
Contributor
|
This PR has been automatically closed because it has unresolved review comments and no new commits in the last 28 days. If you would still like to merge these changes, please:
Thank you for your contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reopens #2575, which was auto-closed by the stale bot on 2026-08-04 (unresolved review threads + no commits in 28 days). All change-request feedback was addressed in the 2026-06-26 force-push with follow-ups on 7/2 and 7/10; this PR carries the same branch forward so the outstanding threads can be resolved and merged.
This PR adds a new
azure-data-collection-rulesskill for Azure Monitor DCR authoring.Summary of feedback addressed (from #2575)
AppSecretparameter entirely (auth now usesaz account get-access-token/ Managed Identity), fixed PS 5.1 array unrolling with@($parsed), multi-stage routing check accepts a namedtransformin addition to inlinetransformKql.evals/azure-data-collection-rules/(9 stimuli: 4 positive routing, 2 negative routing, 3 correctness).version.json+0.0.0-placeholderversion, removed[Parameter(Mandatory)]from all scripts, compactedget-dcr.ps1output.All changes were regression-tested against a live Azure DCR (validation, deployment, and log ingestion passing).
/cc Rick Winter (@RickWinter) JasonYeMSFT (@JasonYeMSFT) Tom Meschter (@tmeschter) Jon Gallant (@jongio)