feature: add discover-azure-skills skill - #3053
feature: add discover-azure-skills skill#3053JasonYeMSFT (JasonYeMSFT) wants to merge 7 commits into
Conversation
Example UX in Copilot CLI. |
There was a problem hiding this comment.
Pull request overview
Adds a new discover-azure-skills agent skill intended to help users discover relevant Azure-related skills/plugins and includes test registration plus a routing eval to validate invocation behavior.
Changes:
- Registers
discover-azure-skillsintests/skills.jsonand adds it to the integration schedule. - Adds a new skill package under
plugins/azure-skills/skills/discover-azure-skillswith install reference docs. - Adds a new Vally integration eval suite for routing/invocation coverage.
Show a summary per file
| File | Description |
|---|---|
tests/skills.json |
Adds the new skill to the azure plugin skill list and integration schedule. |
plugins/azure-skills/skills/discover-azure-skills/version.json |
Introduces per-skill NBGV version configuration. |
plugins/azure-skills/skills/discover-azure-skills/SKILL.md |
Defines the discovery workflow and links to install instructions. |
plugins/azure-skills/skills/discover-azure-skills/references/install/other.md |
Provides generic install instructions via npx skills add. |
plugins/azure-skills/skills/discover-azure-skills/references/install/copilot-cli.md |
Provides Copilot CLI installation steps. |
plugins/azure-skills/skills/discover-azure-skills/references/install/claude-code.md |
Provides Claude Code installation steps. |
evals/azure-skills/discover-azure-skills/eval.yaml |
Adds an integration routing eval validating skill invocation across prompts. |
Review details
Tip
Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 7/7 changed files
- Comments generated: 8
- Review effort level: Lite
| 1. Install each skill using `skills` package | ||
|
|
||
| ``` | ||
| npx skills add https://github.com/microsoft/azure-skills/tree/main/.github/plugins/{plugin-dirname}/skills/{skill-name} |
There was a problem hiding this comment.
Notes for reviewers:
- How do you feel about requiring npx? I haven't found any other well-known tool that handles skill installation as well as skills.sh while being more multi-platform friendly.
- VS Code is currently categorized under "other" because its built-in mechanism doesn't support specifying which plugin to install from a repo. Allow users to specify which plugin to install in "Install Plugin from Source" command vscode#330047
- Skills installed in this way won't send telemetry since hooks won't be installed.
| @@ -0,0 +1,42 @@ | |||
| --- | |||
| name: discover-azure-skills | |||
| description: "Searches the Azure skills catalog and recommends installable agent skills by matching an Azure task to skill metadata and plugin installation guidance. WHEN: \"search the Azure skills catalog\", \"find an Azure skill for this task\", \"recommend Azure skills not currently installed\", \"which plugin contains this Azure skill\"." | |||
There was a problem hiding this comment.
Does the skill only get triggered when the user shares an intent to discover a skill?
There was a problem hiding this comment.
Would it be possible to update the description to clarify that this skill serves as a fallback for Azure tasks that are not covered by any enabled skills? It should also explicitly instruct the model not to invoke this skill when an existing enabled skill already supports the user's requested task.
There was a problem hiding this comment.
Does the skill only get triggered when the user shares an intent to discover a skill?
This is what I plan to do. In my opinion, making this skill to proactively recommend skills is too aggressive and will also run into scalability issues. To recommend a skill from everything we offer without knowing what the user may ask, we will need to load everything into the context. It's not possible load everything into the context as we add more plugins/skills.
| pattern: "(?i)fatal error|unhandled exception|stack trace" | ||
|
|
||
| - name: "Find a skill to generate Azure Kusto graph" | ||
| prompt: "Find an Azure agent skill that can generate an Azure Kusto graph." |
There was a problem hiding this comment.
If we update this skill so that it's invoked based on the user's scenario rather than the intent to discover a skill, I think this evaluation will need to be updated as well.
There was a problem hiding this comment.
I rewrote the test prompts to cover these cases:
- Ask agent to search a skill not in the core plugin
- Ask agent to find a skill in the core plugin
- Ask a generic how-to question
- Give a direct command asking the agent to do something
|
Does this skill require me to ask for discovery intentionally? Would the customer need to do "discover" explicitly? Should the skill be more of a guide that happens based on intent to do something Azure? |
I expect the user to at least express the intent to learn how to do something in Azure to trigger this skill. I intend to avoid having the skill triggered when it's not the case. We can discuss more details of the design in this issue. https://github.com/microsoft/GitHub-Copilot-for-Azure-pr/issues/358 |
c8fcb25 to
0e1c899
Compare
Description
A new skill to help discover available skills among all the plugins we own.
Checklist
cd tests && npm test)fix:,feat:,feature:,chore:,misc:,test:,eval:tests/,npm run test:integration -- <skill>ornpm run test:vally -- --skill <skill>)Related Issues
resolves #2794