Skip to content

Commit c3c909c

Browse files
adarshdigievoadarshdigievo
committed
CI: add a cron test for latest version compatibility checks
1 parent 3536018 commit c3c909c

3 files changed

Lines changed: 156 additions & 0 deletions

File tree

‎.github/workflows/latest-sdk-compatibility.yml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
name: Latest AI SDK compatibility
2+
3+
on:
4+
schedule:
5+
- cron: "23 5 * * 1"
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
10+
11+
concurrency:
12+
group: ${{ github.workflow }}-${{ github.ref }}
13+
cancel-in-progress: true
14+
15+
jobs:
16+
latest-sdk:
17+
name: ${{ matrix.provider }} newest stable release
18+
runs-on: ubuntu-latest
19+
timeout-minutes: 20
20+
strategy:
21+
fail-fast: false
22+
matrix:
23+
include:
24+
- provider: agno
25+
distribution: agno
26+
- provider: autogen
27+
distribution: autogen-agentchat
28+
- provider: claude-agent-sdk
29+
distribution: claude-agent-sdk
30+
- provider: crewai
31+
distribution: crewai
32+
- provider: google-adk
33+
distribution: google-adk
34+
- provider: haystack
35+
distribution: haystack-ai
36+
- provider: langchain
37+
distribution: langchain
38+
- provider: langgraph
39+
distribution: langgraph
40+
- provider: llamaindex
41+
distribution: llama-index-core
42+
- provider: microsoft-agent-framework
43+
distribution: agent-framework-openai
44+
- provider: openai-agents
45+
distribution: openai-agents
46+
- provider: pydantic-ai
47+
distribution: pydantic-ai
48+
- provider: semantic-kernel
49+
distribution: semantic-kernel
50+
- provider: smolagents
51+
distribution: smolagents
52+
53+
steps:
54+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v4
55+
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
56+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
57+
with:
58+
python-version: "3.13"
59+
60+
- run: uv sync --dev --locked
61+
- name: Create the isolated SDK environment
62+
run: uv run tox -r -n -e latest-${{ matrix.provider }}
63+
- name: Install the newest PyPI release
64+
env:
65+
SDK_DISTRIBUTION: ${{ matrix.distribution }}
66+
TOX_ENVIRONMENT: latest-${{ matrix.provider }}
67+
run: |
68+
latest_version="$(
69+
curl --fail --silent --show-error \
70+
"https://pypi.org/pypi/$SDK_DISTRIBUTION/json" \
71+
| ".tox/$TOX_ENVIRONMENT/bin/python" -c \
72+
'import json, sys; print(json.load(sys.stdin)["info"]["version"])'
73+
)"
74+
uv pip install --upgrade \
75+
--python ".tox/$TOX_ENVIRONMENT/bin/python" \
76+
"$SDK_DISTRIBUTION==$latest_version"
77+
installed_version="$(
78+
".tox/$TOX_ENVIRONMENT/bin/python" -c \
79+
'import importlib.metadata as m, sys; print(m.version(sys.argv[1]))' \
80+
"$SDK_DISTRIBUTION"
81+
)"
82+
echo "$SDK_DISTRIBUTION: $installed_version"
83+
test "$installed_version" = "$latest_version"
84+
- name: Test the newest stable SDK release
85+
run: >-
86+
uv run tox --skip-env-install
87+
-e latest-${{ matrix.provider }}
88+
- name: Report resolved package versions
89+
if: always()
90+
env:
91+
TOX_ENVIRONMENT: latest-${{ matrix.provider }}
92+
run: |
93+
if test -x ".tox/$TOX_ENVIRONMENT/bin/python"; then
94+
uv pip freeze --python ".tox/$TOX_ENVIRONMENT/bin/python"
95+
else
96+
echo "tox environment was not created"
97+
fi

‎docs/development/testing.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,18 @@ CI runs the base suite on Python 3.10 through 3.14 and installs each framework
4545
extra separately. This catches dependency conflicts that a shared development
4646
environment can hide.
4747

48+
## Newest SDK release canary
49+
50+
The regular integration environments install SDK versions from the supported ranges declared in `pyproject.toml`. The `latest-*` environments install the base wheel without an SDK extra or SDK version constraint. The scheduled workflow then asks PyPI for the newest stable release and installs that exact version, even when it is outside the package's current upper bound. A failure is an early warning to add support; it does not expand the package's declared support range.
51+
52+
GitHub Actions runs this canary every Monday at 05:23 UTC and on manual dispatch. Each matrix job prints its resolved package versions after the test. PyPI pre-releases are excluded unless they become part of pip's normal stable resolution.
53+
54+
Run one canary locally:
55+
56+
```bash
57+
uv run tox -r -e latest-openai-agents
58+
```
59+
4860
## Fake OpenAI-compatible integration tests
4961

5062
These tests start a local FastAPI server. They exercise real framework agent

‎tox.ini‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,53 @@ set_env =
8888
SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = langgraph
8989
commands = {posargs:python -m pytest -q -m integration tests/integration_agents}
9090

91+
[testenv:latest-{agno,autogen,claude-agent-sdk,crewai,google-adk,haystack,langchain,langgraph,llamaindex,microsoft-agent-framework,openai-agents,pydantic-ai,semantic-kernel,smolagents}]
92+
base_python = 3.13
93+
deps =
94+
agno: agno
95+
agno: openai
96+
autogen: autogen-agentchat
97+
autogen: autogen-core
98+
autogen: autogen-ext[openai]
99+
claude-agent-sdk: claude-agent-sdk
100+
crewai: crewai
101+
google-adk: google-adk
102+
google-adk: litellm
103+
google-adk: orjson
104+
haystack: haystack-ai
105+
langchain: langchain
106+
langchain: langchain-core
107+
langchain: langchain-openai
108+
langgraph: langchain
109+
langgraph: langchain-core
110+
langgraph: langchain-openai
111+
langgraph: langgraph
112+
llamaindex: llama-index-core
113+
llamaindex: llama-index-llms-openai
114+
microsoft-agent-framework: agent-framework-openai
115+
openai-agents: openai-agents
116+
pydantic-ai: pydantic-ai
117+
semantic-kernel: openai
118+
semantic-kernel: semantic-kernel
119+
smolagents: openai
120+
smolagents: smolagents
121+
set_env =
122+
agno: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = agno
123+
autogen: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = autogen
124+
claude-agent-sdk: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = claude-agent-sdk
125+
crewai: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = crewai
126+
google-adk: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = google-adk
127+
haystack: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = haystack
128+
langchain: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = langchain
129+
langgraph: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = langgraph
130+
llamaindex: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = llamaindex
131+
microsoft-agent-framework: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = microsoft-agent-framework
132+
openai-agents: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = openai-agents
133+
pydantic-ai: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = pydantic-ai
134+
semantic-kernel: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = semantic-kernel
135+
smolagents: SERPAPI_SEARCH_TOOL_REQUIRED_PROVIDER = smolagents
136+
commands = {posargs:python -m pytest -q -m integration tests/integration_agents}
137+
91138
[testenv:openai-compat]
92139
base_python = 3.13
93140
extras = frameworks

0 commit comments

Comments
 (0)