feat(codemode): replace Yaegi with Expr

.github/workflows/go.yml

@@ -38,7 +38,6 @@ jobs:
         run: |
           go test -v -coverprofile=coverage.out -covermode=atomic $(go list ./... | grep -v '/examples') || true
 
-
       - name: Show coverage summary
         run: go tool cover -func=coverage.out
 

go.mod

@@ -3,6 +3,7 @@ module github.com/universal-tool-calling-protocol/go-utcp
 go 1.25.0
 
 require (
+	github.com/expr-lang/expr v1.17.8
 	github.com/google/uuid v1.6.0
 	github.com/gorilla/websocket v1.5.3
 	github.com/joho/godotenv v1.5.1
@@ -13,7 +14,6 @@ require (
 	github.com/pion/webrtc/v3 v3.3.5
 	github.com/spf13/cast v1.7.1
 	github.com/stretchr/testify v1.10.0
-	github.com/traefik/yaegi v0.16.1
 	google.golang.org/grpc v1.75.1
 	google.golang.org/protobuf v1.36.10
 	gopkg.in/yaml.v3 v3.0.1

go.sum

@@ -1,6 +1,8 @@
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/expr-lang/expr v1.17.8 h1:W1loDTT+0PQf5YteHSTpju2qfUfNoBt4yw9+wOEU9VM=
+github.com/expr-lang/expr v1.17.8/go.mod h1:8/vRC7+7HBzESEqt5kKpYXxrxkr31SaO8r40VO/1IT4=
 github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
 github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
 github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
@@ -104,8 +106,6 @@ github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXl
 github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
 github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
 github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
-github.com/traefik/yaegi v0.16.1 h1:f1De3DVJqIDKmnasUF6MwmWv1dSEEat0wcpXhD2On3E=
-github.com/traefik/yaegi v0.16.1/go.mod h1:4eVhbPb3LnD2VigQjhYbEJ69vDRFdT2HQNrXx8eEwUY=
 github.com/wlynxg/anet v0.0.3 h1:PvR53psxFXstc12jelG6f1Lv4MWqE0tI76/hHGjh9rg=
 github.com/wlynxg/anet v0.0.3/go.mod h1:eay5PRQr7fIVAMbTbchTnO9gG65Hg/uYGdc7mguHxoA=
 github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=

src/plugins/codemode/OPTIMIZATION_SUMMARY.md

@@ -1,95 +1,36 @@
-# CodeMode Optimization Summary
+# CodeMode Runtime Migration Summary
 
-> Historical note: the measurements below describe the original minimal-stdlib optimization. For current benchmarks and the latest orchestration changes, see [PERFORMANCE.md](PERFORMANCE.md).
+The CodeMode runtime on this branch replaces Yaegi with Expr.
 
-## ✅ Completed Optimizations
+## Runtime changes
 
-### 1. Performance Optimization (40% faster)
+- Removed the Yaegi interpreter and standard-library injection path.
+- Added `github.com/expr-lang/expr` v1.17.8.
+- Generated programs are now Expr expressions rather than Go-like snippets.
+- Tool access is exposed through a single `codemode` API object.
+- Tool errors propagate through Expr's normal `(value, error)` function semantics.
+- Streaming tools are collected into an Expr array by `codemode.CallToolStream`.
+- Tool-result map fields can be extracted with `codemode.Get`.
+- The existing tool whitelist and orchestration cache remain unchanged.
 
-**Problem**: CodeMode was loading the entire Go stdlib (~50+ packages) on every execution, taking ~1.5ms and allocating ~2MB.
+## Expr safety model
 
-**Solution**: Load only the 3 packages actually needed (context, fmt, reflect) and cache them with `sync.Once`.
+Expr provides parsing, static checking, optimization, and bytecode execution without exposing arbitrary Go imports or Yaegi's standard library. CodeMode therefore has a smaller execution surface than the previous interpreter-based implementation.
 
-**Results**:
-- **Execution time**: 1.63ms → 0.98ms (40% faster)
-- **Memory usage**: 2.13MB → 1.34MB (37% less)
-- **Throughput**: 723 ops/sec → 1186 ops/sec (64% increase)
+Expr programs also do not expose arbitrary infinite loops. Remote UTCP calls still receive a cancellable context and CodeMode maintains an overall execution timeout.
 
-### 2. Walrus Operator Fix
+## Workflow shape
 
-**Problem**: The `convertOutWalrus` function was incorrectly converting `:=` to `=` in all contexts, breaking multi-variable declarations and causing compilation errors like `expected ';', found ':='`.
+A generated multi-tool workflow is now compact:
 
-**Solution**: Updated the regex to only convert `__out :=` when `__out` is the sole variable being declared at the start of a line, preserving multi-variable declarations like `__out, err := ...`.
-
-**Implementation**:
-```go
-func convertOutWalrus(code string) string {
-	// Only convert `__out :=` when __out is the sole variable being declared
-	// This avoids breaking multi-variable declarations like `result, err := ...`
-	// Use capture groups to preserve both leading whitespace and spacing before :=
-	re := regexp.MustCompile(`(?m)^(\s*)__out(\s*):=`)
-	return re.ReplaceAllString(code, "${1}__out${2}=")
-}
-```
-
-**Test Coverage**: Added comprehensive tests in `preprocess_test.go` to verify:
-- ✅ Simple `__out :=` conversion
-- ✅ Preserves leading whitespace
-- ✅ Does NOT convert multi-variable declarations (`__out, err := ...`)
-- ✅ Works in nested blocks
-- ✅ Leaves other variable declarations untouched
-
-## 📊 Final Benchmark Results
-
-```
-BenchmarkExecuteSimple-8   1186   981,468 ns/op   1,336,425 B/op   15,226 allocs/op
-```
-
-Compared to original:
-```
-BenchmarkExecuteSimple-8    723  1,632,590 ns/op   2,126,798 B/op   16,162 allocs/op
+```expr
+let first = codemode.CallTool("math.add", {"a": 2, "b": 3});
+let value = codemode.Get(first, "result");
+codemode.CallTool("math.multiply", {"value": value, "factor": 10})
 ```
 
-## 🧪 Testing
-
-All tests pass:
-- ✅ All existing codemode tests
-- ✅ New walrus operator conversion tests  
-- ✅ Preprocessing tests
-- ✅ Orchestrator tests
-
-## 📝 Files Modified
-
-1. `/src/plugins/codemode/codemode.go`
-   - Added `getMinimalStdlib()` function with caching
-   - Updated `injectHelpers()` to use minimal stdlib
-   - Fixed `convertOutWalrus()` regex
-
-2. `/src/plugins/codemode/benchmark_test.go` (new)
-   - Added benchmarks for interpreter initialization
-   - Added benchmarks for stdlib loading
-   - Added benchmarks for code execution
-
-3. `/src/plugins/codemode/preprocess_test.go` (new)
-   - Added tests for `convertOutWalrus()`
-   - Added tests for `preprocessUserCode()`
-
-4. `/src/plugins/codemode/PERFORMANCE.md` (new)
-   - Documented performance improvements
-   - Included benchmark comparisons
-   - Outlined future optimization opportunities
-
-## 🎯 Impact
-
-- **Faster agent operations**: 40% reduction in code execution time
-- **Better scalability**: 64% higher throughput
-- **Lower resource usage**: 37% less memory per execution
-- **More reliable**: Fixed walrus operator bug that was causing compilation errors
-- **Better tested**: Added comprehensive test coverage for preprocessing logic
+The last expression is the workflow result, eliminating the old `__out` plumbing.
 
-## 🔮 Future Optimization Opportunities
+## Benchmark follow-up
 
-1. **Interpreter Pooling**: Reuse interpreters instead of creating new ones (~5ms saved per execution)
-2. **AST Caching**: Cache parsed/wrapped programs for repeated code snippets
-3. **Lazy Helper Injection**: Only inject UTCP helpers when tools are actually called
-4. **Parallel Compilation**: Compile user code in parallel with helper injection
+The old benchmark suite measured Yaegi interpreter initialization and helper injection and was removed. New benchmarks should measure Expr compilation, warm execution, chained tool execution, streaming collection, and cached orchestration independently.

src/plugins/codemode/PERFORMANCE.md

@@ -1,48 +1,24 @@
-# CodeMode Performance Report
+# CodeMode Performance Notes
 
-- **Date:** 2026-07-14
-- **Baseline:** `ae717a6d715b` (`fix codemode/orchestrator isValidSnippet`)
-- **Current:** `78f597d45301` (`perf: optimize client repository and transports`)
-- **Environment:** Apple M2, Go 1.26.3, Darwin/arm64, `-count=3`
+CodeMode now executes generated workflows with Expr's parser, type checker, optimizer, and bytecode VM instead of Yaegi.
 
-## Summary
+The old benchmark numbers in this file measured Yaegi initialization and helper injection and are therefore not comparable to the new runtime. They were intentionally removed rather than presenting stale measurements as Expr results.
 
-CodeMode execution is substantially faster after the optimization work:
+## What to benchmark
 
-- Simple snippets are **23.9x faster** and use **95.8% less memory**.
-- A representative two-tool chain is **8.1x faster** and uses **92.0% less memory**.
-- The cached orchestration path is **3.4x faster** with **one** model round trip instead of two after selection caching.
+The useful Expr benchmarks are:
 
-## Benchmark results
+- expression compilation
+- expression execution with a warm compiled program
+- synchronous two-tool chaining
+- streaming result collection
+- cached tool catalog lookup
+- planner/model round trips
 
-Values are medians of three benchmark processes. `B/op` and `allocs/op` are per operation.
+Provider and network latency should be measured separately because UTCP tool calls can dominate end-to-end workflow latency.
 
-| Path | Before | After | Change |
-| --- | ---: | ---: | ---: |
-| Helper injection | 910.8 us, 1.25 MB, 14,030 allocs | 6.2 us, 18.6 KB, 102 allocs | **146.8x faster**, 98.5% less memory |
-| Simple execution | 1.026 ms, 1.40 MB, 15,328 allocs | 42.9 us, 59.1 KB, 747 allocs | **23.9x faster**, 95.8% less memory |
-| Chained execution (two tools) | 1.108 ms, 1.46 MB, 16,143 allocs | 136.1 us, 116.9 KB, 1,562 allocs | **8.1x faster**, 92.0% less memory |
-| Cached tool-spec lookup | 3.38 us, 20.0 KB, 1 alloc | 3.79 us, 20.0 KB, 1 alloc | 12.3% slower |
-| Cached orchestration with mock model | 30.1 us, 19.7 KB, 106 allocs | 8.8 us, 12.3 KB, 69 allocs | **3.4x faster**, 37.4% less memory |
+## Runtime characteristics
 
-The small cached lookup regression is about 0.4 us and is outweighed by the execution and orchestration improvements.
+Expr is designed as a safe expression evaluator with static checking and a bytecode VM. It does not expose arbitrary Go imports or Yaegi's standard library. Expr also does not provide arbitrary infinite loops, so the old interpreter-loop timeout benchmark is no longer representative.
 
-## What was measured
-
-- **Simple execution** runs an arithmetic CodeMode snippet through Yaegi.
-- **Chained execution** calls local mock `math.add` and `math.multiply` tools in sequence, measuring interpreter, helper, and chaining overhead without network latency.
-- **Cached orchestration** uses cached tool metadata and an in-process mock model. The baseline performed a decision plus code-generation model call after its selection cache was warm; the current path plans and generates in one call.
-
-The orchestration benchmark intentionally excludes provider and network latency. Real model-backed requests should save the latency of one model round trip; remote tool-call latency will still dominate the chained-execution total when tools are network-bound.
-
-## Reproducing the current measurements
-
-Run the focused suite from the repository root:
-
-```sh
-go test ./src/plugins/codemode -run '^$' \
-  -bench '^(BenchmarkInjectHelpers|BenchmarkExecuteSimple|BenchmarkExecuteChainedToolCalls|BenchmarkToolSpecs_WithCache_Hit|BenchmarkPlanAndGenerate_OneRoundTrip|BenchmarkCallTool_OneRoundTrip)$' \
-  -benchmem -count=3
-```
-
-To reproduce the before values, run the matching benchmarks from a clean checkout of `ae717a6d715b`. `BenchmarkExecuteChainedToolCalls` was added with the optimization work; use the equivalent two-tool benchmark shown above when testing the baseline.
+Remote tool calls remain cancellable through the CodeMode execution context. CodeMode still applies an overall timeout around expression execution.