Skip to content

feat: add BreezeBlue TTS extension - #2379

Open
TiagoAgora wants to merge 3 commits into
TEN-framework:mainfrom
TiagoAgora:feat/breezeblue-tts
Open

TiagoAgora wants to merge 3 commits into
TEN-framework:mainfrom
TiagoAgora:feat/breezeblue-tts

Conversation

@TiagoAgora

@TiagoAgora TiagoAgora commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds breezeblue_tts_python, a TTS extension for BreezeBlue built on AsyncTTS2HttpExtension. It uses the HTTP streaming endpoint (POST /v1/text-to-speech/{voice_id}/stream).

  • Streams raw PCM16, 24 kHz, mono straight through. No conversion is needed. A trailing odd byte is held back so a sample is never split across chunks.
  • Sends only whitelisted request fields. The API rejects unknown fields with a 422 (e.g. the playground's generic sample_rate).
  • Splits text over the 1000-character per-request cap at sentence boundaries. The base class does not chunk, so a long LLM chunk would otherwise be rejected. max_text_length is configurable for accounts with a higher cap.
  • Retries 502/503/504 before any audio is sent. Maps 401/403, and a 422 caused by a malformed key, to an invalid-key error. Rejects a 200 response that is not audio.
  • cancel() closes the in-flight response and the stream ends with a flush.
  • Supports voice_id, model_id, language_code, instructions, speed, volume, guidance_scale and use_speaker_boost.

Other changes:

  • .env.example: BREEZE_API_KEY and BREEZE_TTS_VOICE_ID.
  • TTS guarder tests/bin/start: add the extension to the fixed-sample-rate list.
  • docs/ai: gotchas for strict request schemas, unchunked text and PCM16 passthrough.

Not in this PR

  • Word-level timestamps / subtitle alignment.
  • The realtime WebSocket transport (lower latency, more lifecycle code). A possible follow-up if latency matters.

Test plan

  • Extension-local tests: 64 pass (config, client, and adapter-level events: audio-end accounting, errors, cancellation, metrics, state machine). A few deliberate client bugs were confirmed to fail the matching tests.

  • Real API: voice listing, English, Chinese, the multilingual model, speed/volume, long text (split into two requests, one tts_audio_end).

  • Full TTS guarder, default config, against the live API at commit cc6fc4d4e:

    task tts-guarder-test EXTENSION=breezeblue_tts_python
    52 passed, 19 skipped in 473.86s
    

    The skips are capability-based (connection status is WebSocket-only, no subtitle alignment, and checks for other vendors' hooks).

A guarder screenshot and the downloadable PCM output (24000 Hz, 16-bit signed little-endian, mono) will be attached in a comment on this PR.

Add breezeblue_tts_python (AsyncTTS2HttpExtension) for BreezeBlue's HTTP
streaming API (POST /v1/text-to-speech/{voice_id}/stream).

- stream raw PCM16 24 kHz mono straight through, holding back a trailing odd
  byte so a sample is never split across chunks
- send only whitelisted request fields: the API rejects unknown ones with a
  422, e.g. the playground's generic sample_rate
- split text over the 1000 char per-request cap at sentence boundaries, since
  the base class does not chunk
- retry 502/503/504 before any audio; map 401/403 and malformed-key 422 to an
  invalid-key error; reject non-audio 200 responses
- cancel() closes the in-flight response and ends the stream with a flush
- extension-local tests for config, client and adapter events (audio-end
  accounting, errors, cancellation, metrics, state machine)
- document BREEZE_API_KEY and BREEZE_TTS_VOICE_ID, add the extension to the
  guarder fixed-sample-rate list, and record the gotchas in docs/ai
@github-actions

Copy link
Copy Markdown

[P1] Redact authentication headers before logging the configuration

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/config.py:120–124

Header-based authentication is explicitly supported by validate() and tested, but to_str(sensitive_handling=True) only masks params.api_key. Supplying headers={"xi-api-key": "brz_..."} or an Authorization header leaves the credential verbatim in the returned string, which the inherited HTTP extension logs at INFO during every initialization. Redact authentication headers case-insensitively before returning the config string, and add a regression test for header-only authentication.

[P1] Attach the required guarder evidence before merging the new extension

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/manifest.json:1–4

This extension is absent from the merge base, so the new-extension guarder evidence gate applies. The PR body reports 52 passed / 19 skipped at the current HEAD, but no owner/workflow evidence comment supplies the required result screenshot or a downloadable PCM output. This is a merge blocker under .agents/skills/ten-tts-review/references/guarder.md. Add an owner/workflow comment identifying the tested HEAD, exact command, counts and skip rationale, with the successful-run screenshot and a non-empty guarder-produced PCM download identified as 24 kHz, PCM16 little-endian, mono.

[P2] Make cancellation interrupt requests awaiting response headers

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/breezeblue_tts.py:135–138

If a flush arrives while connecting or waiting for response headers, _response is still None: it is assigned only after client.stream().__aenter__() returns. Consequently cancel() only sets a flag and leaves that HTTP operation running; the flag cannot be checked until the headers arrive. The inherited serial input loop remains occupied by the cancelled request, delaying subsequent speech until the vendor responds or the 60-second timeout expires. Track and cancel the in-flight operation before a response exists, and cover this ordering with a blocked-header fake followed by a successful request.

[P2] Put the flush test credentials inside params

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/tests/test_basic.py:335–338

test_flush_logic supplies api_key and voice_id at the top level, whereas BreezeBlueTTSConfig reads them exclusively from params. Pydantic ignores those extra fields, leaving params empty, so initialization emits a fatal missing-key error before creating the mocked client. ExtensionTesterFlush never stops on that error and sends its flush only after receiving audio, so the test cannot reach its assertions and can hang the local suite. Use the valid nested configuration and stop/fail the tester on unexpected initialization errors.

[P2] Cover request ownership and abnormal completion in the local suite

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/tests/test_audio_end.py:211–213

The controlled-clock assertion covers success only; partial-error and no-audio tests omit request_event_interval_ms, the flush test allows a duration tolerance without checking that interval, and dump verification covers success only. The suite also never appends text to one request ID or interleaves unfinished request IDs. These leave mandatory lifecycle cases from .agents/skills/ten-tts-review/references/tests.md uncovered. Add deterministic adapter tests asserting request ownership, exactly one terminal event, both exact audio-end fields, recovery, and emitted-PCM dump contents/finalization on error and cancellation, including cancellation before audio starts.

@TiagoAgora

TiagoAgora commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Guarder evidence for breezeblue_tts_python:

  • Extension: breezeblue_tts_python
  • Tested commit (PR HEAD): cc6fc4d
  • Command: task tts-guarder-test EXTENSION=breezeblue_tts_python (default tests/configs)
  • Result: 52 passed, 19 skipped, 0 failed in 473.86s
  • Skips are capability-based: connection status is WebSocket-only, there's no subtitle alignment in this version, and the rest are other vendors' hooks and endpoints.
  • PCM output from this run is in the attached zip: raw PCM, 24000 Hz, 16-bit signed little-endian, mono, 19.2 s.
Captura de ecrã 2026-10-30, às 15 44 29

guarder_output_24000hz_s16le_mono.pcm.zip

pylint reads the pydantic Field attributes (params, headers) as FieldInfo and
reports E1101, which fails the lint step of the agents CI. Replace the inline
pragmas, which black moves off the flagged line, with one scoped disable per
config method. No behaviour change.
@TiagoAgora

Copy link
Copy Markdown
Contributor Author

Note on the guarder evidence above

The guarder run was done at cc6fc4d4e. The PR HEAD is now 754f2a763, which only fixes a lint failure that the agents CI reported (pylint E1101 on pydantic field access in config.py).

  • The diff is limited to breezeblue_tts_python/config.py: the inline # pylint: disable pragmas are replaced by one scoped disable per method. No behaviour change. See the commit: 754f2a7.
  • Re-verified at 754f2a763: pylint with the repo's .pylintrc is 10.00/10 (exit 0), black --check is clean, and the 64 extension-local tests pass.
  • The full guarder suite was not rerun at 754f2a763. The result above (52 passed, 19 skipped, 0 failed) is for cc6fc4d4e. Happy to rerun it if a reviewer wants evidence at the exact HEAD.

@github-actions

Copy link
Copy Markdown

[P1] Redact authentication headers before logging configuration

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/config.py:115–119

Header authentication is explicitly accepted by validate() and covered by test_auth_header_satisfies_api_key_requirement, but to_str(sensitive_handling=True) only masks params.api_key. A credential supplied through headers["xi-api-key"] or headers["Authorization"] remains verbatim in the configuration that the HTTP base logs at INFO during initialization. Redact authentication headers case-insensitively and add a header-authentication redaction regression test.

[P1] Refresh the guarder evidence for the current HEAD

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/manifest.json:1–4

The attached screenshot and downloadable PCM are accessible, and the archive contains a non-empty PCM file. However, they document cc6fc4d4e, while HEAD is 754f2a76386f3460b753d5c1cacfb75445641f09; the owner's follow-up explicitly confirms the full guarder suite was not rerun. Although the intervening change addresses lint, .agents/skills/ten-tts-review/references/guarder.md explicitly requires a full passing run at the current HEAD. This remains a merge blocker: rerun the default command and attach the current SHA, counts, skip rationale, result screenshot, and PCM output with its format.

[P2] Interrupt cancellation while waiting for response headers

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/breezeblue_tts.py:135–137

When a flush arrives during connection establishment or response-header wait, _response is still None: it is assigned only after the streaming context enters at line 209. Cancellation therefore sets a flag without interrupting the HTTP operation. The base consumes get() serially, so subsequent speech remains blocked until headers arrive or the 60-second HTTP timeout fires, even after flush completion. Track and interrupt the operation before a response exists, and cover a blocked-header cancellation followed by a successful request.

[P2] Initialize the flush test with valid nested credentials

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/tests/test_basic.py:335–338

These top-level credentials are ignored by the config model, leaving params empty and causing initialization to emit a fatal missing-key error. The tester ignores error events. The base's later client-reinitialization path bypasses validation, and the permissive HTTP mock accepts the resulting url=None, allowing the flush test to pass after initialization has failed. Put the credentials inside params and stop/fail on unexpected errors so this test verifies cancellation of a successfully initialized extension.

[P2] Cover abnormal completion and request ownership in adapter tests

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/tests/test_audio_end.py:211–213

The sole request_event_interval_ms assertion covers success. Partial-error and no-audio cases omit it; the flush test permits a duration tolerance without checking the interval. The suite also lacks appended/interleaved unfinished request IDs and dump verification on error, flush, or cancellation. These are mandatory local-test cases in .agents/skills/ten-tts-review/references/tests.md. Add deterministic adapter tests asserting request ownership, exactly one terminal event, both exact audio-end fields with a controlled clock, dump contents/finalization, and recovery after cancellation, including cancellation before audio starts.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

[P1] Redact credentials in header-based auth before logging the config
ai_agents/agents/ten_packages/extension/breezeblue_tts_python/config.py:108-119
This is still present at 1e698e3c. validate() accepts headers["xi-api-key"] or headers["Authorization"] as the only credential, and test_auth_header_satisfies_api_key_requirement covers that case. However, to_str(sensitive_handling=True) masks only params["api_key"], so a key supplied through headers is printed in full in the config string that the HTTP base logs at INFO on init. Mask xi-api-key and Authorization in config.headers, matching names case-insensitively, and add a regression test that calls to_str() with header-only auth.

[P1] Guarder evidence does not match the current PR HEAD
ai_agents/agents/ten_packages/extension/breezeblue_tts_python/manifest.json:1-4
The screenshot and PCM zip are attached and accessible, but they were produced at cc6fc4d4e. HEAD is now 1e698e3ca, which adds the pylint commit 754f2a763 and a merge from main. The owner's note confirms the full suite was not rerun. The new-extension gate requires a full passing task tts-guarder-test EXTENSION=breezeblue_tts_python run at the current HEAD. Please rerun it and post the SHA, the counts, the skip rationale, a screenshot, and the PCM file.

[P2] Flush cannot interrupt a request still waiting for response headers
ai_agents/agents/ten_packages/extension/breezeblue_tts_python/breezeblue_tts.py:132-144 (see also lines 202-209 and 229)
cancel() can close only self._response, and that field is set only after client.stream(...) returns headers (line 209). A flush that arrives during connect or TTFB, or during the asyncio.sleep between 502/503/504 retries (line 229), only sets the flag. The base consumes get() serially, so the next utterance waits until the vendor returns headers or the 60 s timeout fires, and the stale request finishes only then. The retry path can also open a new request after the flush. Fix suggestion: run the request in a task that cancel() can interrupt, or wait on an asyncio.Event alongside the request. Check _is_cancelled before each retry sleep and before opening each split-text chunk. Add a local test that cancels while the fake stream() is still blocked on headers. The existing cancel tests (test_cancel_closes_the_active_response, test_cancel_mid_stream_flushes) only cover cancellation after headers have arrived.

[P2] No independent TTFB hook is registered for this extension
ai_agents/agents/ten_packages/extension/breezeblue_tts_python/extension.py:17
integration_tests/tts_guarder/tests/ttfb_reference.py has HOOKS/CONFIGS entries only for the elevenlabs, bytedance, minimax and soul WebSocket extensions. breezeblue_tts_python has no entry and there is no observer in tests/ttfb_hooks/, so the guarder's TTFB check compares only against the production timer. Add an HTTP observer for POST /v1/text-to-speech/{voice_id}/stream. It should take a monotonic timestamp when the request body is sent and another at the first non-empty response chunk, keyed by request ID. Register it with BreezeBlueTTSConfig/url, or state the limitation explicitly if the HTTP transport cannot be observed.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

[P1] Redact authentication headers before logging configuration

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/config.py:115–119

validate() explicitly supports header authentication, but to_str(sensitive_handling=True) masks only params.api_key. Credentials supplied through headers["xi-api-key"] or headers["Authorization"] remain verbatim in the configuration that the HTTP base logs at INFO during initialization. Redact authentication headers case-insensitively and add a regression test for header-only authentication.

[P1] Refresh the required guarder evidence at the current HEAD

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/manifest.json:1–4

The owner's screenshot and downloadable archive are accessible, and the archive contains a non-empty PCM file, but both document cc6fc4d4e. Current HEAD is 1e698e3ca4251022309c9bae6afd55de0d3037ad, and no owner/workflow comment supplies a full passing run for it. This remains a merge blocker under .agents/skills/ten-tts-review/references/guarder.md, which requires current-HEAD evidence for a new extension. After the fixes, rerun task tts-guarder-test EXTENSION=breezeblue_tts_python and attach the tested SHA, counts, capability-based skip rationale, result screenshot, and guarder-produced PCM with its sample rate, bit depth, and channel count.

[P2] Interrupt cancellation while waiting for response headers

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/breezeblue_tts.py:135–137

If a flush arrives during connection establishment or response-header wait, _response is still None; assignment happens only after the streaming context enters at line 209. Cancellation therefore sets a flag without interrupting the HTTP operation. The inherited input loop awaits request_tts() serially, so subsequent speech remains blocked until headers arrive or the 60-second timeout expires, even after tts_flush_end. Track and interrupt the operation before a response exists, and cover blocked-header cancellation followed by a successful request.

[P2] Initialize the flush test with valid nested credentials

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/tests/test_basic.py:335–338

These top-level credentials are ignored by the config model, leaving params empty and causing initialization to emit a fatal missing-key error. The tester ignores error events, while the base's subsequent client-reinitialization path bypasses validation and the permissive HTTP mock accepts url=None. Consequently, the flush test can pass after initialization has failed. Put the credentials inside params and fail on unexpected errors so the test exercises cancellation of a successfully initialized extension.

[P2] Complete the mandatory lifecycle and termination assertions

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/tests/test_audio_end.py:211–213

Only success asserts request_event_interval_ms; partial-error and no-audio cases omit it, and the flush test allows a duration tolerance without checking the interval. The suite also lacks appended/interleaved unfinished request IDs, recovery after cancellation before audio starts, and dump-content/finalization checks on error or cancellation. These are mandatory local-test cases in .agents/skills/ten-tts-review/references/tests.md. Add deterministic adapter tests asserting request ownership, exactly one terminal event, both exact audio-end fields with a controlled clock, finalized dump contents, and subsequent-request recovery.

[P2] Register an independent TTFB observer for the HTTP transport

ai_agents/agents/ten_packages/extension/breezeblue_tts_python/extension.py:17

The adapter uses HTTP streaming, but the guarder's ttfb_reference.py has no BreezeBlue entry in HOOKS or CONFIGS, and its installer only wraps WebSocket transports. It therefore yields None for this extension, leaving the guarder with only a tester elapsed-time upper bound; the local metric test likewise checks only ttfb >= 200. Neither detects an incorrectly placed production timer. The review baseline requires an independent reference for HTTP providers too: add and register an HTTP observer of text-send and first non-empty PCM arrival, with request-isolation tests and a reference-enabled live run. Subtitle alignment is not required because this adapter supplies no timestamps.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants