Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
7c891f6
docs: plan live dictation and Soniox bypass
Snehit70 Jun 26, 2026
7d7ca92
feat: add live dictation text writer
Snehit70 Jun 26, 2026
b7c41de
feat: add live dictation config
Snehit70 Jun 26, 2026
42adde3
feat: add live provider transcript contract
Snehit70 Jun 26, 2026
55e684f
feat: wire live dictation transcript events
Snehit70 Jun 26, 2026
6bf20c6
feat: add soniox streaming provider
Snehit70 Jun 26, 2026
d33d824
feat: wire soniox live bypass hotkey
Snehit70 Jun 26, 2026
0dc530b
docs: document live dictation configuration
Snehit70 Jun 26, 2026
57bed1e
feat(config): add retypeFormatted and paragraphPauseMs to live dictat…
Snehit70 Jun 26, 2026
b9425c6
feat(soniox): add token spacing, paragraph breaks, LLM formatting, an…
Snehit70 Jun 26, 2026
937b218
feat(service): wire LLM formatting, retype mechanism, and IPC soniox-…
Snehit70 Jun 26, 2026
1ad204f
feat(cli): add soniox-toggle subcommand via IPC socket
Snehit70 Jun 26, 2026
ad0a16b
test(soniox): add paragraph break, spacing, and tag-stripping tests
Snehit70 Jun 26, 2026
4541f60
docs: document Soniox formatting pipeline, config options, and PRD
Snehit70 Jun 26, 2026
d04966d
fix(soniox): use join('') for tokens that already include spaces
Snehit70 Jun 26, 2026
eb70101
fix(soniox): strip spaces before punctuation in live tokens
Snehit70 Jun 26, 2026
53cfab4
fix(live-dictation): use Shift+End escape sequence on Wayland
Snehit70 Jun 26, 2026
a48050b
fix(live-dictation): use wtype native key simulation for select on Wa…
Snehit70 Jun 26, 2026
61a02f5
feat(soniox): pass boost words to Soniox as context.terms
Snehit70 Jun 26, 2026
02ffa1d
feat(soniox): add context.general, contextText, languageHintsStrict c…
Snehit70 Jun 27, 2026
fba140b
feat(soniox): send context.general, contextText, languageHintsStrict …
Snehit70 Jun 27, 2026
fdaea01
test: add tests for Soniox context config fields; update docs
Snehit70 Jun 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,14 @@ _Avoid_: hotkey, shortcut
The on-screen status surface that reflects live daemon state.
_Avoid_: popup, HUD

**Live Dictation**:
Entering stable transcript text in the currently focused text field while a recording is still active, then preserving the final transcript through the normal clipboard and history paths. When driven by Deepgram streaming, the full quality pipeline runs after stop. When driven by Soniox, it operates as a [[Provider Bypass]]. During recording, only final tokens are typed — interim/partial tokens are discarded to avoid flickering and fragility. Tokens are joined with a single space; double spaces are collapsed and trailing spaces are trimmed. Paragraph breaks are inserted during live typing when the gap between consecutive Soniox token messages exceeds `liveDictation.soniox.paragraphPauseMs` (default: 3000ms) — a paragraph break is emitted as a double-space prefix to the next token. Self-corrections are not backspaced — mistakes remain in the typed text and can be corrected manually after recording stops. After stop, a minimal Groq/Llama 3.3 pass adds only paragraph breaks at natural sentence boundaries — no filler removal, no punctuation fixes, no rewriting. A config option (`liveDictation.retypeFormatted`, default: true) controls whether the LLM-formatted text replaces what was typed (via Home + Shift+End selection + retype, avoiding Ctrl+A risk in editors) or only affects clipboard/history (off). Per-token debug logging is available at `LOG_LEVEL=debug`. Structured perf entries (`type: "perf"`) include `paragraphBreakCount` and `llmFormattingMs` for statistical analysis.
_Avoid_: live paste, streaming paste

**Provider Bypass**:
A recording path that uses one live transcription provider directly and skips the Groq plus Deepgram merge and quality pipeline by design. The Soniox live dictation path is the primary instance — it trusts Soniox output for real-time typing and skips validation, hallucination detection, and recovery entirely. Token spacing is normalized (joined with space, double spaces collapsed, trailing spaces trimmed) and paragraph breaks are inserted based on configurable inter-token pause thresholds. After stop, a minimal LLM formatting pass may add paragraph breaks only (no rewriting). See [[Live Dictation]] for the full formatting pipeline.
_Avoid_: fallback, fast mode

## Observability

**Readiness**:
Expand Down
56 changes: 55 additions & 1 deletion docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ If an API key is missing from `config.json`, the application will fall back to t
- `GROQ_API_KEY`: Fallback for `apiKeys.groq`
- `GROQ_FALLBACK_API_KEY`: Fallback for `apiKeys.groqFallback`
- `DEEPGRAM_API_KEY`: Fallback for `apiKeys.deepgram`
- `SONIOX_API_KEY`: Fallback for `apiKeys.soniox` when Soniox live dictation is enabled

### Logging
- `LOG_LEVEL`: Sets the minimum logging level. Options: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `silent`. Default: `info`.
Expand All @@ -44,7 +45,8 @@ The configuration is a JSON file structured into several sections.
"apiKeys": {
"groq": "gsk_...",
"groqFallback": "gsk_...",
"deepgram": "00000000-0000-0000-0000-000000000000"
"deepgram": "00000000-0000-0000-0000-000000000000",
"soniox": "soniox_..."
},
"behavior": {
"hotkey": "Right Control",
Expand Down Expand Up @@ -87,6 +89,22 @@ The configuration is a JSON file structured into several sections.
"Groq",
"Deepgram"
]
},
"liveDictation": {
"enabled": false,
"insertionCommand": "auto",
"retypeFormatted": true,
"soniox": {
"enabled": false,
"triggerKey": "Right Alt",
"paragraphPauseMs": 3000,
"languageHintsStrict": true,
"contextGeneral": [
{ "key": "domain", "value": "software development" },
{ "key": "topic", "value": "technical architecture and implementation" }
],
"contextText": "The speaker is dictating technical prompts for AI coding agents, software architecture discussions, code comments, and developer workflow instructions. Content includes programming terminology, API references, database schemas, CLI commands, and system administration tasks."
}
}
}
```
Expand All @@ -104,6 +122,7 @@ Authentication credentials for the transcription services.
| `groq` | String | N/A | API key for Groq (Whisper V3). | Must start with `gsk_`. | [Groq Console](https://console.groq.com/keys) |
| `groqFallback` | String | Optional | Secondary Groq API key used only when the primary merge key is rate-limited/quota-limited. | Must start with `gsk_` if provided. | [Groq Console](https://console.groq.com/keys) |
| `deepgram` | String | N/A | API key for Deepgram (Nova-3). | 40-char hex string or UUID. | [Deepgram Console](https://console.deepgram.com/) |
| `soniox` | String | Optional | API key for Soniox real-time STT. Required only when `liveDictation.soniox.enabled` is used. | Non-empty string. | [Soniox Console](https://console.soniox.com/) |

#### How to obtain API Keys

Expand All @@ -117,6 +136,11 @@ Authentication credentials for the transcription services.
- Navigate to **API Keys** and create a new key.
- **Format**: The key is typically a **40-character hexadecimal string** (e.g., `abcdef1234567890abcdef1234567890abcdef12`). Legacy keys or specific project IDs might use a UUID format, both are supported.

3. **Soniox API Key**:
- Go to the [Soniox Console](https://console.soniox.com/).
- Create an API key for real-time speech-to-text.
- Hyprvox reads it from `apiKeys.soniox` or `SONIOX_API_KEY`.

---

### 2. Behavior (`behavior`)
Expand Down Expand Up @@ -300,6 +324,36 @@ In streaming mode, performance logs include Deepgram finalization observability:

These fields are for analysis and future tuning. Do not treat a frequent `finalize_timeout` by itself as proof that endpointing should change; compare the final transcript quality and late finalization signals first.

### 5. Live Dictation (`liveDictation`)

Live Dictation controls focused-text insertion while recording. It is disabled by default.

| Option | Type | Default | Description | Validation Rules |
| :--- | :--- | :--- | :--- | :--- |
| `enabled` | Boolean | `false` | Type stable live transcript text into the currently focused input during the normal streaming path. | Requires `transcription.streaming: true` to receive Deepgram live transcript events. |
| `insertionCommand` | String | `"auto"` | Command used for focused text insertion. | `"auto"`, `"wtype"`, or `"xdotool"`. |
| `retypeFormatted` | Boolean | `true` | After stop, replace the typed text with the LLM-formatted version using Home + Shift+End selection. When off, formatted text only affects clipboard and history. | N/A |
| `soniox.enabled` | Boolean | `false` | Enable the separate Soniox provider-bypass hotkey. | Requires `apiKeys.soniox` or `SONIOX_API_KEY` when used. |
| `soniox.triggerKey` | String | `"Right Alt"` | Separate hotkey for Soniox live dictation. | Same hotkey format as `behavior.hotkey`. |
| `soniox.paragraphPauseMs` | Number | `3000` | Minimum gap between consecutive Soniox token messages (in ms) before a paragraph break is inserted. | Integer `500`-`10000`. |
| `retypeFormatted` | Boolean | `true` | After stop, replace typed text with LLM-formatted version (Home + Shift+End + retype). If false, formatting only affects clipboard/history. | N/A |
| `soniox.paragraphPauseMs` | Number | `3000` | Minimum pause (ms) between Soniox token messages to trigger a paragraph break during live typing. | Integer `>= 500` |
| `soniox.languageHintsStrict` | Boolean | `true` | Restrict Soniox transcription to only the languages specified in `language_hints`. Prevents language drift. | N/A |
| `soniox.contextGeneral` | Array | Software dev metadata | Key-value pairs providing domain context to Soniox for improved recognition. | Array of `{key: string, value: string}` objects. |
| `soniox.contextText` | String | Technical context | Free-form text (max 10,000 chars) describing the typical dictation content for Soniox context. | String, max 10,000 chars. |

#### Normal Live Dictation

When `liveDictation.enabled` and `transcription.streaming` are both enabled, Hyprvox types committed Deepgram streaming transcript chunks into the focused input as they arrive. The final transcript still follows the normal Groq plus Deepgram merge, clipboard, history, and validation path after recording stops.

#### Soniox Provider Bypass

When `liveDictation.soniox.enabled` is enabled, pressing `liveDictation.soniox.triggerKey` starts a Soniox real-time STT session. Recorder PCM is sent directly to Soniox, stable final tokens are typed into the focused input, and the final Soniox transcript is copied to clipboard and appended to history when recording stops.

Tokens are joined with a single space; double spaces are collapsed and trailing spaces are trimmed. Paragraph breaks are inserted when the inter-token pause exceeds `soniox.paragraphPauseMs`. After stop, a minimal LLM pass adds paragraph breaks only — no rewriting. When `retypeFormatted` is true, the formatted text replaces what was typed via Home + Shift+End selection.

This path intentionally skips Groq, Deepgram batch transcription, merge, repair, and quality recovery. Use it when low-latency live dictation is more important than the normal multi-provider quality pipeline.

#### Language Options

For **v1.0**, `hyprvox` is optimized for and officially supports **English only**.
Expand Down
101 changes: 101 additions & 0 deletions docs/ISSUES-LIVE-DICTATION-SONIOX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Offline Issue Breakdown: Live Dictation And Soniox Provider Bypass

## 1. Live Dictation Text Writer Contract

**Type**: AFK

**Blocked by**: None - can start immediately

**User stories covered**: 1, 3, 4, 5, 6, 7, 8, 15, 18

## What to build

Build the smallest focused-input writer path that can accept stable transcript events and type only the committed delta into the currently focused field. The slice should include a command-backed text injection boundary with test doubles for command execution.

## Acceptance criteria

- [ ] Stable transcript chunks insert only new committed text.
- [ ] Repeated transcript events do not duplicate already inserted text.
- [ ] Wayland uses `wtype` by default when available.
- [ ] X11 uses `xdotool type` when Wayland is not active.
- [ ] Tests do not type into the real desktop.
- [ ] Insertion errors are returned as structured failures that callers can log and recover from.

## 2. Live Dictation Config And Readiness

**Type**: AFK

**Blocked by**: Issue 1

**User stories covered**: 8, 9, 13, 14, 18

## What to build

Add configuration for Live Dictation and Soniox credentials without changing default transcription behavior. Readiness checks should report missing optional dependencies only when the feature is enabled.

## Acceptance criteria

- [ ] Existing minimal configs still parse and keep Live Dictation disabled by default.
- [ ] Live Dictation config validates trigger key and insertion command settings.
- [ ] Soniox credentials can be read from config or environment.
- [ ] Missing Soniox credentials do not break default Groq plus Deepgram recording.
- [ ] Tests cover config defaults, enabled Live Dictation config, and invalid trigger key handling.

## 3. Live Provider Contract Around Existing Deepgram Streaming

**Type**: AFK

**Blocked by**: Issue 1

**User stories covered**: 1, 2, 9, 16, 17, 18

## What to build

Introduce a live provider contract that represents streaming transcript events, PCM input, and final transcript stop behavior. Adapt the current Deepgram streaming path to that contract while preserving current default behavior.

## Acceptance criteria

- [ ] Normal streaming recordings still produce the same final clipboard behavior.
- [ ] Live provider events can feed the Live Dictation text writer when enabled.
- [ ] Deepgram streaming failures still fall back to existing batch behavior.
- [ ] Tests prove the contract with a fake provider and do not call Deepgram.

## 4. Soniox Provider Bypass Recording Path

**Type**: AFK

**Blocked by**: Issues 1, 2, 3

**User stories covered**: 10, 11, 12, 13, 14, 16, 17, 18

## What to build

Add Soniox as a live provider and wire a separate provider-bypass trigger key. The path streams recorder PCM to Soniox, feeds stable transcript events to Live Dictation, and writes the final Soniox transcript to clipboard and history without running Groq, Deepgram batch, merge, repair, or quality recovery.

## Acceptance criteria

- [ ] Soniox provider connects to the documented real-time STT WebSocket endpoint.
- [ ] Provider bypass does not call Groq, Deepgram batch, merge, repair, or quality recovery.
- [ ] Final Soniox transcript is copied to clipboard and appended to history.
- [ ] Provider errors surface as user-facing Soniox errors.
- [ ] Tests use a fake WebSocket transport and do not hit Soniox.

## 5. Runtime Verification And PR Readiness

**Type**: HITL

**Blocked by**: Issues 1, 2, 3, 4

**User stories covered**: 1-18

## What to build

Run the focused local test suite, verify no default behavior changed, perform a manual live dictation check on the user machine, then prepare incremental commits and a PR.

## Acceptance criteria

- [ ] Focused unit tests pass.
- [ ] Default config and normal transcription behavior remain unchanged.
- [ ] Live Dictation can be manually tested in a scratch focused input.
- [ ] Soniox provider bypass can be manually tested when credentials are available.
- [ ] Commits are incremental and scoped to planning, infrastructure, provider contract, Soniox path, and verification.
Loading