Skip to content

feat: transcribe dictation into the channel composer - #473

Merged
iamnbutler merged 3 commits into
mainfrom
dictation
Oct 16, 2026
Merged

iamnbutler merged 3 commits into
mainfrom
dictation

Conversation

@iamnbutler

Copy link
Copy Markdown
Contributor

Part of #38.

The channel composer's mic button now dictates. Pressing it starts the platform's speech recognition (webkitSpeechRecognition) and writes the transcript into the editor at the cursor as you speak, interim words included, while the existing recording glow (useMic / Composer.Glow) runs. Stop, Send, or silence ends it.

  • packages/ui/src/hooks/use-dictation.ts: a small useDictation(active, onText, onEnd) plus canDictate. Turning it off aborts the recognizer and detaches its handlers, so no late result can land in an editor that has already moved on. The words you saw last stay in the editor.
  • ChatComposer: mic now means "dictate where the runtime can". It is gated on canDictate, so Firefox and other browsers without speech recognition never show the button. The existing submit, record, and stop states, the glow, and the red ring are reused unchanged.
  • apps/app/src/composer.tsx: enables mic for channel chats. The dashboard's new-channel composer keeps mic={false} for now because another change is in flight in that file.
  • Desktop: electrobun.config.ts declares NSMicrophoneUsageDescription and NSSpeechRecognitionUsageDescription. Electrobun writes both into Info.plist, which I checked on a dev build. native/entitlements.plist adds com.apple.security.device.audio-input, which release builds need because they sign with the hardened runtime.

Product decisions

  • I used the Web Speech API instead of a provider model or a native bridge. It runs inside WKWebView with no host or channel changes. On macOS it uses Apple's speech recognizer; in Chrome it uses Google's service.
  • Recognition uses navigator.language.
  • The mic only appears while the composer is empty, which is the existing submit-button behavior. To add more dictation after stopping, you have to clear the text first.

Verification

  • I ran a bare WKWebView harness (Swift, same WebKit as the app) on this Mac. webkitSpeechRecognition is present, and start → audiostart → speechstart → interim results → final result all worked with live speech, as long as the window was visible. If the page isn't visible, it fails with not-allowed: Page is not visible to user.
  • The dev desktop build (bun desktop dev) builds. Its Info.plist has both usage strings, and its signature carries audio-input.
  • Not verified: clicking the mic in the built Ace app end to end. The Mac's session locked before I could drive the UI or answer the first-run macOS mic and speech prompts. I also couldn't check the glow's getUserMedia running at the same time as recognition inside WKWebView. Electrobun's own "This page wants to access your microphone" alert may also show before the macOS prompts.
  • bun types and bun run ci pass.

Remaining for #38

  • Run it end to end in the desktop app, including the first-run permission prompts and the glow running alongside recognition.
  • Dictation in the dashboard's new-channel composer.
  • Appending dictation to existing text. This needs a mic affordance while the composer isn't empty.

Built in Ace

Built with Claude Code outside Ace, in a session already running there; no Ace gap involved.

The composer's mic button now runs WebKit/Chromium speech recognition and writes the
transcript, interim words included, into the editor while the recording glow shows. The
desktop app declares microphone and speech recognition usage strings and the audio-input
entitlement its hardened runtime needs.

Why: packages/ui/src/hooks/use-dictation.ts#function useDictation | deactivating aborts and detaches handlers, so no result lands after Stop or Send moves the editor on
Why: apps/desktop/electrobun.config.ts | WKWebView dictation needs NSMicrophoneUsageDescription and NSSpeechRecognitionUsageDescription in Info.plist, or macOS ends the app on first use
@iamnbutler iamnbutler mentioned this pull request Oct 16, 2026
3 of 4 tasks
@iamnbutler
iamnbutler merged commit ba38001 into main Oct 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant