Voice mode: all bundled ASR models fail silently — MultiModalProcessor routing bug for nemotron_speech (RNNT) in Foundry Local Core

Describe the bug

/voice records audio successfully (level meter reacts, mic capture confirmed via raw PulseAudio capture) but every transcription comes back empty, for all three models offered in the /voice model picker:

nemotron-3.5-asr-streaming-0.6b
nemotron-speech-streaming-en-0.6b
nemotron-speech-streaming-es-0.6b

All three share the same nemotron_speech (RNNT) architecture, so switching models does not help — there is no working model in the current picker.

Root cause (traced to source)

The Foundry Local Core native audio transcription path throws on every call:

Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: MultiModalProcessor cannot be created. nemotron_speech is not a registered multi-modal model type.
   at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr)
   at Microsoft.ML.OnnxRuntimeGenAI.MultiModalProcessor..ctor(Model)
   at Microsoft.AI.Foundry.Local.AudioClient.Transcribe(String, String, Nullable`1)

This is not a stale/outdated onnxruntime-genai engine issue. I downloaded and compared the public onnxruntime-genai v0.14.0 release (published 2026-05-29, after the "Multilingual Streaming Nemotron ASR" PR merged 2026-05-22) against the bundled runtime (labeled 0.14.1 in deps_versions.json, but not a publicly-tagged release). Both contain full, identical NemotronSpeechModel/NemotronSpeechState support (confirmed via symbol inspection of libonnxruntime-genai.so).

Cross-checking against microsoft/onnxruntime-genai's src/models/model.cpp (https://github.com/microsoft/onnxruntime-genai/blob/main/src/models/model.cpp) confirms this is architecturally expected:

nemotron_speech is an RNNT-type model (ModelType::IsRNNT) and is correctly routed to NemotronSpeechModel when the model is loaded (CreateModel() in model.cpp) — this is why Model loaded successfully: nemotron-3.5-asr-streaming-0.6b-generic-cpu:3 appears in the Foundry log and the model shows as loaded.
However, MultiModalProcessor's constructor uses a separate, hardcoded processor_factory_ registry that only contains vision+text chat model types (e.g. phi4mm, gemma4, etc.) — RNNT/TDT/ALM (Whisper) audio model types were never meant to go through MultiModalProcessor at all.

So the bug is that Microsoft.AI.Foundry.Local.Core's AudioClient.Transcribe/streaming session path unconditionally constructs a MultiModalProcessor for audio transcription, instead of dispatching RNNT/TDT/ALM model types (nemotron_speech, Parakeet TDT, Whisper) through their own dedicated processing path (the same one used successfully at model-load time). Since Microsoft.AI.Foundry.Local.Core.so is a closed-source component, this can't be worked around or patched client-side — no onnxruntime-genai version swap fixes it, since the engine itself already works correctly and simply refuses (correctly, by its own design) to construct a MultiModalProcessor for a non-multimodal-chat model type.

Affected version

GitHub Copilot CLI 1.0.69-0 (also reproduced on 1.0.66-1 — same bundled runtime pins: onnxruntime-genai 0.14.1, foundry-local-core 1.2.3, so the bug is not CLI-version-specific).

Steps to reproduce the behavior

On Linux x64 (reproduced under WSL2/WSLg, but likely affects any Linux x64 install), run /voice, enable it, select any of the three offered models.
Press the voice-record shortcut, speak clearly for several seconds (confirmed via raw parec capture that real speech is captured at healthy signal levels, and the in-app level meter visibly reacts).
Stop recording.

Expected behavior

Spoken audio is transcribed to text and inserted into the input box.

Actual behavior

No error is shown to the user; the recording UI behaves normally, but the transcript is always empty. ~/.github-copilot-cli/logs/foundry.core*.log shows:

[ERR] Error executing audio_transcribe: Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: MultiModalProcessor cannot be created. nemotron_speech is not a registered multi-modal model type.
...
Audio stream stop: session <id>, final flush text: '', full transcript: ''

Additional context

Confirmed mic capture and PulseAudio pipeline are fully healthy (not a WSLg/audio-hardware issue) via direct raw parec -d RDPSource capture showing strong, correctly-timed speech signal.
Confirmed via a standalone script using the bundled foundry-local-sdk package directly (bypassing the Copilot CLI JS entirely) that audioClient.transcribe() throws the identical MultiModalProcessor/nemotron_speech error against a real captured WAV file, ruling out any Copilot-CLI-specific bug in the JS layer.
Suggested fix: in Foundry Local Core's audio transcription path, dispatch by ModelType::IsRNNT/IsTDT/IsALM (mirroring CreateModel()'s dispatch in onnxruntime-genai's model.cpp) instead of unconditionally constructing a MultiModalProcessor.

Thank you for the detailed investigation. We will resolve this issue ASAP.

@sylvanc

@sylvanc after looking into this, I have one question, are you using below models

nemotron-3.5-asr-streaming-0.6b
nemotron-speech-streaming-en-0.6b
nemotron-speech-streaming-es-0.6b

to do the one-shot ASR or you are doing the streaming process?

Yes, using those - those are the models available in Copilot CLI.

Follow-up with a correction to my original diagnosis and the complete WSLg
capture findings.

Important clarification: CLI uses streaming ASR

The Copilot CLI /voice path uses
createLiveTranscriptionSession(), not one-shot audioClient.transcribe().

I tested both paths directly against the bundled Foundry runtime:

One-shot audioClient.transcribe() still throws the
MultiModalProcessor cannot be created ... nemotron_speech error described
above. That remains a Foundry Local Core routing bug.
Live streaming ASR works. Feeding a known-good 16 kHz mono PCM speech sample
through createLiveTranscriptionSession() produced the expected transcript
with both short and long inputs.

Therefore the empty /voice results under WSLg were not solely explained by
the one-shot Foundry bug. The CLI streaming path also had WSLg microphone
lifecycle problems.

Reproduction environment

Copilot CLI 1.0.71-0
WSL2 / WSLg
Windows Terminal
WSLg PulseAudio server: unix:/mnt/wslg/PulseServer
Source: RDPSource, s16le, mono, native 44.1 kHz
Bundled recorder: @picovoice/pvrecorder-node

Findings

1. Cold `RDPSource` fails its first PvRecorder read

PvRecorder(-1) correctly selects RDP Source, and start() succeeds, but
the first read() after PulseAudio has suspended the source throws
PvRecorderStatusIOError. A second read on the same recorder handle succeeds.

Direct parec capture also confirms that the physical microphone and Windows
permissions are working.

Recommended WSL-specific handling:

Enumerate devices and explicitly select the index named RDP Source.
If the first read fails while waking a suspended WSLg source, retry that
read once on the same handle.

2. Disabling PulseAudio idle suspension is the wrong workaround

Unloading module-suspend-on-idle, or forcing RDPSource to remain merely
IDLE without a consumer, causes WSLg's virtual source to free-run/build a
backlog.

Measured over five wall-clock seconds:

Requested rate	Samples represented	Effective rate
16 kHz	42.58 s	8.52x
44.1 kHz	42.22 s	8.44x
48 kHz	43.10 s	8.62x

This PCM has plausible non-zero amplitude but is not recognizable speech to
either bundled streaming ASR model. It explains the level meter moving while
the final transcript remains empty.

With normal suspension restored, a true cold eight-second PulseAudio capture
contained 7.14 seconds of correctly clocked audio (the missing ~0.86 seconds
was source wake-up). Once a real consumer kept the source RUNNING, ten
100 ms PvRecorder frames took 1.02 seconds, as expected.

3. Correct WSLg clock/wake handling

The working WSL-specific design is:

When the Copilot voice process starts, open one real PulseAudio capture
consumer for RDPSource.
Keep it alive only for the lifetime of the voice process.
Keep the latest ten 100 ms / 16 kHz mono frames in a one-second in-memory
ring.
Do not write audio to persistent storage.
Do not change system PulseAudio suspension policy.
Stop the consumer when the voice process exits; restart it on unexpected
failure.

This keeps WSLg's source genuinely RUNNING and correctly clocked while voice
mode is available. The one-second memory ring also supplies pre-roll when the
operator begins speaking immediately after activating push-to-talk. The ring
is not sent to ASR unless the user explicitly starts a voice session.

4. CLI drops microphone frames while the STT session is opening

The microphone and Foundry session are opened in parallel, but the current PCM
handler returns unless the session status is already active. Consequently,
audio captured while Foundry is opening is discarded.

Source-level fix:

openSession:
    currentSession.pendingPcm = []

onPcmFrame:
    if status == opening:
        append frame to a bounded pendingPcm queue
        return
    if status == active:
        append frame to Foundry normally

onSessionWarmed:
    status = active
    append pendingPcm frames to Foundry in order
    clear pendingPcm

The WSL one-second pre-roll is returned as the first recorder frame and follows
this same opening buffer path, so the first words are preserved.

5. Commit drains only one frame and truncates trailing speech

The current microphone close path uses a boolean commitDrain. After release,
the read loop forwards exactly one additional frame and returns, with a
150 ms safety timeout. In practice this cuts off the last word when the user
releases Space immediately after speaking.

The working change was:

commitDrain: false  -> commitDrain: 0
on commit:             commitDrain = 5
after each frame:      decrement; return at zero
drain timeout:         150 ms -> 750 ms

At the current 1,600-sample / 16 kHz frame size, this retains 500 ms of real
microphone tail before finalization.

Confirmed result

With these changes:

/voice transcribes normally under WSLg.
Speech begun immediately after activation is retained.
Short recordings no longer disappear during startup.
Releasing Space immediately after the final word retains the ending.
No system service, shell configuration, or persistent audio file is needed;
the WSL keeper and ring live entirely inside the Copilot voice process.

This should be implemented in the unbundled recorder/voice-session source
rather than patching the distributed voice-server.js bundle.

Thank you so much looking into this.

One-shot audioClient.transcribe() still throws the
MultiModalProcessor cannot be created ... nemotron_speech error described
above. That remains a Foundry Local Core routing bug.

This will use the API which will AudioTranscribe() api in the Core code, which is designed for the none streaming model. As such, we need Nemotron streaming model to support this feature, and simulate the streaming for the whole voice clip.

Thanks

Ok! I have this working locally on my machine with a patch. Due to minimization, it's a 343 kb patch, but I'm happy to post it here if that's helpful, just let me know.

Sure, or you can send to my Teams, I continue working on this and testing it during the weekend. Hopefully get back to you by Monday. Thanks

Rebased and reliability-corrected for Copilot CLI 1.0.71-3.

Versioned patches are retained together here:

https://gist.github.com/sylvanc/088fbd867e544eb0a5a73044f2bc8b11

1.0.71-3 exact raw patch

https://gist.githubusercontent.com/sylvanc/088fbd867e544eb0a5a73044f2bc8b11/raw/a1335dbf99956ddb3e1516a3ded7d4e1adc956e3/copilot-cli-1.0.71-3-wsl-voice-final-prettified.patch

SHA-256: cdecabf948edc78509d8ae97124a99f8a3fddf329313e7c9f29bf7d56fba9f2f

Why the reliability revision was necessary

Live 1.0.71-3 logs showed the personalized speaker gate scoring the actual user at 0.24–0.40, applying gains from 0.00–0.63, and producing empty/partial transcriptions. Speaker gating is now experimental opt-in only via COPILOT_VOICE_SPEAKER_FILTER=1; normal operation is fail-open.
RNNoise previously initialized lazily on the first recording. It now preloads asynchronously when the voice server boots, without opening the microphone.
The 1.17–1.37 second cold start was isolated to the built-in Intel MIPI SoundWire microphone path: direct Windows capture reproduced it without WSLg. Aeropex by AfterShokz measured 204 ms directly and 157 ms end-to-end through a fresh WSLg RDPSource.
The WSL keeper remains enabled by default for slow microphone endpoints, but it is now runtime-switchable and persistent with /voice keeper on, /voice keeper off, and /voice keeper status. COPILOT_VOICE_WSL_KEEPER remains the initial default when no saved preference exists.
The working live validation captured the complete beginning and end of a long utterance; adaptive tail completed in 350 ms.

1.0.71-3 audit and retained behavior

Upstream changes are limited to invalid-settings warnings and terminal-setup handling; pristine voice-server.js and PvRecorder are byte-identical to 1.0.71-2.
Preserves upstream /voice devices, listAudioDevices/selectAudioDevice, and persisted {name, occurrence} microphone selection.
Adds a wslMicKeeper voice-worker RPC and persisted voice.wslKeeper setting without changing upstream device-selection behavior.
Explicit saved devices bypass the WSL source-selection override; system-default deviceId=-1 resolves to RDP Source.
Retains first-Space PTT, 10 ms capture, 40 ms Foundry packets, opening buffering, adaptive tail, RNNoise, whisper leveling, PCM guards, selector voice, release-to-submit, and agent-facing ASR context metadata.
Choice-only ask_user prompts remain protected: PTT is disabled when allowFreeform=false.
Deliberately excludes tap-to-speak, Nemotron 3.5, and forced language=en.

Validation

Exact published bytes apply cleanly from stable formatted pristine 1.0.71-3 bundles and reproduce the installed sources.
PTT state machine remains identical to the validated prior version.
Typed-prefix + spoken insertion + typed-suffix composition passes.
DSP smoke: 31,920 output bytes for 32,000 input bytes; malformed odd-byte PCM falls back raw.
Schema-2 speaker profile remains available for explicit opt-in; model SHA-256 is enforced.
Runtime keeper lifecycle and RPC tests passed disabled → enabled/active → disabled/inactive.
Actual slash-command smoke tests passed for status, off, on, and saved preference persistence.
Keeper shutdown, /voice keeper off, and COPILOT_VOICE_WSL_KEEPER=0 leave no keeper capture consumer.

Dependencies after applying

npm install --no-save --package-lock=false --ignore-scripts --no-audit --no-fund \
  @echogarden/rnnoise-wasm@0.2.0 \
  @echogarden/speex-resampler-wasm@0.3.0 \
  sherpa-onnx-node@1.13.4

mkdir -p voice-models
curl -L --fail \
  https://huggingface.co/csukuangfj/speaker-embedding-models/resolve/main/nemo_en_speakerverification_speakernet.onnx \
  -o voice-models/nemo_en_speakerverification_speakernet.onnx

printf "%s  %s\n" \
  d204dc8aac0014b8543f05fc8e310510c7022bc65b6452c203ec205ef7a66b23 \
  voice-models/nemo_en_speakerverification_speakernet.onnx | sha256sum -c -

The SpeakerNet weights remain a separate download; review their applicable NVIDIA model license before redistribution.

Stable Copilot CLI 1.0.71 voice patch updated with a first-PCM readiness cue.

Versioned patches remain together here:

https://gist.github.com/sylvanc/088fbd867e544eb0a5a73044f2bc8b11

Stable 1.0.71 exact raw patch

https://gist.githubusercontent.com/sylvanc/088fbd867e544eb0a5a73044f2bc8b11/raw/a8f089368671e6ed11ebf7e314d407b2abf7ab14/copilot-cli-1.0.71-wsl-voice-final-prettified.patch

SHA-256: 2528122b5a6da8c921b167f34a306d8e976636d1c5143b8f015b598499b99628

Readiness cue

While Windows/WSLg is still waking the microphone, the meter displays Mic warming…
The UI changes to green Speak now only after the first real PCM envelope arrives from the voice worker.
This lets users with slow cold-start endpoints avoid losing initial speech without keeping a pre-roll consumer active.
The transition is based on the existing sessionAudio event, not merely on model/session initialization.

Retained behavior

Preserves final stable 1.0.71 behavior, including /voice devices, selected-device persistence, settings validation, terminal setup, mouse support, and configurable Ctrl+R history.
Retains first-Space speculative PTT, quick-Space typing, opening PCM buffering, adaptive tail, RNNoise/SpeexDSP processing, whisper leveling, PCM guards, selector voice, release-to-submit, typed prefix/suffix preservation, and agent-only ASR context.
Retains persistent /voice keeper on|off|status; COPILOT_VOICE_WSL_KEEPER is only the default when no saved preference exists.
Personalized speaker gating remains opt-in through COPILOT_VOICE_SPEAKER_FILTER=1.

Validation

Cold built-in-microphone live test rendered Mic warming before Speak now.
Source bundles pass syntax checks.
The published patch replays from the pristine formatted stable package and reproduces the installed files byte-for-byte.
Existing PTT, keeper lifecycle/RPC, slash-command persistence, device picker, and DSP smoke tests remain passing.

Normalize before applying:

for _ in 1 2 3; do
  npx --yes prettier@3.6.2 --write app.js voice-server.js >/dev/null
done

Then apply the patch and install dependencies:

npm install --no-save --package-lock=false --ignore-scripts --no-audit --no-fund \
  @echogarden/rnnoise-wasm@0.2.0 \
  @echogarden/speex-resampler-wasm@0.3.0 \
  sherpa-onnx-node@1.13.4

The SpeakerNet model remains a separate download because its model license must be reviewed independently.

Rebased for Copilot CLI 1.0.72-0.

Versioned patches remain together here:

https://gist.github.com/sylvanc/088fbd867e544eb0a5a73044f2bc8b11

1.0.72-0 exact raw patch

https://gist.githubusercontent.com/sylvanc/088fbd867e544eb0a5a73044f2bc8b11/raw/c350ea7c2f4bb423450b43a67b35fb46a2720594/copilot-cli-1.0.72-0-wsl-voice-final-prettified.patch

SHA-256: 34369caa3d93bab184faf0fb21dde0563a8cd97de87007dc9719369b46699e67

The 1.0.72 upstream changes are unrelated to voice. The rebased app delta retains the validated 1.0.71 behavior with exactly the same 38-hunk, 262-addition, 45-deletion semantic structure, remapped to the current 1.0.72 identifiers. The unchanged voice worker and PvRecorder portions remain byte-identical to the validated stable implementation.

Retained behavior includes first-Space speculative PTT, quick-Space typing, opening PCM buffering, adaptive tail, RNNoise/SpeexDSP processing, selector voice, choice-only prompt gating, typed prefix/suffix preservation, /voice devices, persistent /voice keeper on|off|status, and the first-real-PCM Mic warming… → green Speak now readiness cue.

Validation

Source syntax and compatible-Node CLI startup/help checks pass.
The patch applies from repeatedly Prettier-normalized pristine 1.0.72-0 bundles and reproduces all seven source files byte-for-byte.
Installed DSP and speaker-filter fail-open smoke tests pass.
An isolated installed voice server creates its RPC socket and exits cleanly with the keeper disabled.
A fresh independent review found no blockers and returned SHIPPING-READY.
The raw gist bytes were downloaded and matched against the published SHA-256.

The patch header contains normalization, dependency-install, and separately licensed SpeakerNet download instructions.

خودم هستم بلد نیستم درست کار کنم با این برنامه و این که تو این چند روز از کیف پول های من پرداخت شده بی اجازه لطفا این برداشت های بی اجازه رو ردیابی کنید نه این که گوشی من و ۲ کوشی از خانوادمو ردیابی کنید در تاریخ جمعه ۳ ژوئیه ۲۰۲۶،‏ ۲۱:۲۰ Sylvan Clebsch ***@***.***> نوشت:

…

*sylvanc* created an issue (github/copilot-cli#4024) <#4024> Describe the bug /voice records audio successfully (level meter reacts, mic capture confirmed via raw PulseAudio capture) but *every transcription comes back empty*, for *all three* models offered in the /voice model picker: - nemotron-3.5-asr-streaming-0.6b - nemotron-speech-streaming-en-0.6b - nemotron-speech-streaming-es-0.6b All three share the same nemotron_speech (RNNT) architecture, so switching models does not help — there is no working model in the current picker. Root cause (traced to source) The Foundry Local Core native audio transcription path throws on every call: Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: MultiModalProcessor cannot be created. nemotron_speech is not a registered multi-modal model type. at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) at Microsoft.ML.OnnxRuntimeGenAI.MultiModalProcessor..ctor(Model) at Microsoft.AI.Foundry.Local.AudioClient.Transcribe(String, String, Nullable`1) This is *not* a stale/outdated onnxruntime-genai engine issue. I downloaded and compared the public onnxruntime-genai v0.14.0 release (published 2026-05-29, after the "Multilingual Streaming Nemotron ASR" PR merged 2026-05-22) against the bundled runtime (labeled 0.14.1 in deps_versions.json, but not a publicly-tagged release). Both contain full, identical NemotronSpeechModel/NemotronSpeechState support (confirmed via symbol inspection of libonnxruntime-genai.so). Cross-checking against microsoft/onnxruntime-genai's src/models/model.cpp ( https://github.com/microsoft/onnxruntime-genai/blob/main/src/models/model.cpp) confirms this is architecturally expected: - nemotron_speech is an *RNNT-type* model (ModelType::IsRNNT) and is correctly routed to NemotronSpeechModel when the model is *loaded* ( CreateModel() in model.cpp) — this is why Model loaded successfully: nemotron-3.5-asr-streaming-0.6b-generic-cpu:3 appears in the Foundry log and the model shows as loaded. - However, MultiModalProcessor's constructor uses a *separate, hardcoded processor_factory_ registry* that only contains vision+text chat model types (e.g. phi4mm, gemma4, etc.) — RNNT/TDT/ALM (Whisper) audio model types were never meant to go through MultiModalProcessor at all. So the bug is that *Microsoft.AI.Foundry.Local.Core's AudioClient.Transcribe/streaming session path unconditionally constructs a MultiModalProcessor for audio transcription*, instead of dispatching RNNT/TDT/ALM model types (nemotron_speech, Parakeet TDT, Whisper) through their own dedicated processing path (the same one used successfully at model-load time). Since Microsoft.AI.Foundry.Local.Core.so is a closed-source component, this can't be worked around or patched client-side — no onnxruntime-genai version swap fixes it, since the engine itself already works correctly and simply refuses (correctly, by its own design) to construct a MultiModalProcessor for a non-multimodal-chat model type. Affected version GitHub Copilot CLI 1.0.69-0 (also reproduced on 1.0.66-1 — same bundled runtime pins: onnxruntime-genai 0.14.1, foundry-local-core 1.2.3, so the bug is not CLI-version-specific). Steps to reproduce the behavior 1. On Linux x64 (reproduced under WSL2/WSLg, but likely affects any Linux x64 install), run /voice, enable it, select any of the three offered models. 2. Press the voice-record shortcut, speak clearly for several seconds (confirmed via raw parec capture that real speech is captured at healthy signal levels, and the in-app level meter visibly reacts). 3. Stop recording. Expected behavior Spoken audio is transcribed to text and inserted into the input box. Actual behavior No error is shown to the user; the recording UI behaves normally, but the transcript is always empty. ~/.github-copilot-cli/logs/foundry.core*.log shows: [ERR] Error executing audio_transcribe: Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: MultiModalProcessor cannot be created. nemotron_speech is not a registered multi-modal model type. ... Audio stream stop: session <id>, final flush text: '', full transcript: '' Additional context - Confirmed mic capture and PulseAudio pipeline are fully healthy (not a WSLg/audio-hardware issue) via direct raw parec -d RDPSource capture showing strong, correctly-timed speech signal. - Confirmed via a standalone script using the bundled foundry-local-sdk package directly (bypassing the Copilot CLI JS entirely) that audioClient.transcribe() throws the identical MultiModalProcessor/ nemotron_speech error against a real captured WAV file, ruling out any Copilot-CLI-specific bug in the JS layer. - Suggested fix: in Foundry Local Core's audio transcription path, dispatch by ModelType::IsRNNT/IsTDT/IsALM (mirroring CreateModel()'s dispatch in onnxruntime-genai's model.cpp) instead of unconditionally constructing a MultiModalProcessor. — Reply to this email directly, view it on GitHub <#4024?email_source=notifications&email_token=BMBZYCWUK4P774AS5BTZKJD5C7W73A5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBYGA2TIMJQGY4THJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BMBZYCVYBZVQZSZW3DYQ3WD5C7W73AVCNFSNUABFKJSXA33TNF2G64TZHM2TQNJYGYYDMNRUHNEXG43VMU5TIOBQGU2DCMBWHEZ2C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BMBZYCT27W6ASQ7DTB4VHKT5C7W73A5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBYGA2TIMJQGY4THJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/BMBZYCUCSNUOBKXMQ4RZ2RL5C7W73A5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBYGA2TIMJQGY4THJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

Rebased for Copilot CLI 1.0.72-1.

Versioned patches remain together here:

https://gist.github.com/sylvanc/088fbd867e544eb0a5a73044f2bc8b11

1.0.72-1 exact raw patch

https://gist.githubusercontent.com/sylvanc/088fbd867e544eb0a5a73044f2bc8b11/raw/8a532eedfd63bed6285f703654087ae02c534d9d/copilot-cli-1.0.72-1-wsl-voice-final-prettified.patch

SHA-256: a4fd2f9137748592befb66fcdfd42667e89e7575967d1dbd030f6528e7504acc

The 1.0.72-1 app bundle has broad generated-identifier churn, while pristine voice-server.js and PvRecorder remain unchanged. The voice behavior was mapped directly onto the pristine 1.0.72-1 owners rather than accepting a fuzzy mixed-identifier merge.

In addition to retaining first-Space speculative PTT, quick-Space typing, opening PCM buffering, adaptive tail, RNNoise/SpeexDSP, selector voice, choice-only gating, /voice devices, persistent /voice keeper on|off|status, typed main-composer prefix/suffix submission, and the first-real-PCM Mic warming… → green Speak now cue, this revision hardens two edge paths:

no-choice freeform prompts now retain typed prefixes when voice submits;
saved keeper state is synchronized when reusing a server and propagated through dedicated-server, direct-worker, and fallback boot paths.

Validation

PTT state-machine and delayed-Space integration checks pass, including speculative session replacement while preserving the newly armed state.
Source syntax, Node 24 CLI startup/help, DSP fail-open, package dry-run, and installed keeper RPC checks pass.
The patch replays from repeatedly Prettier-normalized pristine 1.0.72-1 and reproduces all seven source files byte-for-byte.
The installed pre-patch 1.0.72-1 singleton was retired so stale in-memory RPC code cannot be reused.
Downloaded raw gist bytes match the published SHA-256.

The patch header contains normalization, dependency-install, and separately licensed SpeakerNet download instructions.

Corrected the 1.0.72-1 WSL voice patch for a catalog-startup deadlock.

Updated exact patch:

https://gist.githubusercontent.com/sylvanc/088fbd867e544eb0a5a73044f2bc8b11/raw/ebb894006536ec2a7d223f0994867e62f889ed55/copilot-cli-1.0.72-1-wsl-voice-final-prettified.patch

SHA-256: 06832d869c7d65df133552148fcac012d89ecc728e20df9f2ea801779e9367e2

Root cause

The live engine remained in fetching-catalog for hours, but the Azure catalog and cached models were healthy. Foundry initialization stopped at its first Configuring services log line, before telemetry or HTTP. The trigger was the WSL keeper/PvRecorder startup path:

WSLg had failed to load module-rdp-source, so RDPSource was absent and parec --device=RDPSource exited immediately.
The keeper retried that failed process every second.
pv_recorder.node and Foundry native initialization also proved unable to initialize reliably in the same process: PvRecorder-first froze Foundry; Foundry-first froze PvRecorder.

Correction

WSL recording now uses direct parec and does not load pv_recorder.node; non-WSL behavior remains native PvRecorder.
Keeper startup is delayed until Foundry reaches a stable catalog/model state.
A recording session takes ownership of the existing keeper stream, preserving pre-roll and avoiding a close/reopen race.
Missing RDPSource is detected with bounded pactl; it reports no device/unsupported keeper and waits without spawning failed capture loops.
Saved keeper on/off preference is persisted even while the source is unavailable.
Pending reads are bounded, active-child shutdown/exit cleanup is explicit, and unexpected capture exit re-arms the keeper.

Validation

Standard installed singleton reaches ready and returns all three supported ASR variants.
With the currently missing RDPSource: device list is empty, keeper reports unsupported/inactive, microphone open returns no-device, and zero parec children exist after repeated retry intervals.
A mocked healthy-source test reuses one keeper stream, delivers the one-second pre-roll plus 160-sample frames, bounds pending reads, and recovers after unexpected child exit.
Non-WSL native PvRecorder loading still succeeds.
The updated patch replays from normalized pristine 1.0.72-1 and reproduces all seven source files byte-for-byte.

support file-based ASR for nemotron streaming model (#885)

Navigation Menu

Repository navigation

Voice mode: all bundled ASR models fail silently — MultiModalProcessor routing bug for nemotron_speech (RNNT) in Foundry Local Core #4024

Describe the bug

Root cause (traced to source)

Affected version

Steps to reproduce the behavior

Expected behavior

Actual behavior

Additional context

Important clarification: CLI uses streaming ASR

Reproduction environment

Findings

1. Cold `RDPSource` fails its first PvRecorder read

2. Disabling PulseAudio idle suspension is the wrong workaround

3. Correct WSLg clock/wake handling

4. CLI drops microphone frames while the STT session is opening

5. Commit drains only one frame and truncates trailing speech

Confirmed result

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Voice mode: all bundled ASR models fail silently — MultiModalProcessor routing bug for nemotron_speech (RNNT) in Foundry Local Core #4024

Description

Describe the bug

Root cause (traced to source)

Affected version

Steps to reproduce the behavior

Expected behavior

Actual behavior

Additional context

Activity

rui-ren commented on Jul 6, 2026

rui-ren commented on Jul 6, 2026

sylvanc commented on Jul 11, 2026

sylvanc commented on Jul 11, 2026

Important clarification: CLI uses streaming ASR

Reproduction environment

Findings

1. Cold RDPSource fails its first PvRecorder read

2. Disabling PulseAudio idle suspension is the wrong workaround

3. Correct WSLg clock/wake handling

4. CLI drops microphone frames while the STT session is opening

5. Commit drains only one frame and truncates trailing speech

Confirmed result

rui-ren commented on Jul 11, 2026

sylvanc commented on Jul 11, 2026

rui-ren commented on Jul 11, 2026

sylvanc commented on Jul 12, 2026

sylvanc commented on Jul 16, 2026

sylvanc commented on Jul 16, 2026

Tak2121 commented on Jul 16, 2026

sylvanc commented on Jul 17, 2026

sylvanc commented on Jul 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. Cold `RDPSource` fails its first PvRecorder read