Repository navigation
Conversation
…e count The decoder graph capacity doubled from patch_size until the request fit and the whole capacity was computed, so a 5.6 s phrase (140 latent frames) decoded a 256-frame (10.24 s) graph. The decoder is causal, so the padding never changes the kept samples; it only costs compute. Without voxcpm2.audiovae_latent_capacity the graph is now built at the request's frame count (rounded to patch_size) and the allocator is kept across rebuilds. Setting the option keeps the old reservation behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing windows off the generator thread Streaming decoded every window inside the generator's chunk callback, on the session backend, so generation stopped while the AudioVAE ran. On a 12-core Zen 5 CPU the decoder is ~40% of a request (offline 5.6 s of 20.7 s for 18.7 s of audio), and the generator scales poorly past ~6 threads (OpenMP barrier waits are ~20% of a perf profile), so the two serialized stages left cores idle. New session option voxcpm2.audiovae_threads (CPU only, default 0 = unchanged): the AudioVAE gets its own CPU ExecutionContext with that many threads, and streaming requests hand windows to a worker thread in patch order. Only the worker touches the StreamingAudioController and the event sink, and it is joined before the request returns, so events stay sequential. A failing sink (client gone) stops generation at the next patch and the error is rethrown. Output is bit-identical to the shared-backend path (offline, streaming, clone). Also documents voxcpm2.audiovae_latent_capacity's new meaning (previous commit). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Snake (x + sin(a*x)^2 / (a + 1e-9)) repeated alpha to the activation's full [1, C, T] size, ran scale_bias over that full tensor, then five more full-size passes; the sample-rate condition repeated its scale and bias the same way. ggml_mul / ggml_div broadcast [1, C, 1] directly, so the repeats go, and on the CPU backend Snake is one ggml_map_custom2 pass (the same float operations in the same order, rows split across the graph's threads; the pattern kokoro_tts already uses). Other backends keep the graph form. Output is bit-identical (offline, 3 and 6 sentences). AudioVAE decode on a Ryzen 9 9900X, 12 threads, 17.4 s of audio: 5.41 s -> 3.61 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cache ultimate_clone=true uses the voice reference itself as the continuation prompt, with reference_text as its transcript - upstream VoxCPM2's prompt_wav_path + prompt_text together with reference_wav_path. Server voice presets can carry it, so OpenAI-shaped clients get it with voice=<name>. Without it a reference_text sent with voice_ref has no prompt audio and is unused (reference_text is an alias of prompt_text). Output is identical to passing the reference again as audio_input with prompt_text. voxcpm2.prompt_cache_dir (session option) keeps encoded prompts on disk. The file name is a hash of the clip(s), transcript and AudioVAE storage type, so a changed input is a miss and requests cannot name a path; an unreadable file is encoded again and replaced (written to .tmp, then renamed). Ultimate clone encoding is ~3.2 s on a Ryzen 9 9900X; a disk hit is ~1 ms, same audio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
@ersensari I'm not sure
For this part, we only need a documentation fix to correct the example and clarify the roles of these inputs. |
Owner
|
@ersensari #981 has been merged. Please rebase the PR while keeping the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stacked on #981. This branch contains #981's three commits; only the last commit,
feat(voxcpm2): ultimate_clone request option, on-disk encoded prompt cache, is new here. It touchessession.cppnext to #981's streaming changes, so it does not apply tomainon its own. Once #981 is merged, I'll rebase this branch.ultimate_clonerequest option. Withultimate_clone=true, the voice reference itself is used as the continuation prompt, andreference_textis its transcript. This is upstream VoxCPM2'sprompt_wav_path+prompt_textused together withreference_wav_path.voice=<name>.reference_textsent withvoice_refhas no prompt audio and is silently unused, becausereference_textis an alias ofprompt_text. Callers believed they were giving the model the transcript, but they were running a reference-only (controllable) clone.audio_inputtogether withprompt_text.ultimate_clone=trueand no transcript, the request fails with a clear error.voxcpm2.prompt_cache_dirsession option. Encoded prompts are kept on disk..tmp, then renamed.Why: a voice assistant sends one reply as several requests, one per sentence or group of sentences. With a fixed seed, each request still started from nothing, so voice and pace drifted between them. Using the ultimate clone (reference = prompt) keeps every request on the same voice. The cost is re-encoding the prompt for every request: ~3.2 s on CPU.
Validation
Same build as #981, CPU,
voxcpm2-q8_0.gguf, Ryzen 9 9900X, 12 threads. Re-run on this branch, rebased onmain@ ccdbe76:voxcpm2.prompt_encode_mssession.wall_msvoxcpm2-prompt-6a967775f30d3dea.bin, 88 KB)3bfcc3f0…3bfcc3f0…(identical)Also checked during development:
ultimate_clone=truewithout a transcript gives a clear error."request_options": {"ultimate_clone": "true"}andprompt_cache_diron a volume: after a restart, all 9 voices are warm again from disk.I compared plain clone, continuation and ultimate clone across multi-request replies by ear, and ultimate clone gave the most consistent voice from one request to the next.
On CUDA (RTX 5060 Ti, sm_120), this branch serves 9 preset voices in a self-hosted voice assistant (in use since 2026-10-19).
Known limitations
ultimate_clone, a(style)/ control instruction does not apply. As in upstream VoxCPM2, control instructions only work in controllable-clone mode, so a caller that wants a style should leaveultimate_cloneoff for that request.🤖 Generated with Claude Code