What It Takes to Run Hugging Face speech-to-speech Fully Locally¶

Audience / Adoption Decision

Audience: AI application developers comparing local or self-managed voice-agent infrastructure

A good fit	Consider another approach
You want to replace speech recognition, the LLM, or speech synthesis independently	You need one audio-native model to preserve prosody and overlapping speech
You want to reuse a Realtime API client	You require every OpenAI Realtime event to behave identically
You want to control audio data and inference endpoints	You expect production-quality Japanese with no configuration work

Hugging Face's huggingface/speech-to-speech is an Apache-2.0 voice-agent runtime. Version 0.2.11 reached PyPI on July 17, 2026, and the package is still classified as Alpha.¹²

The name sounds like a single model that converts speech directly into speech. It is not. It is a cascaded pipeline with four swappable stages: VAD, STT, LLM, and TTS.

The practical question is what can run locally and when this architecture is the right choice.

A four-stage VAD, STT, LLM, and TTS pipeline with a switchable cloud or local LLM endpoint

The four replacement points are the product¶

speech-to-speech sends incoming audio through four processing stages. Each stage runs in its own thread and passes results to the next stage through queues.¹

Stage	Default job	What replacement changes
VAD	Silero VAD v5 detects speech start and stop	Silence and interruption sensitivity
STT	Parakeet TDT converts speech to text	Language coverage, accuracy, and compute
LLM	An OpenAI-compatible API generates replies and tool calls	Intelligence, latency, and data destination
TTS	Qwen3-TTS converts text back to audio	Voice, language, and synthesis speed

In this project, "speech-to-speech" names the runtime rather than one model. Unlike an end-to-end audio model, the LLM receives the text produced by STT, so it does not directly process prosody or pauses.

The tradeoff is modularity. A team can optimize English recognition, move only the LLM into its network, or replace only the Japanese voice. The architecture lets a team replace the bottleneck instead of accepting one model's fixed ceiling.

The default launch still sends text to a cloud LLM¶

The shortest quickstart is not fully local. STT and TTS run locally, but the default LLM backend uses the OpenAI Responses API and asks for an API key. The current repository configuration names gpt-5.4-mini as the default model.¹³

pip install speech-to-speech
export OPENAI_API_KEY=...
speech-to-speech

Raw speech recognition and synthesis stay on the machine in this setup, but the transcript goes to the LLM provider. "The audio file stays local" and "the conversation stays local" are different privacy claims.

To keep every stage local, run the LLM behind llama.cpp or vLLM and point responses_api_base_url at that server. The current official example serves Gemma 4 through llama.cpp and connects with an empty API key.¹⁵

speech-to-speech \
  --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
  --responses_api_base_url "http://127.0.0.1:8080/v1" \
  --responses_api_api_key ""

The repository's Apache-2.0 license does not automatically make every model, voice, and inference server use the same terms. A production review still needs the license and data destination for each stage.

Realtime API compatibility separates clients from inference¶

The runtime also exposes a WebSocket endpoint at /v1/realtime. A client sends audio chunks and events such as session.update; the server returns transcripts, audio deltas, tool calls, and completion events.⁴

This allows a Realtime API client to remain largely unchanged while the backend moves from a hosted endpoint to self-hosted models. Hugging Face's Reachy Mini setup follows this pattern by changing the connection URL to a local speech-to-speech server.⁵

The important word is "compatible," not "identical." The implementation documents a core event set, not a guarantee that every OpenAI event and future protocol change is covered. Teams should test the events, audio formats, and tool-result flow their client actually uses.

Streaming and cancellation create the low-latency behavior¶

A cascade accumulates delay across all four stages. The official README identifies LLM inference as the most compute-intensive and highest-latency part of the pipeline.¹

speech-to-speech streams LLM text and TTS audio so playback can begin before the full answer is complete. When a user interrupts, VAD reports new speech and a shared CancelScope invalidates stale LLM and TTS output before flushing the relevant queues.⁴

That makes the project more than glue around four model calls. The hard part of a voice loop is stopping an obsolete reply and moving conversation state to the new utterance without replaying stale audio.

The repository does not guarantee one end-to-end latency across hardware and model combinations. For this article, SmartScope reviewed the source and configuration statically; it did not download the models, record live audio, or run a latency benchmark. An adoption test should measure time from the end of speech to the first returned audio on the target hardware.

A Japanese setup needs a different STT backend¶

The default Parakeet TDT 0.6B v3 covers 25 European languages and does not include Japanese.⁶ For Japanese input, the practical starting point is to replace it with one of the Whisper backends already supported by the repository.

Whisper large-v3 is multilingual, and its Hugging Face model card lists 99 languages.⁷ The default Qwen3-TTS model supports ten languages, including Japanese.⁸

speech-to-speech \
  --stt whisper \
  --stt_model_name openai/whisper-large-v3 \
  --language ja \
  --qwen3_tts_language ja

Those flags do not prove production quality. Names, accents, background noise, overlapping speech, and TTS pronunciation vary by use case. An STT error reaches the LLM, and the resulting response then reaches TTS, so small stage-level errors can compound across the conversation.

Adoption should follow the boundary you need to replace¶

SmartScope's assessment is narrower than "use this if you want local voice AI." The project fits teams that can decide which of STT, LLM, and TTS they want to own and where each stage should run.

If preserving audio-native prosody is the primary requirement, compare end-to-end speech models first. If zero-configuration Japanese quality or large-scale concurrency is the priority, compare the operating cost of an Alpha-stage runtime with a managed voice API.

The design is a strong fit when a team wants to keep an existing Realtime client and replace speech recognition, the LLM, or speech synthesis independently. Fully local operation is not a product toggle. It is the result of choosing a local execution target for all four stages.