Building a Real-Time AI Voice Assistant with Whisper and TTS

Voice interfaces offer the most natural way to interact with artificial intelligence. By combining OpenAI's **Whisper** for ultra-accurate speech-to-text (STT) transcription, a language model for reasoning, and a high-performance **Text-to-Speech (TTS)** engine, you can build a fully localized, real-time voice assistant in Python.

This comprehensive guide walks through capturing microphone audio, transcribing speech with Whisper, generating intelligent conversational responses, and synthesizing realistic voice output.

1. Environment Setup and Audio Dependencies

Create a virtual environment and install `openai-whisper`, `sounddevice`, `pyttsx3` (or gTTS/Bark), and supporting audio libraries.

              
C

            # Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate

# Install Whisper, PyTorch, audio recording, and TTS dependencies
pip install torch torchvision torchaudio
pip install openai-whisper sounddevice scipy numpy pyttsx3 openai

2. Recording Microphone Audio and Transcribing with Whisper

Record user voice input from the microphone as a temporary WAV file and transcribe it using OpenAI's local Whisper model.

              
            
C

            
            import sounddevice as sd
from scipy.io.wavfile import write
import whisper

# Load local Whisper model (e.g., 'base' or 'small')
model = whisper.load_model("base")

def record_and_transcribe(duration: int = 5, fs: int = 16000):
    print("Listening...")
    audio = sd.rec(int(duration * fs), samplerate=fs, channels=1, dtype='int16')
    sd.wait()
    print("Processing audio...")
    
    # Save temporary recording
    file_path = "output.wav"
    write(file_path, fs, audio)
    
    # Transcribe speech
    result = model.transcribe(file_path)
    return result["text"]

# Test transcription
# user_text = record_and_transcribe()
# print(f"User said: {user_text}")
          

3. Generating Conversational Responses with an LLM

Pass the transcribed text user query to your LLM of choice to generate a natural, context-aware conversational response.

              
            
C

            
            from openai import OpenAI

client = OpenAI()

def get_assistant_response(prompt_text: str):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful, concise voice assistant. Keep answers short and conversational."},
            {"role": "user", "content": prompt_text}
        ]
    )
    return response.choices[0].message.content
          

4. Synthesizing Speech Output with TTS

Convert the LLM's text response into spoken audio using a Text-to-Speech engine so the assistant can talk back.

              
            
C

            
            import pyttsx3

def speak_response(text: str):
    engine = pyttsx3.init()
    # Configure voice properties (speed, volume)
    engine.setProperty('rate', 175)
    engine.setProperty('volume', 1.0)
    
    engine.say(text)
    engine.runAndWait()

# Full voice loop pipeline test
if __name__ == "__main__":
    # 1. Listen & Transcribe
    user_input = record_and_transcribe(duration=4)
    print(f"User: {user_input}")
    
    if user_input.strip():
        # 2. Generate LLM Reply
        reply = get_assistant_response(user_input)
        print(f"Assistant: {reply}")
        
        # 3. Speak Reply
        speak_response(reply)
          

Conclusion

By integrating **Whisper** for robust speech recognition, an **LLM** for core reasoning, and a **TTS engine** for voice synthesis, you have built a powerful, real-time interactive AI voice assistant.