Building a Reliable Real-Time Audio Pipeline
Project: VoiceConnect — AI Calling Agent
Area: Speech-to-Text, Text-to-Speech, WebSockets, asynchronous processing, concurrency, connection recovery
The Challenge
While developing VoiceConnect, an AI-powered calling agent, I worked on the audio layer responsible for receiving a caller's speech, converting it into text, passing it to the AI agent, and delivering the response through speech synthesis.
Initially, the integration appeared straightforward: receive audio, send it to a speech recognition service, process the resulting text, and play the AI-generated response.
However, making this work reliably during a conversation turned out to be much more challenging. Speech recognition produces multiple events, audio processing happens asynchronously, and network connections can fail while a conversation is in progress.
I had to investigate how speech completion actually works, how to prevent the agent's own voice from interfering with recognition, and how to maintain a reliable connection throughout a conversation.
Understanding Speech Completion
One of my first challenges was determining when a caller had actually finished speaking.
I initially needed to understand the difference between a finalized transcript segment and a completed spoken utterance. They are not necessarily the same thing.
Deepgram can return multiple finalized transcript segments while a caller is speaking. Processing each segment immediately could result in incomplete questions being sent to the AI agent.
I researched Deepgram's endpointing configuration and itsspeech_final and UtteranceEnd events to understand how the service determines when speech has ended.
Based on this investigation, I implemented transcript buffering. Finalized segments are accumulated until an utterance-completion event triggers processing. I also handled cases where both completion events arrive for the same utterance, preventing duplicate processing.
This required understanding the order in which events could arrive and accounting for missing or delayed completion events.
Handling the Agent's Own Voice During Local Testing
Another challenge emerged when testing the audio pipeline locally. I used my laptop's microphone and speakers to simulate a conversation. However, Deepgram was also picking up the voice generated by Murf when the agent played its response.
This created an unwanted feedback problem: the speech recognition service could interpret the agent's own response as new caller input.
To address this during local testing, I deliberately adopted a half-duplex communication approach. The system listens to the caller, pauses microphone audio transmission while the agent processes the request and plays its response, and resumes listening after playback finishes.
This required coordinating microphone callbacks, audio queues, and background sender threads. I implemented synchronized mute and unmute operations and queue management to prevent audio captured before listening resumed from being transmitted as new input.
I also changed the startup sequence so that speech recognition and microphone initialization happen while the greeting is playing. Caller audio transmission is enabled only after the greeting finishes and the audio components are ready.
Separating Local Testing from Production Architecture
The half-duplex approach was a deliberate local testing solution, not a requirement intended for the production calling architecture.
With a laptop microphone and speakers, the agent's voice can travel back into the microphone. This made it necessary to pause audio transmission while testing locally.
For production, the intended architecture uses Exotel as the telephony layer. This provides a different way to handle caller audio and agent playback without relying on the laptop microphone and speaker arrangement used during development.
I documented this distinction and the local testing limitation in the project's README, along with the recommendation to change the audio handling approach when deploying in the production telephony environment.
Investigating Connection Timeouts
During live testing, I encountered another unexpected problem: the Deepgram WebSocket connection closed while the agent was generating and playing a longer response.
The problem was related to the communication lifecycle. While the agent was speaking, caller audio transmission was paused. As a result, Deepgram could receive no audio for an extended period and eventually close the idle connection.
I investigated Deepgram's connection behavior and the installed SDK's capabilities. This led me to implement a KeepAlive heartbeat that periodically sends a small message to keep the connection active even when audio is temporarily paused.
I also implemented bounded reconnection attempts with increasing delays between retries, allowing temporary connection failures to be handled without repeatedly attempting to reconnect without limits.
Solving a Connection Recovery Race Condition
While reviewing the recovery implementation, I discovered a more subtle concurrency problem.
When an old Deepgram connection failed, its receiver thread could still be shutting down while a replacement connection was already established.
A delayed event from the old connection could then incorrectly change the state of the new connection, making a successfully recovered session appear disconnected.
I addressed this by associating event handlers and receiver threads with the specific connection that created them. Before modifying shared connection state or triggering recovery, the system checks whether the event belongs to the currently active connection. Events from obsolete connections are ignored.
I also added a bounded wait for the old receiver thread during reconnection to help release resources without risking an indefinite wait.
Testing and Reliability
Because these problems involved timing and asynchronous events, testing only successful conversations was not sufficient.
I added automated tests using mocked connections and services to reproduce situations such as delayed speech-completion events, audio arriving during mute transitions, connection timeouts, reconnection attempts, and delayed events from obsolete connections.
All automated tests passed using mocked services, without making live API requests or consuming external service credits. the latency is also significantly decreased.
Engineering Principle
This challenge taught me that integrating real-time services is not just about connecting APIs. Correctness also depends on understanding event ordering, managing shared state across threads, handling network failures, and designing reliable recovery mechanisms.
More importantly, I learned not to assume that an event means exactly what its name suggests. I had to investigate the service's actual behavior, reproduce failure scenarios, and revise my implementation based on what I discovered.
I also learned to distinguish between a solution that makes local testing reliable and an architecture that is appropriate for production deployment. Those two environments do not always have the same constraints.