ElatoAI: Realtime Voice AI on ESP32 with 100+ Models

What changed

ElatoAI is an open-source project that facilitates the integration of real-time voice AI capabilities into devices powered by the Arduino ESP32 microcontroller. The system supports a vast array of over 100 speech-to-text (STT), large language model (LLM), and text-to-speech (TTS) models. It achieves this by leveraging secure WebSockets for communication between the ESP32 device and edge functions, which can be deployed on platforms like Deno Edge or Cloudflare Workers. This architecture allows for global, low-latency interactions, enabling uninterrupted conversations of up to 20 minutes. Recent updates include the ability to launch over 100 STT, LLM, TTS voice pipeline systems with a FastAPI server, and the introduction of local AI toys on ESP32 devices supporting frontier local LLMs and TTS models like Qwen and Mistral with MLX. Additionally, a feature was added to create a global devices/toys network using Cloudflare Voice Agents and Durable Objects, with Cloudflare Workers AI providing native Deepgram STT/TTS, requiring only an LLM API key for a scalable pipeline.

Key Components and Technologies:

• IoT Client: ESP32-S3 running on the Arduino Framework, utilizing PlatformIO. It handles audio capture and playback, communicating via secure WebSockets with Opus audio compression.
• Edge Functions: Deno Edge or Cloudflare Workers serve as the backend, managing LLM, TTS, and STT API calls. This layer supports over 80 LLM models, 10+ TTS models (including Deepgram, MeloTTS), and 5 STT models (including Whisper, Deepgram).
• Frontend Client: A Next.js application hosted on Vercel provides a user interface for creating and interacting with AI agents, and for managing ESP32 devices.
• Database: Supabase DB is used for storing conversation history and managing device authentication.

Why it matters for builders

ElatoAI provides a comprehensive framework for developers to embed advanced voice AI into resource-constrained hardware. This project lowers the barrier to entry for creating interactive AI-powered devices, moving beyond simple command-response systems to more natural, extended conversations. Builders can experiment with a wide range of AI models without needing powerful local hardware, as the heavy lifting is offloaded to edge functions. The project's modular design, with distinct frontend, edge, and IoT components, allows for flexibility in customization and deployment.

Practical impact

Developers can leverage ElatoAI to build a variety of applications, such as AI companions for children, interactive educational toys, or smart home devices with enhanced voice interaction. The project's quick start guides and build instructions for both PlatformIO and Arduino IDE simplify the initial setup. The ability to deploy globally using Cloudflare Workers and Deno Edge functions means applications can achieve low latency worldwide. Builders can explore creating custom AI agents with unique personalities and voices, and integrate features like tool calling for more complex interactions. The project also supports OTA updates for the ESP32 firmware, simplifying maintenance.

Caveats and source limits

The project notes a cold start time of 2-3 seconds when connecting to the edge server. While tested for up to 17 minutes of uninterrupted conversations, the edge server may stop if a wall clock time is exceeded. There is no speech interruption detection implemented on the ESP32 side. The project is licensed under the MIT License. The source does not provide specific pricing for API usage or details on independent benchmark performance for the various models integrated.