GetStream Releases Open Vision Agents v0.6.8

What changed

GetStream has released version 0.6.8 of its open-source Vision Agents project, a framework designed for building multi-modal AI agents that can process video, audio, and text. The project emphasizes real-time capabilities and aims to provide developers with the building blocks for intelligent video experiences. Key highlights include support for real-time video AI, combining object detection models like YOLO with LLMs such as Gemini and OpenAI. The system is engineered for low latency, with quick join times and audio/video latency under 30ms, facilitated by Stream's edge network. The framework is open and designed to work with any video edge network.

Vision Agents offer native SDK methods for interacting with LLMs from OpenAI, Gemini, and Claude, ensuring access to the latest model capabilities. The project includes SDKs for popular platforms like React, Android, iOS, Flutter, React Native, and Unity. Installation is managed via uv, with an option to include extra integrations for services like GetStream, OpenAI, ElevenLabs, and Deepgram. Developers can obtain Stream API credentials, with a free tier offering 333,000 participant minutes per month and additional credits available through the Maker Program.

The framework supports a pluggable processor pipeline for video analysis, allowing integration of models like YOLO, Roboflow, or custom PyTorch/ONNX models before or after LLM calls. It also features turn detection for natural conversation flow, tool calling for API execution, and phone integration via Twilio or Telnyx. Retrieval-augmented generation (RAG) is supported with vector search engines like TurboPuffer and Qdrant, or Gemini FileSearch. Agents can maintain context across sessions through Stream Chat memory. Additional features include a text back-channel for silent communication with the agent and production-ready components such as an HTTP server, Prometheus metrics, and Kubernetes deployment support.

Vision Agents boast out-of-the-box integrations with a wide array of LLMs (OpenAI, Gemini, xAI, OpenRouter, Hugging Face, Kimi AI, MiniMax, Telnyx), real-time models (OpenAI Realtime, Gemini Live, AWS Nova Sonic, Qwen, Inworld), STT services (Deepgram, AssemblyAI, Fast-Whisper, etc.), TTS services (ElevenLabs, Cartesia, Deepgram, etc.), and vision models (Ultralytics, Roboflow, Moondream, TwelveLabs, NVIDIA, Decart). It also integrates with avatar solutions, turn detection services, and other platforms like Twilio and Telnyx.

Why it matters for builders

This release empowers developers to construct sophisticated, real-time multi-modal AI agents with greater ease. The framework's modular design and extensive integrations allow for rapid prototyping and deployment of applications that require simultaneous understanding of video, audio, and text. Builders can leverage this to create interactive coaching systems, advanced monitoring tools, or dynamic customer service interfaces.

Practical impact

Developers can begin by installing Vision Agents using uv add vision-agents and optionally adding specific integrations. The project provides a quickstart guide for building a first agent, with examples demonstrating use cases like AI golf coaching using YOLO and Gemini Live. The availability of SDKs across multiple platforms facilitates integration into existing or new applications. Builders can explore the comprehensive documentation at VisionAgents.ai for detailed guides on video processors, RAG, phone calling, and production deployment.

Caveats and source limits

The source material indicates current limitations for Vision Agents, including potential difficulties with video AI accurately interpreting small text, context degradation in longer video sessions (over 30 seconds), the necessity of combining specialized models with larger LLMs for most use cases, and the requirement for audio or text input to trigger agent responses from video alone. The release date of v0.6.8 is not explicitly stated, but the project is marked as a 'fresh release' with a published date of August 7, 2026, in the metadata. Pricing details for the Stream API beyond the free tier are not provided.