Jarvis AI Agent: Self-Hosted Linux Automation with Multi-LLM Support

What changed

Jarvis is presented as a self-hosted, autonomous AI agent designed for Linux environments. It aims to plan and execute tasks using natural language commands. Key functionalities include controlling a Linux desktop through VNC for live monitoring or intervention, integrating with WhatsApp for task delegation via text or voice messages, and maintaining a RAG-based knowledge base for context-aware responses. The agent supports a multi-LLM architecture, allowing seamless switching between various AI providers such as Google Gemini, Anthropic Claude, OpenRouter, local Ollama models, and any OpenAI-compatible endpoint. It supports both native tool/function calling and prompt-based tool usage, extending its capabilities to models without direct tool support. Jarvis also features a multi-agent system where the main agent can spawn autonomous sub-agents for parallel processing and offers role delegation to named agents with specific system prompts and tool subsets. For user interaction, it includes multi-user chat capabilities with multimedia attachments (images, audio, video, PDFs) that are processed for LLM context. The knowledge base employs a hybrid RAG approach, combining semantic search with BM25 lexical search for improved accuracy, and supports incremental re-indexing. A modular skill system allows for easy extension of Jarvis's capabilities, compatible with OpenClaw skills. The agent includes vision and face recognition capabilities, and a robust security layer with sandboxed execution for network users, prompt-injection detection, automatic account lockout, and per-user private /tmp directories. Authentication options include Active Directory/LDAP and 2FA/TOTP. It also offers integrations with Google Workspace and browser automation via CDP. Jarvis includes a self-improvement mechanism where user feedback triggers LLM analysis and generation of better alternatives, and a "Cognitive Evolution Skill" for self-extension and code patching.

Why it matters for builders

Jarvis provides developers with a comprehensive framework for building sophisticated AI-driven automation on Linux. The ability to self-host offers greater control over data and infrastructure, crucial for sensitive applications. Its multi-LLM support allows flexibility in choosing the most cost-effective or performant models for specific tasks, including fully offline local models. The multi-agent system and role delegation enable the creation of complex, parallelized workflows and the implementation of fine-grained access control for different user roles. The integrated RAG system with hybrid search and multimedia handling means builders can easily equip the agent with custom knowledge and context. The emphasis on security, with sandboxing and prompt-injection defenses, is vital for deploying agents in shared or team environments.

Practical impact

Builders can leverage Jarvis to automate a wide range of desktop tasks on Linux, from file management and code execution to interacting with applications and generating documents (Word, Excel, PDF). The WhatsApp integration offers a convenient way to delegate tasks remotely. The multi-LLM support means developers can experiment with different models, including open-source options via Ollama, for cost savings or specific performance needs. The sandboxed execution and security features allow for safer deployment in multi-user scenarios. Developers can also extend Jarvis's functionality by creating custom skills or integrating it with existing systems via its API. The self-improvement features suggest potential for agents that can adapt and learn over time.

Caveats and source limits

The provided source is a GitHub repository README, which details the features and architecture of the Jarvis AI agent. Specific details regarding performance benchmarks, exact pricing for any integrated services (beyond mentioning support for various LLM APIs), or a definitive release date are not present. The GitHub metrics (29 stars, 10 forks) indicate early community adoption. The project is described as open-source under the Apache 2.0 license. The source does not include information on the ease of installation for beginners or detailed system requirements beyond it running on Linux.