OpenGUI: Android GUI Agent Framework Released

What changed

The Core-Mate organization has released OpenGUI, a framework designed for AI agents to interact with Android graphical user interfaces (GUIs). This framework allows AI agents to perceive, strategize, and execute actions within real mobile applications on Android devices. The latest version, v0.1.1, is available, with recent updates including enhanced remote control capabilities via a local REST API and a new Skill for dispatching Android tasks from coding agents. Troubleshooting guides for common issues like backend connections and Android permissions have also been added, alongside Discord integration for remote task dispatch and hardened local startup procedures to mitigate port conflicts.

OpenGUI integrates with platforms like DeepSeek Harness and WorkBuddy. For DeepSeek Harness, a streamlined installation process is available on macOS using a prompt that installs the OpenGUI plugin without requiring a full backend deployment. This process supports specific DSH versions (0.1.0-rc.7 through 0.1.1-rc.2) and preserves existing DSH configurations. For WorkBuddy, version 0.3.1 is a public testing prerelease that installs the MCP, Skill, and Hooks together, requiring WorkBuddy 5.5.6+ for live installation.

The full OpenGUI stack can be bootstrapped using coding agents like Claude Code or Codex, which automate the setup of the local backend, Android client, and ADB configurations. This process requires an Android 11 (API 30) or newer device with USB debugging, AccessibilityService, overlay permission, and battery optimization exemption enabled. Model API keys are necessary for task execution, with recommendations for vision-capable models like Doubao VLM, Qwen VLM, and OpenAI's models.

Why it matters for builders

OpenGUI provides AI developers with a robust tool to bridge the gap between AI models and the Android mobile ecosystem. Builders can now create agents that can autonomously navigate and operate mobile apps, opening up new possibilities for automation, user experience testing, and the development of intelligent mobile assistants. The framework's integration with popular AI development platforms simplifies adoption and deployment.

Practical impact

Developers can integrate OpenGUI into their AI agent workflows for tasks such as automated UI operation and regression testing on authorized devices, social media management with human confirmation, and repetitive game testing. The framework supports various vision models, allowing builders to select the most suitable one for their specific use case based on performance and cost. The availability of different integration methods, including DeepSeek Harness, WorkBuddy, and a full stack setup, offers flexibility for diverse development environments.

Caveats and source limits

The source material indicates that version 0.3.1 for WorkBuddy is a macOS public-testing prerelease and not a full marketplace approval. It also notes that the current source implementation admits only one OpenGUI task per DSH session and that the managed browser remains globally serial, which is not presented as a release claim. Specific pricing for model API keys is not provided, and the performance of Grok vision-capable models for this workflow is still considered experimental. The source also mentions that permission names and menu locations can vary across Android vendors, requiring users to consult a complete Android permission setup guide.