agent-device: Mobile App Automation for AI Agents

What changed

Callstack has introduced agent-device, a project designed to empower AI coding agents with the ability to automate and verify mobile applications. This tool provides a comprehensive interface for AI agents to interact with applications across a wide range of platforms, including iOS, Android, HarmonyOS, tvOS, Android TV, Amazon Vega OS TV, web, macOS, and Linux. The system offers three primary interaction methods: a Command Line Interface (CLI), a built-in Message Communication Protocol (MCP) server, and a typed Node.js API. These interfaces allow AI agents to not only inspect the state of a running application through accessibility snapshots and component trees but also to perform actions such as tapping, filling fields, and scrolling.

Key functionalities include the ability for agents to receive token-efficient accessibility snapshots rather than relying solely on screenshots, enabling more nuanced understanding of the UI. Agents can act on UI elements using references and selectors, and crucially, they can save evidence of their actions for later review. The system also handles coordination of device access, preventing conflicts when multiple agents are working in parallel, and supports connections to remote device clouds. The latest release, v0.20.9, signifies ongoing development and refinement of these capabilities. Installation requires Node.js version 22.12 or newer, with web automation needing Node.js 24 or newer.

Why it matters for builders

For AI builders and developers, agent-device offers a robust framework for integrating AI into the mobile app development lifecycle. It bridges the gap between AI's reasoning capabilities and the tangible execution and verification of code on live applications. This means AI agents can be tasked with implementing features, debugging issues, and verifying changes directly within the app environment, providing immediate feedback that can accelerate development cycles. The tool's support for various platforms and interaction methods makes it adaptable to diverse development setups and agent architectures.

Developers can use agent-device to build more sophisticated AI agents capable of end-to-end testing, automated QA, and even proactive bug detection. The ability to capture detailed evidence, such as screenshots, videos, logs, and performance data, ensures that AI-driven verification processes are transparent and reviewable. This level of automation and verification can significantly reduce manual testing efforts and improve the overall quality of applications.

Practical impact

Developers can begin using agent-device by installing the CLI via npm and running agent-device doctor to check their setup. The help workflow command provides access to guides for debugging, replay, and profiling. For direct interaction, the CLI allows users to open applications (e.g., agent-device open Contacts --platform ios), inspect the screen using agent-device snapshot -i, and perform actions like pressing buttons (agent-device press @e2 --settle) or filling text fields (agent-device fill @e7 "Ada" --settle). Evidence can be captured with agent-device screenshot ./contact-form.png before closing the session.

For programmatic control, the typed Node.js API, accessed via createAgentDeviceClient(), enables developers to script interactions directly within their Node.js applications or agent orchestrations. The MCP server (agent-device mcp) exposes these commands as structured tools, compatible with various AI agent frameworks like Claude Code, Codex, and Cursor. Developers can leverage these capabilities for tasks such as implementing onboarding screens, reproducing crashes with detailed logs, checking for performance regressions like unnecessary React Native re-renders, or verifying pull requests on physical devices. The tool can also generate replay scripts (.ad files) or export them as Maestro YAML for CI/CD integration.

Caveats and source limits

The provided source material details the capabilities and usage of agent-device extensively, but specific performance benchmarks or independent verification of its effectiveness across all supported platforms are not included. While the tool supports a wide array of targets, the depth of command support can vary, and users are advised to check agent-device capabilities --platform <target> for specific details. The source mentions that newer backends, such as HarmonyOS and Vega OS, currently cover a subset of commands. Pricing information for any potential enterprise features or support is also not detailed in the provided excerpts. The project is described as a "fresh release," indicating it is actively under development, and users should be aware of potential changes in future versions.