Why it matters
This project lowers the barrier for AI agents to interact with iPhones, enabling developers to build more sophisticated AI applications that leverage real-world mobile device capabilities. Builders can now explore AI-driven mobile automation and testing scenarios previously limited by complex setup requirements.

What changed

The ESP32 iPhone Agent project has introduced significant updates that enhance its ability to allow AI models to control an iPhone. A key new feature is the integration of audio processing, enabling the ESP32 board to forward AirPlay audio from the iPhone. This audio is then converted into text, allowing for listening exercises and voice prompts to function within the AI agent's workflow. The project also introduces a "Free Jev line" which optimizes the agent loop by processing plain steps rapidly and delegating complex tasks to the large language model, resulting in a single model call per step. Furthermore, the agent now supports "Long tasks," removing the previous 40-step limit and allowing any stopped run to be resumed. Each run can also be replayed step-by-step on the phone, providing a detailed trace of actions.

The project's core innovation lies in its approach to connecting AI models to iPhones. Unlike traditional methods that require a Mac, Xcode, developer mode, and USB pairing, the ESP32 iPhone Agent uses a $7 ESP32 board to act as the intermediary. This board handles the computer's role, simplifying the setup to flashing the board and installing an app via TestFlight. The agent utilizes AirPlay for screen mirroring, forwarding the H.264 stream to an app on the same iPhone. This app decodes the stream, sends screenshots to an OpenAI-compatible vision model, and then translates the model's output into actions like taps and swipes via USB HID (Human Interface Device) commands. The phone uses its cellular data for model communication, as the ESP32 board creates its own Wi-Fi hotspot.

Key Hardware and Setup Details:

  • Board: Tested with Seeed Studio XIAO ESP32S3 (plain version, 8 MB PSRAM). Any ESP32-S3 with native USB and PSRAM should be compatible.
  • iPhone: Requires iOS 18 or later with cellular data. USB-C iPhones connect directly; Lightning models need a Lightning to USB Camera Adapter. The phone powers the board.
  • Vision Model: Compatible with any OpenAI-compatible chat API that accepts images.
  • Setup: Involves flashing the ESP32 board (via browser or toolchain), installing the companion app (TestFlight or built from source), and a four-step in-app setup: Bluetooth pairing, Wi-Fi connection for AirPlay, AssistiveTouch calibration, and configuring the AI model endpoint and key.

Why it matters for builders

This project democratizes AI-driven iPhone automation by removing the need for expensive hardware or complex developer setups. Builders can now experiment with AI agents controlling iPhones using readily available hardware and an open-source framework. This opens up new avenues for developing AI-powered mobile testing, user experience research, and task automation that were previously inaccessible to many.

Practical impact

Developers can leverage the esp32agent CLI, which acts as an MCP server, to allow coding agents on their computers to control the iPhone. This integration enables AI coding assistants to interact with and test mobile applications directly. For those not wanting to flash a board, a ready-to-plug "Handsy" device is also planned. The project encourages community contribution through stars and forks, and provides detailed documentation for hardware, setup, and usage.

Caveats and source limits

The project is open-source and relies on community support and contributions. While the core functionality is available, some features like the one-tap hotspot join require a paid Apple developer account. The free model line used for some operations can experience busy or error states, suggesting users switch to their own API keys for consistent performance. The project is currently in active development, with the latest release being v1.0.6-build17, and the app is listed in the App Store review process for TestFlight. The phone uses cellular data for model communication, as the board occupies the Wi-Fi connection.

Sources

Written with AI assistance from the linked sources; every claim below was checked against them automatically. How we produce articles.

Claim check: 12/12 supported claims - 12 evidence links - 100% avg confidence
  • An AI agent can use a real iPhone with a $7 ESP32 microcontroller for control.supported - github.com
  • The ESP32 iPhone Agent enables AirPlay eyes and USB HID hands for AI interaction.supported - github.com
  • The system works with any OpenAI-compatible vision model.supported - github.com
  • No Mac, developer account, or jailbreak is required to use the ESP32 iPhone Agent.supported - github.com
  • The ESP32 board forwards AirPlay audio from the iPhone, which is converted to text for listening exercises and voice prompts.supported - github.com
  • The agent loop is rebuilt around one model call per step for efficiency.supported - github.com
  • The system supports long tasks without a 40-step limit, and stopped runs can be continued.supported - github.com
  • Each run can be replayed step-by-step on the phone.supported - github.com
  • The ESP32 iPhone Agent uses a $7 ESP32 board to perform the computer's job in connecting AI to iPhones.supported - github.com
  • The project requires an iPhone with iOS 18 or later and cellular data.supported - github.com
  • The ESP32 iPhone Agent uses AirPlay for screen mirroring and forwards the H.264 stream to an app on the iPhone.supported - github.com
  • The app decodes the screen stream, sends screenshots to an OpenAI-compatible vision model, and translates output into actions via USB HID.supported - github.com

Caveats

  • Single-source caution: verify critical details at the linked source.
Radar score 88/100 - how it was calculated
Reliability82
Freshness95
Novelty81
Technical85
Developer96
Ecosystem72
Confidence100
  • Reliability 82: GitHub metadata supports source trust
  • Freshness 95: Fresh GitHub release date
  • Novelty 81: Fresh GitHub release
  • Technical 85: Repository technical metadata
  • Developer 96: Developer tooling signals
  • Ecosystem 72: Fresh GitHub release
  • Confidence 100: Claims have reliable evidence
Share
XLinkedInHacker News

Discussion

Loading comments...

Related articles

Agents - Nov 16, 2026Floe Agent: Open-Source AI Workspace for iOS and AndroidFloe Agent is an open-source AI workspace designed for iOS, iPadOS, and Android, offering an on-device Linux environment. It supports BYOK (Bring Your Own Key) for cloud AI models and local model inference on supported iOS devices, enabling users to work with documents, code, and creative projects locally.Agents - Nov 17, 2026BuilderIO Releases Agent-Native TypeScript FrameworkBuilderIO has released Agent-Native, an open-source TypeScript framework for building agentic applications. It enables agents to pair autonomous work with purpose-built UIs, allowing shared actions, data, and application state between the agent and the user interface.Agents - Nov 16, 2026LightAgent v1.0.0: Lightweight Python Agent FrameworkLightAgent has released version 1.0.0, a lightweight, open-source Python framework for building OpenAI-compatible agents. It offers features like tool integration, memory support, multi-agent collaboration, and workflow orchestration.Agents - Nov 15, 2026n8n Releases Nod: Open Source Personal Agentn8n has released Nod, an open-source personal agent designed to run locally on a user's machine. Nod can automate tasks like chasing invoices, preparing for calls, and canceling online trials by interacting with websites and applications.Agents - Nov 17, 2026Trustabl Agent Dashboard for AI Reliability ScansTrustabl has released Agent Dashboard, a local web UI for running reliability scans, skill tests, and network access probes on AI agents. It integrates Trustabl's Analyzer, Forge, and Probe tools, simplifying agent development and testing.Agents - Nov 16, 2026Node9 Proxy Adds Access Control for AI AgentsNode9 Proxy has released version 2.61.1, introducing enhanced access control for AI agents. This tool allows developers to define permissions for agents interacting with tools like Claude Code, Codex, and Gemini, and provides auditing capabilities.