A new open-source project on GitHub shows how an AI model can operate a real iPhone using a microcontroller that costs about $7. The project, called ESP32 iPhone Agent, is released under the MIT license and written in Python. It needs no Mac, no Xcode, no Apple developer account, no jailbreak and no app installed on the phone. According to the project's repository, it runs on macOS or Linux and works with any vision model that exposes an OpenAI-compatible API.
The repository, which had 1,526 stars at the time of publication, is one of a growing number of experiments that give large language models a way to act on a screen rather than only generate text. Its approach is unusual because it avoids installing software on the target device and instead treats the iPhone as a physical display and input target.
How the system sees and touches the phone
The project splits the job into two channels. For sight, the iPhone streams its screen over AirPlay mirroring to UxPlay, an open-source AirPlay receiver. A GStreamer pipeline then captures the stream and sends JPEG frames to the agent. The model therefore works from something close to a live screenshot of the device.
For touch, the setup relies on an ESP32-S3 board plugged into the phone's USB port. The board acts as a USB Human Interface Device, the same class of device used by keyboards and mice. The repository's description says that iOS AssistiveTouch, the phone's built-in accessibility feature for pointer control, converts the mouse movements from the board into taps and swipes. Because the phone sees ordinary pointer input, no special software is needed on its side.
Two models, one loop
The agent uses two model roles. According to the project description, a planner model first writes a step-by-step plan for the task. An actor model then chooses each individual action by looking at the current screenshot. This separation is a common pattern in agent design: a stronger reasoning step sets the direction, while a faster step handles the moment-to-moment decisions that depend on what is on screen.
The description also notes that cloud-hosted and self-hosted vision models both work, as long as they can be reached through an OpenAI-compatible endpoint. That gives users the option to keep screenshots on their own hardware, although the project's documentation should be checked for the exact privacy implications of each configuration.
Why the hardware choice matters
Controlling a phone through software usually requires either a computer with developer tools, a jailbroken device, or an app that has been granted special permissions. Each of these carries cost, setup effort or risk. By contrast, the ESP32-S3 is a widely available, low-cost microcontroller, and the project positions it as the only dedicated hardware needed beyond the phone and a computer.
That simplicity is the main selling point. A tester could, in principle, build the rig from a small parts list and a standard cable, then point the agent at a task. The project's own summary frames the result as an AI system that uses a real iPhone with AirPlay eyes and USB hands, a phrase that captures the design in one line.
What readers should keep in mind
- The project is an experimental tool published on GitHub, not a commercial product with a support team.
- Its functionality depends on AirPlay mirroring, AssistiveTouch and a USB connection working together, so changes to any of these on the phone could affect results.
- Automating an account-protected app raises questions about terms of service and account security, which users should weigh before running an agent on a personal device.
What comes next
The repository's popularity suggests that developers are interested in agents that operate devices the way people do, through the screen. Projects like this one test whether vision models can be trusted with multi-step tasks on a phone, where a single misplaced tap can change the outcome. Reliability, speed and safety checks will determine whether such tools move beyond hobby setups.
For now, ESP32 iPhone Agent is best seen as a proof of concept. It shows that a low-cost board, a standard streaming protocol and an open-source agent loop can together give a language model practical control over a locked-down consumer device, and it invites the wider community to test where that approach works and where it breaks.
Related: Magic Hotspot: Connect Linux to an iPhone Hotspot · AirDrop on Linux: open-source omdrop-owl targets non-Apple Wi-Fi cards
Comments (0)