Let your agent drive your iPhone through Mac mirroring

What it does

Phone Harness is a thin Python wrapper (~500 lines) that treats the macOS iPhone Mirroring window as a bidirectional transport for iPhone automation. It captures the mirroring window, runs Apple’s Vision framework OCR to locate on-screen text with coordinates, and fires HID-level CGEvents to tap, type, swipe, and scroll. An LLM agent can write ad-hoc Python scripts that are executed directly against the live phone, with an editable agent-workspace/agent_helpers.py file for routines the agent invents on the fly.

The interesting bit

Instead of fighting iOS accessibility APIs or WebDriverAgent, it treats the video stream as the ground truth and Mac input events as prosthetic fingers. The README calls this “the poor man’s DOM” — no DOM, no tree, just screenshots, OCR, and brute-force coordinates.

Key highlights

• No jailbreak, Xcode, or WebDriverAgent required; works with stock iOS and macOS Sequoia+.
• Stateless and daemon-free: every invocation re-discovers the window, captures the screen, and executes fresh.
• Agent can extend its own toolkit by editing agent-workspace/agent_helpers.py during execution.
• Supports taps, long-presses, drags, scroll gestures, and unicode typing via HID keycodes.
• Includes a --doctor flag to verify permissions, window detection, and the full capture-to-tap chain.

Caveats

• One phone, one session; physically unlocking the device pauses mirroring and breaks the flow.
• No multi-touch gestures like pinch, and DRM content renders as black rectangles the OCR cannot read.
• Unlabeled icons are invisible to OCR, so navigating by image requires falling back to a vision-capable model.

Verdict

Worth a look if you want to script a real iPhone from macOS without jailbreaking or Xcode. Give it a pass if you need multi-touch, camera flows, or anything that isn’t tethered to a Mac running iPhone Mirroring.