Sticking 3D labels on reality with ARKit and CoreML

What it does

This Swift template captures live camera frames from an ARKit session, pipes them through an Inception V3 CoreML model, and pins 3D SceneKit text labels onto whatever the model recognizes. It is essentially glue code that demonstrates how to keep two hungry Apple frameworks fed and synchronized on an iPhone.

The interesting bit

Rather than letting CoreML stall the render loop, the project shunts inference onto a dedicated dispatch queue so SceneKit can hold 60 FPS on an iPhone 7+ while the model churns in the background. The neat part is feeding the ARFrame’s raw capturedImage pixel buffer straight into Vision instead of managing a separate camera feed.

Key highlights

• Uses ARKit’s ARFrame.capturedImage as the direct input source for CoreML inference
• Offloads CoreML updates to a background queue so ARKit rendering stays independent
• Renders floating 3D text labels in SceneKit positioned by raycasting from screen center to ARKit feature points
• Built as an explicit template for combining ARKit, CoreML, and SceneKit in a single iOS project
• Achieved 60 FPS on iPhone 7+ in testing, though thermal throttling drops it to 30 FPS

Caveats

• SceneKit 3D text is expensive; the README warns that dense labels can crash the app and suggests SpriteKit would be more efficient
• Label placement relies on a naive raycast to the nearest feature point, so labels may jitter or drift
• The author notes uncertainty about whether RGB or YUV data is being passed to the Vision model, though Inception V3 tolerates it for now

Verdict

Worth a look if you are bootstrapping an ARKit-plus-CoreML prototype and can tolerate rough edges. Look elsewhere if you need battle-tested spatial anchoring or polished text rendering.