An egocentric hand tracker built from 4,800 images of board games

What it does

This repository documents a complete pipeline for training a single-shot detector (SSD MobileNet v1) to locate hands in video using the TensorFlow Object Detection API. It focuses on egocentric viewpoints—hands over a table, shot from a head-mounted camera—and includes egohands_dataset_clean.py to convert the Egohands dataset into the tfrecord format TensorFlow expects. The trained model runs at up to 21 FPS on a MacBook Pro CPU without a GPU.

The interesting bit

The author treats dataset selection as the critical experiment: the Oxford Hands Dataset produced poor results, but the Egohands dataset—4,800 frames of people playing chess, cards, and Jenga across 48 environments—provided enough variety to outperform rule-based trackers when lighting changes or hands overlap. The project is essentially a transfer-learning walkthrough, but its real lesson is that a tightly matched dataset matters more than a fancy architecture.

Key highlights

• Achieves real-time inference on a CPU (21 FPS at 320×240, 16 FPS with visualization) using an SSD MobileNet v1 model fine-tuned from the TensorFlow model zoo.
• Ships with egohands_dataset_clean.py to download, split, and convert the Egohands dataset’s polygon annotations into bounding boxes and CSV labels.
• Explicitly contrasts neural detection against rule-based methods (HOG, color histograms), noting the latter fail under occlusion and variable lighting.
• The trained model has been exported to TensorFlow.js (as Handtrack.js for browser use) and TensorFlow Lite (for Android), extending its reach beyond the original Python codebase.

Caveats

• The code is pinned to TensorFlow 1.4.0-rc0; the README warns that other versions may error out and suggests regenerating the frozen inference graph from the provided checkpoints.
• The training instructions are high-level, and the author directs readers to external tutorials for the detailed mechanics of custom object detection training.

Verdict

A solid reference if you are building a custom object-detection pipeline and want a reproducible example of dataset curation and transfer learning. Look elsewhere if you need a plug-and-play hand detector for modern TensorFlow versions.