The system performs near real-time 6-DoF tracking of arbitrary rigid objects from monocular RGBD video while simultaneously building a neural 3D representation. It uses a Neural Object Field learned concurrently with pose graph optimization, with a dynamic memory pool to accumulate geometry and appearance into a consistent 3D model. The method handles challenging cases including partial and full occlusion, untextured surfaces, and large pose changes without requiring additional object information beyond an initial segmentation.