TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

What changed

TrackEverything introduces a new paradigm for 3D point tracking by representing videos as persistent 3D scene tracks in world coordinates. This method overcomes the limitation of existing models that must choose between tracking a few points for a long time or many points for a short time. Key innovations include a voxelization-based de-duplication mechanism to merge overlapping tracks, a decomposition of tracking into endpoint refinement and trajectory refinement for dynamic points, and a novel 3D WAFT method that replaces memory-intensive 4D correlation volumes with efficient feature sampling.

Why it matters for builders

This work is significant for builders as it enables dense tracking of all visible points across videos exceeding 1000 frames while staying within a 40 GB GPU memory limit. This capability is crucial for applications that require detailed, long-term understanding of 3D environments, such as robotics, autonomous driving, and augmented reality.

Practical impact

TrackEverything demonstrates superior performance on the TAPVid-3D benchmark, outperforming open-source dense 3D trackers by over 20% APD on short clips. It also remains competitive with state-of-the-art sparse trackers on long sequences, despite tracking a much larger number of points. This suggests that builders can achieve more comprehensive and accurate scene tracking with reduced computational overhead.

Caveats and source limits

The provided source is a research paper abstract. While it details the methodology and performance claims, it does not include specific implementation details, code availability, or pre-trained model weights. Further information regarding practical deployment and integration would require access to the full paper or associated code repository.