GaussDet: Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors

What changed

Recent advancements in 3D scene reconstruction have seen 3D Gaussian Splatting (3DGS) emerge as a leading technique. A key area of development has been extending 3DGS with language-driven, open-vocabulary understanding, which is crucial for applications like embodied AI. Prior methods often relied on distilling high-dimensional Contrastive Language-Image Pretraining (CLIP) features directly into the scene representation to assign semantics. However, these approaches faced challenges, including the need for a predefined number of instances or susceptibility to noise in bottom-up instance grouping strategies. Furthermore, their reliance on CLIP limited semantic understanding to basic noun phrases, hindering complex spatial reasoning and the grounding of referential expressions.

GaussDet addresses these limitations by moving away from dense CLIP features. Instead, it utilizes discrete, open-vocabulary 2D object detectors that possess referring expression capabilities. The method learns instance features for individual Gaussians, enabling the decomposition of the 3D scene into distinct 3D instance groups. To achieve semantic understanding, these groups are rendered, and semantic votes from multi-view 2D detections are aggregated. This process generates a View-Aggregated Semantic Label Distribution (VASD) for each 3D instance. This view-aggregation strategy acts as a powerful regularizer, mitigating spurious labels that might arise from low-quality instance grouping.

Why it matters for builders

For AI builders, GaussDet presents a significant step forward in enabling more nuanced and flexible language-based interactions with 3D environments. The ability to perform open-vocabulary segmentation and referential grounding without being constrained by predefined instance counts or the limitations of simple noun phrases unlocks new possibilities for applications requiring detailed scene understanding. This method simplifies the process of extending basic language queries to complex referential grounding, making it easier to build systems that can accurately identify and manipulate objects within a 3D scene based on descriptive language.

Practical impact

The practical impact of GaussDet is demonstrated through extensive evaluations on key tasks. The method shows consistent improvements over existing approaches in open-vocabulary segmentation, tested on benchmarks like LeRF-OVS and ScanNet. Crucially, it also excels in referring expression grounding, evaluated on the Ref-LeRF benchmark. In a strict zero-shot setting for referential grounding, GaussDet achieved a notable 16.7% improvement in mean Intersection over Union (mIoU). This suggests that builders can expect more accurate and reliable object identification and segmentation in 3D scenes when using this method, particularly when dealing with complex or specific object references.

Caveats and source limits

The primary source for this information is a research paper published on arXiv. While the paper details the methodology and presents evaluation results, it does not include information regarding implementation availability, specific code repositories, or performance benchmarks beyond those reported within the paper itself. The reported improvements, such as the 16.7% mIoU gain, are specific to the experimental setup and datasets used in the study. Further research and practical implementation would be needed to assess its performance across a wider range of real-world scenarios and hardware configurations. The paper focuses on the technical aspects of the algorithm and does not provide details on potential commercial applications or integration with existing 3D reconstruction pipelines.