Home	Keypoint Matching Benchmark	Geometric Registration Benchmark	RGB-D Reconstruction Datasets

3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

Matching local geometric features on real-world depth images is a challenging task due to the noisy, low-resolution, and incomplete nature of 3D scan data. These difficulties limit the performance of current state-of-art methods, which are typically based on histograms over geometric properties. In this paper, we present 3DMatch, a data-driven model that learns a local volumetric patch descriptor for establishing correspondences between partial 3D data. To amass training data for our model, we propose an unsupervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions. Experiments show that our descriptor is not only able to match local geometry in new scenes for reconstruction, but also generalize to different tasks and spatial scales (e.g. instance-level object model alignment for the Amazon Picking Challenge, and mesh surface correspondence). Results show that 3DMatch consistently outperforms other state-of-the-art approaches by a significant margin.

Overview

From existing RGB-D reconstructions (a), we extract local 3D patches and correspondence labels from scans of different views (b). We collect pairs of matching and non-matching local 3D patches converted into a volumetric representation (c) to train a 3D ConvNet-based descriptor (d). This geometric descriptor can be used to establish correspondences for matching 3D geometry in various applications (e) such as reconstruction, model alignment, and surface correspondence.

Paper

Latest version (5 Dec 2016): arXiv:1603.08182 [cs.CV] or here
Older version (27 Mar 2016): 3DMatch: Learning the Matching of Local 3D Geometry in Range Scans

To appear at IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017 Oral Presentation

Bibtex

Video

Code

All 3DMatch code (for training and testing) can be found in our Github repository here.

Keypoint Matching Benchmark

This benchmark evaluates how well descriptors (both 2D and 3D) can establish correspondences between RGB-D frames of different views. The dataset contains 2D RGB-D patches and 3D patches (local TDF voxel grid volumes) of wide-baselined correspondences, which are sampled from our testing split of the RGB-D reconstruction datasets. The pixel size of each 2D patch is determined by the projection of the 0.3m³ local 3D patch around the interest point onto the image plane. We provide Matlab code for generating similar correspondence datasets here. Although our baselines are 3D approaches that use depth information only, we are also looking for descriptor algorithms (2D or 3D) that leverage color information or both modalities.

Download and Description

There are two Matlab .mat files, one for the validation set and one for the test set. Download links:

The validation set contains 10,000 pairs of RGB-D patches and their ground truth correspondence labels (binary 1 for match and 0 for non-match). The test dataset contains similar data for another 10,000 pairs, except the ground truth correspondence labels have been left out. Each Matlab .mat file contains the variables:

data - a 10,000x2 cell array of structs. Each struct contains the 2D/3D patch data of an interest point, with the variables: labels - (in validation-set.mat only) a 10,000x1 cell array of binary correspondence labels (1 for match, 0 for non-match) for each pair (row) of interest points saved in data.

Update (as of Mar 2018): for convenience, feel free to download the labels for the test set here.

We do not provide a fixed training set, however we do provide the C++/CUDA code here that we used to sample training correspondences on-the-fly to train 3DMatch. You can also generate your own fixed training set by following the instructions here (see makeCorresDataset.m) with the training scenes instead of the testing scenes.

Evaluation

To evaluate on this benchmark, your descriptor algorithm should output a .log file where each row is the descriptor distance (or for some algorithms, the confidence of non-correspondence) between each pair of patches. Our Github toolbox contains an example .log file (for 3DMatch), as well as an example evaluation script for the validation set. Error is computed as false positive rate at 95% recall.

Geometric Registration Benchmark

Similar in spirit to the registration benchmark from Robust Reconstruction of Indoor Scenes, this benchmark evaluates the performance of geometric registration algorithms in the context of scene reconstruction. However, in contrast to prior work, this benchmark uses real-world RGB-D scanning data instead of synthetic data, in order to promote registration algorithms that are robust to depth data from modern commodity range sensors (e.g. Microsoft Kinect, Intel RealSense).

Downloads

This benchmark contains eight sets of scene fragments created from our testing split of the RGB-D reconstruction datasets. These fragments are available for download in the links below. Each fragment is a 3D point cloud of a surface, integrated from 50 depth frames using TSDF volumetric fusion, and saved to a .ply file. We also provide the fusion code to generate these fragments here.

Evaluation

For evaluation, your geometric registration algorithm should determine whether each non-consecutive pair of fragments can be aligned, and if so, output the predicted rigid transformation to a log file. The format of this log file, and more information about the evaluation protocol, are described here. To compute precision and recall from your algorithm's log files, use the evaluation code here (see Matlab script evaluation/geometric-registration/evaluate.m), or refer to this. We particularly seek registration methods that align fragments without requiring an intial alignment.

Geometric Registration on Synthetic Data

In our paper, we also report the results of 3DMatch's performance over the original synthetic benchmark from Robust Reconstruction of Indoor Scenes, where we achieve 65.1% recall and 25.2% precision. To reproduce our results, you can download the evaluation files for that here:

Code to run 3DMatch on both benchmarks can be found here. We also provide several files with intermediate data generated by our geometric registration pipeline for 3DMatch (as well as the other descriptors we compared to), such as TDF voxel grids for all fragments, keypoints, and descriptor vectors. This can be useful if you wish to improve geometric registration results by designing better search algorithms (RANSAC variants). You can download them here:

RGB-D Reconstruction Datasets

We use several existing RGB-D reconstruction datasets to train 3DMatch and generate evaluation benchmarks. For ease-of-use and compatibility with our code (see Github), we've converted these datasets into a unified file structure and format, available for download in the links below. If you find any of these datasets useful, please cite their original paper(s):


Bash script to download all scenes: download.sh
Training and testing scenes split: split.txt


Note: SUN3D scenes were reconstructed with Halber et al. Please also cite their paper if you use the SUN3D scenes.

Dataset Format

Each scene is a folder containing one or more RGB-D video sequences. The folder contents are as follows:

camera-intrinsics.txt - a text file with depth camera intrinsics (3x3 matrix in homogeneous coordinates)
seq-XX

License Agreements

7-Scenes: The data is provided for non-commercial use only. License Agreement
BundleFusion: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License
Analysis by Synthesis: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License



Page last updated: 10-Mar-2017
Posted by: Andy Zeng