Our approach consists in training a Convolutional Neural Network (CNN) to build a feature representation of an image patch. We train by using two patches simultaneously that should either correspond to the same point and thus have similar features, or different points and thus different features. We optimize this by using a Siamese architecture, that is, we input two patches simultaneous and minimize the L2 distance between the features if they correspond to the same point and maximize the L2 distance if they correspond to different points. In order to learn efficient and discriminative representations, we propose a positive and negative mining approach which is shown to be critical for performance.
We train and evaluate our results on the Brown dataset1. We also provide evaluation on other datasets including the DaLI dataset and show we significantly outperform existing approaches. Furthermore, our approach computes 128 dimension vectors that can be compared directly with L2 and thus is suitable as a drop-in replacement for SIFT.
For full details and results please refer to the full paper.