Feature Extraction Research

Feature extraction is a fundamental part of data processing which focuses on converting raw data into compact and useful representations, and has a wide applicability to many different types of problems. The research here focuses on extracting useful, compact, and discriminative features for a diversity of problems such as patch matching, or finding similar styles to certain images.

  • Deep Convolutional Feature Point Descriptors

    Deep Convolutional Feature Point Descriptors

    We learn compact discriminative feature point descriptors using a convolutional neural network. We directly optimize for using L2 distance by training with a pair of corresponding and non-corresponding patches correspond to small and large distances respectively using a Siamese architecture. We deal with the large number of potential pairs with the combination of a stochastic sampling of the training set and an aggressive mining strategy biased towards patches that are hard to classify. The resulting descriptor is 128 dimensions that can be used as a drop-in replacement for any task involving SIFT. We show that this descriptor generalizes well to various datasets.

  • Geodesic Finite Mixture Models

    Geodesic Finite Mixture Models

    There are many cases in which data is found to be distributed on a Riemannian manifold. In these cases, Euclidean metrics are not applicable and one needs to resort to geodesic distances consistent with the manifold geometry. For this purpose, we draw inspiration on a variant of the expectation-maximization algorithm, that uses a minimum message length criterion to automatically estimate the optimal number of components from multivariate data lying on an Euclidean space. In order to use this approach on Riemannian manifolds, we propose a formulation in which each component is defined on a different tangent space, thus avoiding the problems associated with the loss of accuracy produced when linearizing the manifold with a single tangent space. Our approach can be applied to any type of manifold for which it is possible to estimate its tangent space.

  • Deformation and Light Invariant Descriptor

    Deformation and Light Invariant Descriptor

    DaLI descriptors are local image patch representations that have been shown to be robust to deformation and strong illumination changes. These descriptors are constructed by treating the image patch as a 3D surface and then simulating the diffusion of heat along the surface for different intervals of time. Small time intervals represent local deformation properties while large time intervals represent global deformation properties. Additionally, by performing a logarithmic sampling and then a Fast Fourier Transform, it is possible to obtain robustness against non-linear illumination changes. We have created the first feature point dataset that focuses on deformation and illumination changes of real world objects in order to perform evaluation, where we show the DaLI descriptors outperform all the widely used descriptors.

Publications

  • 3D Human Pose Tracking Priors using Geodesic Mixture Models
    • 3D Human Pose Tracking Priors using Geodesic Mixture Models
    • Edgar Simo-Serra, Carme Torras, Francesc Moreno-Noguer
    • International Journal of Computer Vision (IJCV) 122(2):388-408, 2016
  • Fashion Style in 128 Floats: Joint Ranking and Classification using Weak Data for Feature Extraction
  • Discriminative Learning of Deep Convolutional Feature Point Descriptors
    • Discriminative Learning of Deep Convolutional Feature Point Descriptors
    • Edgar Simo-Serra*, Eduard Trulls*, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, Francesc Moreno-Noguer (* equal contribution)
    • International Conference on Computer Vision (ICCV), 2015
  • Lie Algebra-Based Kinematic Prior for 3D Human Pose Tracking
    • Lie Algebra-Based Kinematic Prior for 3D Human Pose Tracking
    • Edgar Simo-Serra, Carme Torras, Francesc Moreno-Noguer
    • International Conference on Machine Vision Applications (MVA) [best paper], 2015
  • DaLI: Deformation and Light Invariant Descriptor
    • DaLI: Deformation and Light Invariant Descriptor
    • Edgar Simo-Serra, Carme Torras, Francesc Moreno-Noguer
    • International Journal of Computer Vision (IJCV) 115(2):135-154, 2015
  • Geodesic Finite Mixture Models

Source Code

  • GFMM
  • StyleNet
  • Deep Descriptor
  • DaLI
  • ceigs

Datasets

  • DaLI Dataset
    • DaLI Dataset
    • Local image patch feature descriptor illumination and deformation invariance evaluation dataset.