In this research we focus on the semantic segmentation of clothings from still images. This is a very complex task due to the large number of classes where intra-class variability can be larger than inter-class variability. We propose a Conditional Random Field (CRF) model that is able to leverage many different image features to obtain state-of-the-art performance on the challenging Fashionista dataset.

Above we show some additional results from our algorithm. First row corresponds to the image, the second row to the ground truth and the third row to the prediction given by our model. For colour labels please refer to the paper.

For related work see our work on predicting fashionability and fashion style.

In order to allow researchers to analyze our results and compare against our results we make the code for learning and performing inference for our model. In order to be able to run the model you must also download the features separately.

Installation

In order to get started first checkout out the source code and then extract the features:

# Check out the git and cd into it as working directory
git clone https://github.com/bobbens/clothes_parsing.git
cd clothes_parsing
# Get and unpack the necessary features
wget https://esslab.jp/~ess/data/poseseg.tar.bz2
tar xvjf poseseg.tar.bz2

The dSP dependency must also be compiled. This can be done by:

cd lib/dSP_5.1
make # First edit the Makefile if necessary

Please note that you will likely need to edit the Makefile to point MATLABHOME at the right location.

Usage

You can reproduce results simply by running from Matlab:

% Load the model, parameters can be set here
sm = segmodel( 'PROFILE', '0.16', 'use_real_pose', false );
% Trains some misc stuff
sm = sm.train_misc_unaries();
% Actually sets up and trains the CRF
sm = sm.train_MRF();
% Performs testing and outputs results
R = sm.test_MRF_segmentation()

This should generate an output like:

BUILDING MRF OUTPUT 29 CLASSES (REAL POSE=0)...
UNARIES:
   bgbias
   logreg:       29
   cpmc_logreg:  29
   cpmc
   shapelets
HIGHER ORDER
   similarity
   limbs
Initializing Image 011 / 350...   0.4 seconds!   

...

Tested MRF in 319.0 seconds
350 / 350... 

R = 

    confusion: [29x29 double]
    order: [29x1 double]
    acc: 0.8432
    pre: [29x1 double]
    rec: [29x1 double]
    f1: [29x1 double]
    voc: [29x1 double]
    avr_pre: 0.3007
    avr_rec: 0.3292
    avr_f1: 0.3039
    avr_voc: 0.2013

Please note that due to stochastic components and differences between software versions, the numbers will not be exactly the same as the paper. For the paper all results were obtained on a linux machine running Ubuntu 12.04 with Matlab R2012a (7.14.0.739) 64-bit (glnxa64).

You can furthermore visualize the output of the model with:

sm.test_MRF_visualize( 'output/' )

This will save both the ground truth segmentations and the predicted segmentations in the directory ‘output/’ as shown in the paper.

If you use this code please cite the paper

@InProceedings{SimoSerraACCV2014,
   author = {Edgar Simo-Serra and Sanja Fidler and Francesc Moreno-Noguer and Raquel Urtasun},
   title = {{A High Performance CRF Model for Clothes Parsing}},
   booktitle = "Proceedings of the Asian Conference on Computer Vision (2014)",
   year = 2014,
}

We would like to give our thanks to Kota Yamaguchi for his excellent code which we have used as a base for our model.

The different codes we have used (in alphabetical order):

1. clothing_parsing
2. CPMC
3. dSP
4. gPb
5. grTheory
6. liblinear
7. Pose
8. Selective Search