Neuroaesthetics in Fashion

Being able to understand and model fashion can have a great impact in everyday life. From choosing your outfit in the morning to picking your best picture for your social network profile, we make fashion decisions on a daily basis that can have impact on our lives. As not everyone has access to a fashion expert to give advice on the current trends and what picture looks best, we have been working on developing systems that are able to automatically learn about fashion and provide useful recommendations to users. In this work we focus on building models that are able to discover and understand fashion. For this purpose we have created the Fashion144k dataset, consisting of 144,169 user posts with images and their associated metadata. We exploit the votes given to each post by different users to obtain measure of fashionability, that is, how fashionable the user and their outfit is in the image. We propose the challenging task of identifying the fashionability of the posts and present an approach that by combining many different sources of information, is not only able to predict fashionability, but it is also able to give fashion advice to the users.

Paper Dataset

For related work, see our work on parsing clothing in images and fashion style.

Method Overview
Overview of our method. We separately model the user, setting and outfit and use all three elements to evaluate the fashionability and recommend a more suitable outfit to the user.

Introduction

Fashion is a very subjective concept. If you ask any two people for fashion advice, you will never get any two answers. This gives in an endless amount of possibilities as garments that were once out of fashion can come back into style, or new combinations can become popular. This subjectivity is precisely what makes trying to tackle fashion with computer systems extremely complicated. It is necessary to have a more objective measure in order to be able to create fashion models that are able to predict how fashionable a certain outfit is. In this work we propose leveraging the largest fashion website chictopia.com and using the votes provided by users as a more democratic and objective measure, which allows us to design models to predict the fashionability of the outfits being worn.

Furthermore, as fashion is an ever changing concept, instead of relying on sets of rules carefully engineered by fashion experts, we use a data-driven approach in which we attempt to automatically infer the different nuances of the fashion realm. This is done by exploiting large amounts of data generated by users to teach the model, which can be constantly updated to keep in sync with the current fashion trends.

Fashion Post
Example of a fashion post used to learn our model. It consists of an image with associated meta-data, that is, additional information like the location, comments, tags...

For this purpose we have collected a large dataset which we call Fashion144k formed by different posts consisting of an image with some additional information. We are able to use this dataset to create models that can understand and give fashion recommendations.

Fashionability Map

Fashionability map of the Fashion144k dataset. Red indicates regions where people are more trendy, while blue indicates less trendy areas according to our data.

Fashion144k Dataset

We collected a total of 144,169 posts to create a dataset to be able to create our models. The posts are distributed all over the world and allow us to get a glimpse of the state of global fashion. However, the data is not uniformly distributed as seen below in the density map.

Density Map

Density map of the Fashion144k dataset. Darker colours indicate a larger amount of posts in that location.

We use the votes given by users to each post as a proxy for the absolute fashionability of the post, which is something subjective that can not be directly measured. While this is not a perfect metric, it is something that allows us to teach a computer to rate and recommend fashion images. This opens up the possibility to automatic rating of outfit of the day posts, and having a personal fashion advisor in your cellphone.

We have run an analysis on the most fashionable cities in the dataset. While some results are surprising such as Manila having a very high mean fashionability, other results are in concordance to what one would think. Such as Los Angeles, Paris, New York or London being above average, and Barcelona being trendier than Madrid.

List of cities with over 1000 posts with their mean fashionability score. The fashionability score is on a 1 to 10 scale.
City Name Posts Fashionability
Manila 4269 6.627
Los Angeles 8275 6.265
Melbourne 1092 6.176
Montreal 1129 6.144
Paris 2118 6.070
Amsterdam 1111 6.059
Barcelona 1292 5.845
Toronto 1471 5.765
Bucharest 1385 5.667
New York 4984 5.514
London 3655 5.444
San Francisco 2880 5.392
Madrid 1747 5.371
Vancouver 1468 5.266
Jakarta 1156 4.398

We can also do a higher level analysis and try to relate the fashionability scores with country attributes such as the mean economy and income class, the Gross Domestic Product (GDP) and the population. It is interesting to see that sadly as expected, countries with more developed economies have larger fashionability scores. Also, in general, countries with larger populations score better on average.

Relation between high level attributes and fashionability in the dataset. Economy and income class are on a scale such that smaller numbers corresponds to most developed country and larger numbers to less developed country. Negative values mean that low values are correlated with high fashionability while high values indicate that high values are correlated with high fashionability.
Attribute Correlation Interpretation
Economy class -0.137 Countries with stronger economies tend to have more trendy inhabitants.
Income class -0.111 Countries with higher mean income tend to have more fashionable inhabitants.
log(GDP) 0.258 Countries with higher Gross Domestic Product tend to have more trendy inhabitants.
log(Population) 0.231 Countries with larger populations tend to have more fashionable inhabitants.

Fashion Recommendations

Of course our goal is not to propose a new dataset, but to instead leverage it to create a model that is able to understand fashion and provide fashion advice. We do this by decomposing each post into three different components: the user, the setting, and the outfit itself.

Conditional Random Fields Fashionability Model

Architecture of our model. We model explicitly the user, setting, outfit and use all the three elements when doing predictions. Each different component uses a different set of features.

The user component attempts to capture the characteristics of each individual, such as their ethnicity, gender, etc. Not all clothing is suitable for all people. A simple example is how high heels in general are not suitable for men. The setting component attempts to capture where the picture was taken. For example wearing a suit at the beach is not something in general considered fashionable. Finally, we attempt to capture the variability of all the outfits themselves.

Fashion Recommendations

Examples of recommendations provided by our model. The numbers in parenthesis represent the fashionability score of that particular outfit.

We believe this is an important step forward to making fashion widely available to the general population. We are very excited by this research and have plans to continue this line of work for the next years.

In the News

I am keeping a bit of track of where this is being reported. Below is a list of places that are talking about our research. Please note that these are for reference only.

Selected News

News and Tech Websites

Fashion Magazines (Online)

International News

Television and Radio

Publications

2015

Neuroaesthetics in Fashion: Modeling the Perception of Fashionability
Neuroaesthetics in Fashion: Modeling the Perception of Fashionability
Edgar Simo-Serra, Sanja Fidler, Francesc Moreno-Noguer, Raquel Urtasun
Conference in Computer Vision and Pattern Recognition (CVPR), 2015
PDF project page poster DOI
In this paper, we analyze the fashion of clothing of a large social website. Our goal is to learn and predict how fashionable a person looks on a photograph and suggest subtle improvements the user could make to improve her/his appeal. We propose a Conditional Random Field model that jointly reasons about several fashionability factors such as the type of outfit and garments the user is wearing, the type of the user, the photograph's setting (e.g., the scenery behind the user), and the fashionability score. Importantly, our model is able to give rich feedback back to the user, conveying which garments or even scenery she/he should change in order to improve fashionability. We demonstrate that our joint approach significantly outperforms a variety of intelligent baselines. We additionally collected a novel heterogeneous dataset with 144,169 user posts containing diverse image, textual and meta information which can be exploited for our task. We also provide a detailed analysis of the data, showing different outfit trends and fashionability scores across the globe and across a span of 6 years. 
@InProceedings{SimoSerraCVPR2015,
   author    = {Edgar Simo-Serra and Sanja Fidler and Francesc Moreno-Noguer and Raquel Urtasun},
   title     = {{Neuroaesthetics in Fashion: Modeling the Perception of Fashionability}},
   booktitle = "Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)",
   year      = 2015,
}