Can deep learning help you find the perfect girl?

One of the first things I did when I moved to Montreal was installing Tinder. For those of you not familiar with the online meat market, Tinder is a dating app showing nearby users you can like or dislike by their profile pictures. After using the app for a while, I discovered that although I consistently disliked girls with a lot of piercings and tattoos (no offence, just not my type), the app kept showing me these profiles. It surprised me that Tinder did not use my history of swipes to learn what type of girls I like.

This observation made me think: can a computer learn to which girls I'm attracted? I have tried by labeling over 9K profile pictures on Tinder and using deep learning, the latest revolution in artificial intelligence. In this blog post, I will provide a high-level view how I used these techniques to predict my Tinder swipes. You can find the technical details in this paper, which was accepted for the ICML deep learning workshop.

Machine learning

Computers have been around for many decades, but there still exists a lot of problems that the most powerful computers struggle to solve, yet we (as human beings) can do without effort. You can think of detecting faces in pictures, recognizing speech and translating text. The problem is that although we perform such tasks easily, we cannot explain how we do it. That means, we can not come up with a set of rules that can be transformed into a computer program.

Is predicting attractiveness from a profile picture such a task? Yes, it is simply too difficult to specify a set of rules to who you're attracted. For some of you this might sound counter-intuitive because you can specify a clear list of traits e.g. blue eyes, blonde hair, etc. However, even if you could do that, you have to believe me that it is too complex to construct a program by hand that detects such characteristics in images.

On the other hand, if I show you a picture of a girl/boy you can decide in a split of a second whether you like her/him. This suggests a better strategy: we let the computer program itself by showing it profile pictures of girls you do or do not find attractive. We call this machine learning because the program learns a set of attractiveness rules from these profile pictures.

Deep learning

Deep learning is a particular class of learning algorithms that has recently lead to a breakthrough in image recognition (and many other artificial intelligence problems). If you want to see yourself how well these techniques work nowadays, I encourage you to upload a picture to

how-old.net to estimate the age of the persons in the picture
imageidentify.com to detect what's in the picture

In deep learning, the structure of the program is an artificial neural network which is inspired by a biological neural network i.e. the brain. Neural networks are called deep when they consists of several layers of connected neurons as shown in the figure above. Each neuron is connected to a set of neurons in the previous layer. The strength of those connections (shown in red) determine to what image pattern the neuron responds. You can think of it as an adjustable filter, only responding when it sees a particular pattern.

The connection strengths of the network are randomly initialized, meaning that all neurons are just random filters. Therefore, when we feed an image to the network (by inputting the intensity of its pixels), only a few neurons in the first layer will respond, which in turn let some random neurons in the next layer to be active (and so forth till the output). As a result, the output of the network will be random like or dislike.

Therefore, in order to make good predictions, we need to train the network. We do so by inputting a profile picture and propagating it through the network to get the output. If the network's prediction is different from the desired output (like or dislike), we adjust its filters (i.e. its connection strengths) to make the prediction more correct. How we exactly adjust the filters is rather difficult and outside the scope of this blog post, all you need to know is that we can do it. The training phase is nothing more than repeating this process for many examples, and then hope that the network predicts well for profile pictures it has not seen yet.

After training, we often observe that the neurons in the first layer respond to edges and color blobs, while the next layer uses these observations to detect more complex patterns such as eyes and ears. The last layer might detect complete faces or piercings and tattoos (which is important to predict my preferences).

Collecting data

Neural networks work especially well when we provide them a lot of labeled examples. So, all I needed to do was intensively use Tinder for some time.

But how do you save the liked and disliked pictures from the Tinder app? The application is communicating with a server to get new profile pictures, and by inspecting how the app did this communication, I could build a program on my laptop that mimics the tinder app (but saves the pictures). You can see the program at work in the screenshot. I have to say that the real Tinder app was a bit more convenient because it can be used anywhere and anytime. Now I really had to sit down and make time to swipe profiles. Nevertheless, I managed to label 9364 profile pictures in less than two months.

Almost 10K profile pictures sounds a lot, but in practice it was not enough to successfully train a deep neural network. What happened was that the network worked very well for the pictures it was trained on, but did not generalize to new profile pictures. Looking at the images, I could see that there was a lot of variation in the profile pictures (compared to the clean datasets we work with in academia). One way to account for those variations is to collect more data, but I didn't want to wait half a year before I had labeled enough profile pictures (or spent my whole day using Tinder..).

Instead, I focused on another way to overcome this issue: we can use another neural network that is successfully trained for a related task (for which there was enough data). The lower-layer filters learned by that network are recognizing patterns that might be useful for our task. We simply copy the trained network, and then only adjust the last layers to predict my attractiveness preferences.

I thought it would be a good idea to use a network to predict gender from profile pictures for two reasons. First, it is easy to collect data for this task. I scraped over 400 000 (!) male and female profile pictures from OKCupid.com in a couple of weeks. Second, the filters extracted by this network could be useful to predict my preferences, since I am probably more attracted to girls that do not look like men.

The data never lie..

Now that I had collected the data, I had to face the facts about my own Tinder behavior:

It was very confronting that I liked more than half (53%) of all profile pictures. Maybe I'm not that picky, but I also believe that Tinder shows you the popular profiles more frequently. There is a clear incentive for Tinder since you keep using the app when you hope to match with pretty girls. I can also imagine that more attractive people are using Tinder, and my swiped profile pictures are thus not representatives of the whole population (I think it would be weird if I was willing to date more than half of the population).
"What was I thinking when I liked this picture??" This feeling popped up several times when scrolling through the labeled pictures. It was tempting to do another round of labeling to filter out this unwanted likes. I realized that the problem is more general: for some pictures I do not have a clear opinion, and my mood determines the label. So, instead I estimated how much "noise" entered the labeling process. I relabeled a 100 random profile pictures a couple of weeks later and compared the results with the original labeling. I judged 12 pictures differently.
From the 12 errors I made, we could assume that I have guessed 24% of the time (because 24*0.5=12). You have to realize that this makes it difficult for any model to learn my preferences. If you are not consistent in the labeling, you are confusing the network what are important image patterns to learn.

The results

I'll spare you the details of processing all this data, and simply summarize the results:

The neural network trained on profile pictures from OKcupid predicted 92.7% of the time correctly if there was a male or female in the image. It took about 3 days to train this network on one of the fastest graphical cards today. It achieves near human performance because I achieved myself around 95% on this dataset. There were a lot of pictures with both males and females in the picture, and also encountered some pictures without somebody in the picture (apparently there are people who put a picture of a plate of food as their profile picture).
Given the success of the gender network, I had good hope that this network could be used to predict my preferences. Unfortunately, this wasn't the case, this network only achieved 61.5% accuracy.
However, I used another network that was trained (not by me) to distinguish all kinds of animals, plants and vehicles. Somewhat surprising, predicting attractiveness using this network worked quite well: 68.1% accuracy.

Almost 70% does not sound that impressive, right? But how well could someone else predict to which girls I am attracted?

My co-author Jason Yosinski tried it. He 'trained' by first looking at all 50 dislike and 50 like pictures side by side, scrolling through them all a few times. Then the training set was shuffled, pictures were displayed one a time, and he produced a label prediction after each photo. The correct label was shown after each picture so that he could learn from his mistakes. See his study program in the screenshot above. He achieved 86%, 82%, 88% and 88% on 4 training rounds.

He then predicted the labels for 100 unseen profile pictures (the same ones as in my consistency test). He made 24 errors, and thus achieved 76% accuracy. Given that's also hard for other people to predict my preferences, the 68% is not that bad!

The future

The techniques I have used are not tailored to my preferences, and could be used for other people whenever they provide their labeled profiles pictures. In fact, deep learning is known to be working much better with bigger datasets, so I am pretty confident that this approach would be even more successful when I train a neural network on data gathered from many people (e.g. when I have access to all data from Tinder). As far as I know, the most popular dating websites (like OKCupid.com) still use questionaires to suggest potential partners. I think the success of Tinder has shown that people care about somebody's appearance, and with the rise of deep learning we now have a tool to actually incorporate this kind of information in matching algorithms.

The time that you were predetermined to marry one of the 50 girls/boys of your town is over. New technologies will continue to change the way how we find our romantic partners. Although randomness sometimes determines love, we should embrace the great opportunities in machine learning to help anyone looking for a soulmate!