Frolian's blog standing on the shoulders of PyGiants

• About
• Archive

A Convolutional Neural Network for Face Keypoint Detection

Thu, 14 Sep 2017

Machine Learning Image Processing Kaggle

Yesterday, I read this recent article on medium about facial keypoint detection. The article suggests that deep learning methods can easily be used to perform this task. It ends by suggesting that everyone should try it, since the data needed and the toolkits are all open source. This article is my attempt, since I've been interested in face detection for a long time and written about it before.

This is the outline of what we'll try:

• loading the data
• analyzing the data
• building a Keras model
• checking the results
• applying the method to a fun problem

Loading the data

The data we will use comes from a Kaggle challenge called Facial Keypoints Detection. I've downloaded the .csv file and put it in a data/ directory. Let's use pandas to read it.

import pandas as pd

df = pd.read_csv('data/training.csv')

df.head()

	left_eye_center_x	left_eye_center_y	right_eye_center_x	right_eye_center_y	left_eye_inner_corner_x	left_eye_inner_corner_y	left_eye_outer_corner_x	left_eye_outer_corner_y	right_eye_inner_corner_x	right_eye_inner_corner_y	...	nose_tip_y	mouth_left_corner_x	mouth_left_corner_y	mouth_right_corner_x	mouth_right_corner_y	mouth_center_top_lip_x	mouth_center_top_lip_y	mouth_center_bottom_lip_x	mouth_center_bottom_lip_y	Image
0	66.033564	39.002274	30.227008	36.421678	59.582075	39.647423	73.130346	39.969997	36.356571	37.389402	...	57.066803	61.195308	79.970165	28.614496	77.388992	43.312602	72.935459	43.130707	84.485774	238 236 237 238 240 240 239 241 241 243 240 23...
1	64.332936	34.970077	29.949277	33.448715	58.856170	35.274349	70.722723	36.187166	36.034723	34.361532	...	55.660936	56.421447	76.352000	35.122383	76.047660	46.684596	70.266553	45.467915	85.480170	219 215 204 196 204 211 212 200 180 168 178 19...
2	65.057053	34.909642	30.903789	34.909642	59.412000	36.320968	70.984421	36.320968	37.678105	36.320968	...	53.538947	60.822947	73.014316	33.726316	72.732000	47.274947	70.191789	47.274947	78.659368	144 142 159 180 188 188 184 180 167 132 84 59 ...
3	65.225739	37.261774	32.023096	37.261774	60.003339	39.127179	72.314713	38.380967	37.618643	38.754115	...	54.166539	65.598887	72.703722	37.245496	74.195478	50.303165	70.091687	51.561183	78.268383	193 192 193 194 194 194 193 192 168 111 50 12 ...
4	66.725301	39.621261	32.244810	38.042032	58.565890	39.621261	72.515926	39.884466	36.982380	39.094852	...	64.889521	60.671411	77.523239	31.191755	76.997301	44.962748	73.707387	44.227141	86.871166	147 148 160 196 215 214 216 217 219 220 206 18...

5 rows × 31 columns

df.shape

(7049, 31)

Analyzing the data

The Image column contains the face data for which the 30 first columns represent the keypoint data (15 x-coordinates and 15 y-coordinates). Let's try to get a feel for the data. First, let's display some faces.

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def string2image(string):
    """Converts a string to a numpy array."""
    return np.array([int(item) for item in string.split()]).reshape((96, 96))

def plot_faces(nrows=5, ncols=5):
    """Randomly displays some faces from the training data."""
    selection = np.random.choice(df.index, size=(nrows*ncols), replace=False)
    image_strings = df.loc[selection]['Image']
    fig, axes = plt.subplots(figsize=(10, 10), nrows=nrows, ncols=ncols)
    for string, ax in zip(image_strings, axes.ravel()):
        ax.imshow(string2image(string), cmap='gray')
        ax.axis('off')

plot_faces()

Let's now add to that plot the facial keypoints that were tagged. First, let's do an example :

keypoint_cols = list(df.columns)[:-1]

xy = df.iloc[0][keypoint_cols].values.reshape((15, 2))
xy 

array([[66.033563909799994, 39.002273684199999],
       [30.227007518800001, 36.4216781955],
       [59.582075188000005, 39.647422556399995],
       [73.130345864700004, 39.9699969925],
       [36.356571428599999, 37.389401503800002],
       [23.452872180500002, 37.389401503800002],
       [56.953263157899997, 29.033648120300001],
       [80.227127819499998, 32.2281383459],
       [40.227609022599999, 29.002321804499999],
       [16.3563789474, 29.647470676699999],
       [44.420571428599999, 57.066803007499999],
       [61.195308270699996, 79.970165413499998],
       [28.614496240600001, 77.388992481199992],
       [43.312601503800003, 72.935458646599997],
       [43.130706766899998, 84.485774436100002]], dtype=object)

plt.plot(xy[:, 0], xy[:, 1], 'ro')
plt.imshow(string2image(df.iloc[0]['Image']), cmap='gray')

<matplotlib.image.AxesImage at 0x116027f60>

Now, let's add this to the function we wrote before.

def plot_faces_with_keypoints(nrows=5, ncols=5):
    """Randomly displays some faces from the training data with their keypoints."""
    selection = np.random.choice(df.index, size=(nrows*ncols), replace=False)
    image_strings = df.loc[selection]['Image']
    keypoint_cols = list(df.columns)[:-1]
    keypoints = df.loc[selection][keypoint_cols]
    fig, axes = plt.subplots(figsize=(10, 10), nrows=nrows, ncols=ncols)
    for string, (iloc, keypoint), ax in zip(image_strings, keypoints.iterrows(), axes.ravel()):
        xy = keypoint.values.reshape((15, 2))
        ax.imshow(string2image(string), cmap='gray')
        ax.plot(xy[:, 0], xy[:, 1], 'ro')
        ax.axis('off')

plot_faces_with_keypoints()

We can make several observations from this image:

• some images are high resolution, some are low
• some images have all 15 keypoints, while some have only a few

Let's do some statistics about the keypoints to investigate that last observation :

df.describe().loc['count'].plot.bar()

<matplotlib.axes._subplots.AxesSubplot at 0x1189121d0>

What this plot tells us is that in this dataset, only 2000 images are "high quality" with all keypoints, while 5000 other images are "low quality" with only 4 keypoints labelled.

Let's start training the data with the high quality images and see how far we get.

fully_annotated = df.dropna()

fully_annotated.shape

(2140, 31)

Building a Keras model

Now on to the machine learning part. Let's build a Keras model with our data. Actually, before we do that, let's do some preprocessing first, using the scikit-learn pipelines (inspired by this great post on scalable Machine Learning by Tom Augspurger).

The idea behind pipelining is that it allows you to easily keep track of the data transformations applied to our data. We need two scalings: one for the input and one for the output. Since I couldn't get the scaling to work for 3d image data, we will only use a pipeline for our outputs.

X = np.stack([string2image(string) for string in fully_annotated['Image']]).astype(np.float)[:, :, :, np.newaxis]

y = np.vstack(fully_annotated[fully_annotated.columns[:-1]].values)

X.shape, X.dtype

((2140, 96, 96, 1), dtype('float64'))

y.shape, y.dtype

((2140, 30), dtype('float64'))

X_train = X / 255.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MinMaxScaler

output_pipe = make_pipeline(
    MinMaxScaler(feature_range=(-1, 1))
)

y_train = output_pipe.fit_transform(y)

In this case, the pipelining process is, how to say this, not very spectacular. Let's move on and train a Keras model! We will start with a simple model, as found in this blog post with a fully connected layer and 100 hidden units.

from keras.models import Sequential
from keras.layers import BatchNormalization, Conv2D, Activation, MaxPooling2D, Dense, GlobalAveragePooling2D

model = Sequential()
model.add(Dense(100, activation="relu", input_shape=(96*96,)))
model.add(Activation('relu'))
model.add(Dense(30))

Now let's compile the model and run the training.

from keras import optimizers

sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(optimizer=sgd, loss='mse', metrics=['accuracy'])
epochs = 200
history = model.fit(X_train.reshape(y_train.shape[0], -1), y_train, 
                 validation_split=0.2, shuffle=True, 
                 epochs=epochs, batch_size=20)

Train on 1712 samples, validate on 428 samples
Epoch 1/200
1712/1712 [==============================] - 1s - loss: 0.0173 - acc: 0.4836 - val_loss: 0.0542 - val_acc: 0.0794
Epoch 2/200
1712/1712 [==============================] - 1s - loss: 0.0170 - acc: 0.4842 - val_loss: 0.0554 - val_acc: 0.0911
Epoch 3/200
1712/1712 [==============================] - 1s - loss: 0.0167 - acc: 0.4842 - val_loss: 0.0546 - val_acc: 0.0888
Epoch 4/200
1712/1712 [==============================] - ETA: 0s - loss: 0.0165 - acc: 0.478 - 1s - loss: 0.0165 - acc: 0.4772 - val_loss: 0.0531 - val_acc: 0.1262
Epoch 5/200
1712/1712 [==============================] - 1s - loss: 0.0161 - acc: 0.4959 - val_loss: 0.0623 - val_acc: 0.0864
Epoch 6/200
1712/1712 [==============================] - 1s - loss: 0.0159 - acc: 0.4813 - val_loss: 0.0523 - val_acc: 0.1075
Epoch 7/200
1712/1712 [==============================] - 1s - loss: 0.0157 - acc: 0.4942 - val_loss: 0.0526 - val_acc: 0.1145
Epoch 8/200
1712/1712 [==============================] - 1s - loss: 0.0155 - acc: 0.4953 - val_loss: 0.0537 - val_acc: 0.1051
Epoch 9/200
1712/1712 [==============================] - 1s - loss: 0.0152 - acc: 0.5006 - val_loss: 0.0503 - val_acc: 0.1472
Epoch 10/200
1712/1712 [==============================] - 1s - loss: 0.0151 - acc: 0.5006 - val_loss: 0.0531 - val_acc: 0.1168
Epoch 11/200
1712/1712 [==============================] - 1s - loss: 0.0148 - acc: 0.5041 - val_loss: 0.0530 - val_acc: 0.11450.496 - ETA: 0s - loss: 0.0151 - a
Epoch 12/200
1712/1712 [==============================] - 1s - loss: 0.0148 - acc: 0.4994 - val_loss: 0.0521 - val_acc: 0.1098
Epoch 13/200
1712/1712 [==============================] - 1s - loss: 0.0144 - acc: 0.5076 - val_loss: 0.0505 - val_acc: 0.1238
Epoch 14/200
1712/1712 [==============================] - 1s - loss: 0.0145 - acc: 0.5152 - val_loss: 0.0505 - val_acc: 0.1215
Epoch 15/200
1712/1712 [==============================] - 1s - loss: 0.0144 - acc: 0.5088 - val_loss: 0.0501 - val_acc: 0.1145
Epoch 16/200
1712/1712 [==============================] - 2s - loss: 0.0138 - acc: 0.5140 - val_loss: 0.0491 - val_acc: 0.1355
Epoch 17/200
1712/1712 [==============================] - 1s - loss: 0.0140 - acc: 0.5239 - val_loss: 0.0562 - val_acc: 0.1098
Epoch 18/200
1712/1712 [==============================] - 1s - loss: 0.0138 - acc: 0.5129 - val_loss: 0.0532 - val_acc: 0.1262
Epoch 19/200
1712/1712 [==============================] - 1s - loss: 0.0138 - acc: 0.5175 - val_loss: 0.0486 - val_acc: 0.1355
Epoch 20/200
1712/1712 [==============================] - 1s - loss: 0.0137 - acc: 0.5041 - val_loss: 0.0517 - val_acc: 0.1425
Epoch 21/200
1712/1712 [==============================] - 1s - loss: 0.0133 - acc: 0.5199 - val_loss: 0.0498 - val_acc: 0.1075
Epoch 22/200
1712/1712 [==============================] - 1s - loss: 0.0132 - acc: 0.5175 - val_loss: 0.0490 - val_acc: 0.1192
Epoch 23/200
1712/1712 [==============================] - 1s - loss: 0.0133 - acc: 0.5175 - val_loss: 0.0492 - val_acc: 0.0981
Epoch 24/200
1712/1712 [==============================] - 1s - loss: 0.0131 - acc: 0.5257 - val_loss: 0.0491 - val_acc: 0.1355
Epoch 25/200
1712/1712 [==============================] - 1s - loss: 0.0129 - acc: 0.5321 - val_loss: 0.0489 - val_acc: 0.1332
Epoch 26/200
1712/1712 [==============================] - 1s - loss: 0.0127 - acc: 0.5275 - val_loss: 0.0504 - val_acc: 0.1355
Epoch 27/200
1712/1712 [==============================] - 1s - loss: 0.0126 - acc: 0.5280 - val_loss: 0.0500 - val_acc: 0.1332
Epoch 28/200
1712/1712 [==============================] - 1s - loss: 0.0124 - acc: 0.5269 - val_loss: 0.0497 - val_acc: 0.1425
Epoch 29/200
1712/1712 [==============================] - 1s - loss: 0.0127 - acc: 0.5199 - val_loss: 0.0490 - val_acc: 0.1332
Epoch 30/200
1712/1712 [==============================] - 1s - loss: 0.0124 - acc: 0.5298 - val_loss: 0.0507 - val_acc: 0.1495
Epoch 31/200
1712/1712 [==============================] - 1s - loss: 0.0124 - acc: 0.5263 - val_loss: 0.0486 - val_acc: 0.1589
Epoch 32/200
1712/1712 [==============================] - 1s - loss: 0.0122 - acc: 0.5362 - val_loss: 0.0482 - val_acc: 0.1355
Epoch 33/200
1712/1712 [==============================] - 1s - loss: 0.0119 - acc: 0.5421 - val_loss: 0.0485 - val_acc: 0.1449
Epoch 34/200
1712/1712 [==============================] - 1s - loss: 0.0118 - acc: 0.5315 - val_loss: 0.0478 - val_acc: 0.1519
Epoch 35/200
1712/1712 [==============================] - 1s - loss: 0.0119 - acc: 0.5333 - val_loss: 0.0494 - val_acc: 0.1495
Epoch 36/200
1712/1712 [==============================] - 1s - loss: 0.0119 - acc: 0.5339 - val_loss: 0.0489 - val_acc: 0.1425
Epoch 37/200
1712/1712 [==============================] - 1s - loss: 0.0117 - acc: 0.5386 - val_loss: 0.0486 - val_acc: 0.1262
Epoch 38/200
1712/1712 [==============================] - 1s - loss: 0.0114 - acc: 0.5403 - val_loss: 0.0484 - val_acc: 0.1449
Epoch 39/200
1712/1712 [==============================] - 1s - loss: 0.0116 - acc: 0.5164 - val_loss: 0.0498 - val_acc: 0.1495
Epoch 40/200
1712/1712 [==============================] - 1s - loss: 0.0113 - acc: 0.5310 - val_loss: 0.0475 - val_acc: 0.1706
Epoch 41/200
1712/1712 [==============================] - 1s - loss: 0.0114 - acc: 0.5450 - val_loss: 0.0500 - val_acc: 0.1449
Epoch 42/200
1712/1712 [==============================] - 1s - loss: 0.0113 - acc: 0.5327 - val_loss: 0.0478 - val_acc: 0.1729
Epoch 43/200
1712/1712 [==============================] - 1s - loss: 0.0111 - acc: 0.5315 - val_loss: 0.0480 - val_acc: 0.1332
Epoch 44/200
1712/1712 [==============================] - 1s - loss: 0.0111 - acc: 0.5286 - val_loss: 0.0518 - val_acc: 0.1192
Epoch 45/200
1712/1712 [==============================] - 1s - loss: 0.0110 - acc: 0.5204 - val_loss: 0.0476 - val_acc: 0.1425
Epoch 46/200
1712/1712 [==============================] - 1s - loss: 0.0110 - acc: 0.5345 - val_loss: 0.0504 - val_acc: 0.1238
Epoch 47/200
1712/1712 [==============================] - 1s - loss: 0.0109 - acc: 0.5292 - val_loss: 0.0480 - val_acc: 0.1449
Epoch 48/200
1712/1712 [==============================] - 1s - loss: 0.0108 - acc: 0.5356 - val_loss: 0.0473 - val_acc: 0.1355
Epoch 49/200
1712/1712 [==============================] - 1s - loss: 0.0108 - acc: 0.5269 - val_loss: 0.0505 - val_acc: 0.1121
Epoch 50/200
1712/1712 [==============================] - 1s - loss: 0.0107 - acc: 0.5386 - val_loss: 0.0480 - val_acc: 0.1729
Epoch 51/200
1712/1712 [==============================] - 1s - loss: 0.0107 - acc: 0.5199 - val_loss: 0.0507 - val_acc: 0.1238
Epoch 52/200
1712/1712 [==============================] - 1s - loss: 0.0105 - acc: 0.5315 - val_loss: 0.0484 - val_acc: 0.1472
Epoch 53/200
1712/1712 [==============================] - 1s - loss: 0.0107 - acc: 0.5333 - val_loss: 0.0481 - val_acc: 0.1986
Epoch 54/200
1712/1712 [==============================] - 1s - loss: 0.0105 - acc: 0.5339 - val_loss: 0.0475 - val_acc: 0.1425
Epoch 55/200
1712/1712 [==============================] - 1s - loss: 0.0106 - acc: 0.5409 - val_loss: 0.0493 - val_acc: 0.1121
Epoch 56/200
1712/1712 [==============================] - 1s - loss: 0.0105 - acc: 0.5310 - val_loss: 0.0494 - val_acc: 0.1355
Epoch 57/200
1712/1712 [==============================] - 1s - loss: 0.0104 - acc: 0.5350 - val_loss: 0.0481 - val_acc: 0.1729
Epoch 58/200
1712/1712 [==============================] - 1s - loss: 0.0102 - acc: 0.5415 - val_loss: 0.0501 - val_acc: 0.1752
Epoch 59/200
1712/1712 [==============================] - 1s - loss: 0.0102 - acc: 0.5333 - val_loss: 0.0479 - val_acc: 0.1379
Epoch 60/200
1712/1712 [==============================] - 1s - loss: 0.0099 - acc: 0.5415 - val_loss: 0.0477 - val_acc: 0.1332
Epoch 61/200
1712/1712 [==============================] - 1s - loss: 0.0099 - acc: 0.5327 - val_loss: 0.0501 - val_acc: 0.1332
Epoch 62/200
1712/1712 [==============================] - 1s - loss: 0.0100 - acc: 0.5461 - val_loss: 0.0510 - val_acc: 0.1449
Epoch 63/200
1712/1712 [==============================] - 1s - loss: 0.0100 - acc: 0.5368 - val_loss: 0.0497 - val_acc: 0.1355
Epoch 64/200
1712/1712 [==============================] - 1s - loss: 0.0101 - acc: 0.5362 - val_loss: 0.0504 - val_acc: 0.1495
Epoch 65/200
1712/1712 [==============================] - 1s - loss: 0.0100 - acc: 0.5333 - val_loss: 0.0491 - val_acc: 0.1589
Epoch 66/200
1712/1712 [==============================] - 1s - loss: 0.0098 - acc: 0.5310 - val_loss: 0.0482 - val_acc: 0.1519
Epoch 67/200
1712/1712 [==============================] - 1s - loss: 0.0098 - acc: 0.5292 - val_loss: 0.0484 - val_acc: 0.1472
Epoch 68/200
1712/1712 [==============================] - 1s - loss: 0.0096 - acc: 0.5321 - val_loss: 0.0486 - val_acc: 0.1846
Epoch 69/200
1712/1712 [==============================] - 1s - loss: 0.0096 - acc: 0.5356 - val_loss: 0.0475 - val_acc: 0.1752
Epoch 70/200
1712/1712 [==============================] - 1s - loss: 0.0097 - acc: 0.5275 - val_loss: 0.0490 - val_acc: 0.1308
Epoch 71/200
1712/1712 [==============================] - 1s - loss: 0.0098 - acc: 0.5275 - val_loss: 0.0475 - val_acc: 0.1776
Epoch 72/200
1712/1712 [==============================] - 1s - loss: 0.0096 - acc: 0.5345 - val_loss: 0.0484 - val_acc: 0.1379
Epoch 73/200
1712/1712 [==============================] - 1s - loss: 0.0096 - acc: 0.5239 - val_loss: 0.0475 - val_acc: 0.1682
Epoch 74/200
1712/1712 [==============================] - 1s - loss: 0.0096 - acc: 0.5386 - val_loss: 0.0484 - val_acc: 0.1472
Epoch 75/200
1712/1712 [==============================] - 1s - loss: 0.0096 - acc: 0.5333 - val_loss: 0.0540 - val_acc: 0.1542
Epoch 76/200
1712/1712 [==============================] - 1s - loss: 0.0100 - acc: 0.5193 - val_loss: 0.0479 - val_acc: 0.1612
Epoch 77/200
1712/1712 [==============================] - 1s - loss: 0.0094 - acc: 0.5368 - val_loss: 0.0475 - val_acc: 0.1565
Epoch 78/200
1712/1712 [==============================] - 1s - loss: 0.0096 - acc: 0.5339 - val_loss: 0.0479 - val_acc: 0.1799
Epoch 79/200
1712/1712 [==============================] - 1s - loss: 0.0094 - acc: 0.5403 - val_loss: 0.0482 - val_acc: 0.1636
Epoch 80/200
1712/1712 [==============================] - 1s - loss: 0.0093 - acc: 0.5350 - val_loss: 0.0471 - val_acc: 0.1846
Epoch 81/200
1712/1712 [==============================] - 1s - loss: 0.0094 - acc: 0.5391 - val_loss: 0.0494 - val_acc: 0.1752
Epoch 82/200
1712/1712 [==============================] - 1s - loss: 0.0093 - acc: 0.5239 - val_loss: 0.0490 - val_acc: 0.1472
Epoch 83/200
1712/1712 [==============================] - 1s - loss: 0.0094 - acc: 0.5350 - val_loss: 0.0491 - val_acc: 0.1379
Epoch 84/200
1712/1712 [==============================] - 1s - loss: 0.0091 - acc: 0.5397 - val_loss: 0.0485 - val_acc: 0.1542
Epoch 85/200
1712/1712 [==============================] - 1s - loss: 0.0093 - acc: 0.5339 - val_loss: 0.0490 - val_acc: 0.1589
Epoch 86/200
1712/1712 [==============================] - 1s - loss: 0.0090 - acc: 0.5280 - val_loss: 0.0487 - val_acc: 0.1636
Epoch 87/200
1712/1712 [==============================] - 1s - loss: 0.0090 - acc: 0.5345 - val_loss: 0.0476 - val_acc: 0.1355
Epoch 88/200
1712/1712 [==============================] - 1s - loss: 0.0092 - acc: 0.5350 - val_loss: 0.0475 - val_acc: 0.1565
Epoch 89/200
1712/1712 [==============================] - 1s - loss: 0.0091 - acc: 0.5368 - val_loss: 0.0495 - val_acc: 0.1729
Epoch 90/200
1712/1712 [==============================] - 1s - loss: 0.0092 - acc: 0.5450 - val_loss: 0.0485 - val_acc: 0.1542
Epoch 91/200
1712/1712 [==============================] - 2s - loss: 0.0090 - acc: 0.5298 - val_loss: 0.0479 - val_acc: 0.1589
Epoch 92/200
1712/1712 [==============================] - 1s - loss: 0.0090 - acc: 0.5426 - val_loss: 0.0489 - val_acc: 0.1893
Epoch 93/200
1712/1712 [==============================] - 1s - loss: 0.0089 - acc: 0.5321 - val_loss: 0.0470 - val_acc: 0.1519
Epoch 94/200
1712/1712 [==============================] - 1s - loss: 0.0091 - acc: 0.5345 - val_loss: 0.0482 - val_acc: 0.1706
Epoch 95/200
1712/1712 [==============================] - 1s - loss: 0.0089 - acc: 0.5537 - val_loss: 0.0483 - val_acc: 0.1636
Epoch 96/200
1712/1712 [==============================] - 1s - loss: 0.0089 - acc: 0.5345 - val_loss: 0.0496 - val_acc: 0.1542
Epoch 97/200
1712/1712 [==============================] - 1s - loss: 0.0088 - acc: 0.5386 - val_loss: 0.0489 - val_acc: 0.1542
Epoch 98/200
1712/1712 [==============================] - 1s - loss: 0.0088 - acc: 0.5444 - val_loss: 0.0522 - val_acc: 0.1285
Epoch 99/200
1712/1712 [==============================] - 1s - loss: 0.0089 - acc: 0.5386 - val_loss: 0.0485 - val_acc: 0.1542
Epoch 100/200
1712/1712 [==============================] - 1s - loss: 0.0089 - acc: 0.5426 - val_loss: 0.0480 - val_acc: 0.1822
Epoch 101/200
1712/1712 [==============================] - 1s - loss: 0.0087 - acc: 0.5374 - val_loss: 0.0503 - val_acc: 0.2103
Epoch 102/200
1712/1712 [==============================] - 1s - loss: 0.0087 - acc: 0.5356 - val_loss: 0.0489 - val_acc: 0.1776
Epoch 103/200
1712/1712 [==============================] - 1s - loss: 0.0087 - acc: 0.5461 - val_loss: 0.0484 - val_acc: 0.1706
Epoch 104/200
1712/1712 [==============================] - 1s - loss: 0.0086 - acc: 0.5409 - val_loss: 0.0478 - val_acc: 0.1449
Epoch 105/200
1712/1712 [==============================] - 1s - loss: 0.0088 - acc: 0.5368 - val_loss: 0.0479 - val_acc: 0.1612
Epoch 106/200
1712/1712 [==============================] - 1s - loss: 0.0086 - acc: 0.5362 - val_loss: 0.0483 - val_acc: 0.1636
Epoch 107/200
1712/1712 [==============================] - 1s - loss: 0.0085 - acc: 0.5356 - val_loss: 0.0483 - val_acc: 0.2056
Epoch 108/200
1712/1712 [==============================] - 1s - loss: 0.0087 - acc: 0.5421 - val_loss: 0.0489 - val_acc: 0.1659
Epoch 109/200
1712/1712 [==============================] - 1s - loss: 0.0084 - acc: 0.5421 - val_loss: 0.0478 - val_acc: 0.1682
Epoch 110/200
1712/1712 [==============================] - 1s - loss: 0.0087 - acc: 0.5415 - val_loss: 0.0488 - val_acc: 0.1799
Epoch 111/200
1712/1712 [==============================] - 1s - loss: 0.0086 - acc: 0.5421 - val_loss: 0.0480 - val_acc: 0.1565
Epoch 112/200
1712/1712 [==============================] - 1s - loss: 0.0086 - acc: 0.5426 - val_loss: 0.0502 - val_acc: 0.1495
Epoch 113/200
1712/1712 [==============================] - 1s - loss: 0.0087 - acc: 0.5350 - val_loss: 0.0481 - val_acc: 0.1846
Epoch 114/200
1712/1712 [==============================] - 1s - loss: 0.0085 - acc: 0.5426 - val_loss: 0.0489 - val_acc: 0.1565
Epoch 115/200
1712/1712 [==============================] - 1s - loss: 0.0085 - acc: 0.5374 - val_loss: 0.0488 - val_acc: 0.1776
Epoch 116/200
1712/1712 [==============================] - 1s - loss: 0.0085 - acc: 0.5456 - val_loss: 0.0481 - val_acc: 0.1869
Epoch 117/200
1712/1712 [==============================] - 1s - loss: 0.0084 - acc: 0.5339 - val_loss: 0.0534 - val_acc: 0.1449
Epoch 118/200
1712/1712 [==============================] - 1s - loss: 0.0085 - acc: 0.5432 - val_loss: 0.0485 - val_acc: 0.1659
Epoch 119/200
1712/1712 [==============================] - 1s - loss: 0.0085 - acc: 0.5432 - val_loss: 0.0499 - val_acc: 0.1308
Epoch 120/200
1712/1712 [==============================] - 1s - loss: 0.0083 - acc: 0.5374 - val_loss: 0.0490 - val_acc: 0.1612
Epoch 121/200
1712/1712 [==============================] - 1s - loss: 0.0085 - acc: 0.5345 - val_loss: 0.0497 - val_acc: 0.1355
Epoch 122/200
1712/1712 [==============================] - 1s - loss: 0.0085 - acc: 0.5397 - val_loss: 0.0497 - val_acc: 0.1542
Epoch 123/200
1712/1712 [==============================] - 1s - loss: 0.0085 - acc: 0.5374 - val_loss: 0.0482 - val_acc: 0.1495
Epoch 124/200
1712/1712 [==============================] - 1s - loss: 0.0083 - acc: 0.5386 - val_loss: 0.0487 - val_acc: 0.1425
Epoch 125/200
1712/1712 [==============================] - 1s - loss: 0.0083 - acc: 0.5397 - val_loss: 0.0480 - val_acc: 0.1542
Epoch 126/200
1712/1712 [==============================] - 1s - loss: 0.0084 - acc: 0.5386 - val_loss: 0.0496 - val_acc: 0.1752
Epoch 127/200
1712/1712 [==============================] - 1s - loss: 0.0082 - acc: 0.5350 - val_loss: 0.0502 - val_acc: 0.1355
Epoch 128/200

1712/1712 [==============================] - 1s - loss: 0.0083 - acc: 0.5356 - val_loss: 0.0484 - val_acc: 0.1939
Epoch 129/200
1712/1712 [==============================] - 1s - loss: 0.0081 - acc: 0.5380 - val_loss: 0.0526 - val_acc: 0.1659
Epoch 130/200
1712/1712 [==============================] - 1s - loss: 0.0083 - acc: 0.5409 - val_loss: 0.0484 - val_acc: 0.1589
Epoch 131/200
1712/1712 [==============================] - 1s - loss: 0.0081 - acc: 0.5491 - val_loss: 0.0494 - val_acc: 0.1425
Epoch 132/200
1712/1712 [==============================] - 1s - loss: 0.0081 - acc: 0.5473 - val_loss: 0.0487 - val_acc: 0.1706
Epoch 133/200
1712/1712 [==============================] - 1s - loss: 0.0082 - acc: 0.5508 - val_loss: 0.0508 - val_acc: 0.1636
Epoch 134/200
1712/1712 [==============================] - 1s - loss: 0.0081 - acc: 0.5643 - val_loss: 0.0525 - val_acc: 0.1542
Epoch 135/200
1712/1712 [==============================] - 1s - loss: 0.0082 - acc: 0.5403 - val_loss: 0.0495 - val_acc: 0.1822
Epoch 136/200
1712/1712 [==============================] - 1s - loss: 0.0082 - acc: 0.5456 - val_loss: 0.0479 - val_acc: 0.1893
Epoch 137/200
1712/1712 [==============================] - 1s - loss: 0.0082 - acc: 0.5432 - val_loss: 0.0492 - val_acc: 0.1729
Epoch 138/200
1712/1712 [==============================] - 1s - loss: 0.0082 - acc: 0.5461 - val_loss: 0.0496 - val_acc: 0.1636
Epoch 139/200
1712/1712 [==============================] - 1s - loss: 0.0081 - acc: 0.5380 - val_loss: 0.0481 - val_acc: 0.1495
Epoch 140/200
1712/1712 [==============================] - 1s - loss: 0.0080 - acc: 0.5461 - val_loss: 0.0492 - val_acc: 0.1565
Epoch 141/200
1712/1712 [==============================] - 1s - loss: 0.0080 - acc: 0.5315 - val_loss: 0.0511 - val_acc: 0.1612
Epoch 142/200
1712/1712 [==============================] - 1s - loss: 0.0079 - acc: 0.5444 - val_loss: 0.0492 - val_acc: 0.1495
Epoch 143/200
1712/1712 [==============================] - 1s - loss: 0.0080 - acc: 0.5426 - val_loss: 0.0484 - val_acc: 0.1565
Epoch 144/200
1712/1712 [==============================] - 1s - loss: 0.0079 - acc: 0.5496 - val_loss: 0.0487 - val_acc: 0.1565
Epoch 145/200
1712/1712 [==============================] - 1s - loss: 0.0080 - acc: 0.5356 - val_loss: 0.0495 - val_acc: 0.1799
Epoch 146/200
1712/1712 [==============================] - 1s - loss: 0.0079 - acc: 0.5532 - val_loss: 0.0526 - val_acc: 0.1519
Epoch 147/200
1712/1712 [==============================] - 1s - loss: 0.0082 - acc: 0.5356 - val_loss: 0.0499 - val_acc: 0.1799
Epoch 148/200
1712/1712 [==============================] - 1s - loss: 0.0080 - acc: 0.5479 - val_loss: 0.0490 - val_acc: 0.1893
Epoch 149/200
1712/1712 [==============================] - 1s - loss: 0.0082 - acc: 0.5491 - val_loss: 0.0508 - val_acc: 0.1379
Epoch 150/200
1712/1712 [==============================] - 1s - loss: 0.0083 - acc: 0.5374 - val_loss: 0.0505 - val_acc: 0.1565
Epoch 151/200
1712/1712 [==============================] - 1s - loss: 0.0080 - acc: 0.5537 - val_loss: 0.0487 - val_acc: 0.1846
Epoch 152/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5374 - val_loss: 0.0494 - val_acc: 0.1659
Epoch 153/200
1712/1712 [==============================] - 1s - loss: 0.0081 - acc: 0.5397 - val_loss: 0.0482 - val_acc: 0.1636
Epoch 154/200
1712/1712 [==============================] - 1s - loss: 0.0079 - acc: 0.5450 - val_loss: 0.0480 - val_acc: 0.1776
Epoch 155/200
1712/1712 [==============================] - 1s - loss: 0.0081 - acc: 0.5520 - val_loss: 0.0518 - val_acc: 0.1402
Epoch 156/200
1712/1712 [==============================] - 1s - loss: 0.0079 - acc: 0.5485 - val_loss: 0.0508 - val_acc: 0.1565
Epoch 157/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5491 - val_loss: 0.0494 - val_acc: 0.1519
Epoch 158/200
1712/1712 [==============================] - 2s - loss: 0.0079 - acc: 0.5496 - val_loss: 0.0493 - val_acc: 0.1752
Epoch 159/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5520 - val_loss: 0.0481 - val_acc: 0.1776
Epoch 160/200
1712/1712 [==============================] - ETA: 0s - loss: 0.0080 - acc: 0.550 - 1s - loss: 0.0080 - acc: 0.5485 - val_loss: 0.0496 - val_acc: 0.1706
Epoch 161/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5380 - val_loss: 0.0498 - val_acc: 0.1799
Epoch 162/200
1712/1712 [==============================] - 1s - loss: 0.0079 - acc: 0.5310 - val_loss: 0.0504 - val_acc: 0.2220
Epoch 163/200
1712/1712 [==============================] - 1s - loss: 0.0079 - acc: 0.5421 - val_loss: 0.0495 - val_acc: 0.1542
Epoch 164/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5555 - val_loss: 0.0502 - val_acc: 0.1636
Epoch 165/200
1712/1712 [==============================] - 1s - loss: 0.0077 - acc: 0.5473 - val_loss: 0.0503 - val_acc: 0.1869
Epoch 166/200
1712/1712 [==============================] - 1s - loss: 0.0076 - acc: 0.5467 - val_loss: 0.0490 - val_acc: 0.1472
Epoch 167/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5397 - val_loss: 0.0496 - val_acc: 0.1659
Epoch 168/200
1712/1712 [==============================] - 1s - loss: 0.0076 - acc: 0.5485 - val_loss: 0.0492 - val_acc: 0.1682
Epoch 169/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5514 - val_loss: 0.0493 - val_acc: 0.1776
Epoch 170/200
1712/1712 [==============================] - 1s - loss: 0.0076 - acc: 0.5514 - val_loss: 0.0487 - val_acc: 0.1822
Epoch 171/200
1712/1712 [==============================] - 1s - loss: 0.0077 - acc: 0.5461 - val_loss: 0.0495 - val_acc: 0.1612
Epoch 172/200
1712/1712 [==============================] - 1s - loss: 0.0076 - acc: 0.5590 - val_loss: 0.0501 - val_acc: 0.1612
Epoch 173/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5380 - val_loss: 0.0508 - val_acc: 0.1565
Epoch 174/200
1712/1712 [==============================] - 1s - loss: 0.0076 - acc: 0.5421 - val_loss: 0.0505 - val_acc: 0.1636
Epoch 175/200
1712/1712 [==============================] - 1s - loss: 0.0076 - acc: 0.5479 - val_loss: 0.0487 - val_acc: 0.2033
Epoch 176/200
1712/1712 [==============================] - 1s - loss: 0.0077 - acc: 0.5467 - val_loss: 0.0503 - val_acc: 0.1589
Epoch 177/200
1712/1712 [==============================] - 1s - loss: 0.0080 - acc: 0.5508 - val_loss: 0.0487 - val_acc: 0.1589
Epoch 178/200
1712/1712 [==============================] - 1s - loss: 0.0075 - acc: 0.5555 - val_loss: 0.0516 - val_acc: 0.1612
Epoch 179/200
1712/1712 [==============================] - 2s - loss: 0.0076 - acc: 0.5543 - val_loss: 0.0499 - val_acc: 0.2220
Epoch 180/200
1712/1712 [==============================] - 1s - loss: 0.0077 - acc: 0.5543 - val_loss: 0.0500 - val_acc: 0.1449
Epoch 181/200
1712/1712 [==============================] - 1s - loss: 0.0075 - acc: 0.5532 - val_loss: 0.0487 - val_acc: 0.1636
Epoch 182/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5456 - val_loss: 0.0490 - val_acc: 0.2009
Epoch 183/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5467 - val_loss: 0.0510 - val_acc: 0.1612
Epoch 184/200
1712/1712 [==============================] - 1s - loss: 0.0077 - acc: 0.5456 - val_loss: 0.0536 - val_acc: 0.1963
Epoch 185/200
1712/1712 [==============================] - 1s - loss: 0.0075 - acc: 0.5502 - val_loss: 0.0493 - val_acc: 0.1565
Epoch 186/200
1712/1712 [==============================] - 1s - loss: 0.0074 - acc: 0.5473 - val_loss: 0.0496 - val_acc: 0.1729
Epoch 187/200
1712/1712 [==============================] - 1s - loss: 0.0075 - acc: 0.5479 - val_loss: 0.0496 - val_acc: 0.1636
Epoch 188/200
1712/1712 [==============================] - 1s - loss: 0.0076 - acc: 0.5461 - val_loss: 0.0501 - val_acc: 0.1706
Epoch 189/200
1712/1712 [==============================] - 1s - loss: 0.0078 - acc: 0.5397 - val_loss: 0.0504 - val_acc: 0.1542
Epoch 190/200
1712/1712 [==============================] - 1s - loss: 0.0074 - acc: 0.5537 - val_loss: 0.0502 - val_acc: 0.1589
Epoch 191/200
1712/1712 [==============================] - 1s - loss: 0.0074 - acc: 0.5421 - val_loss: 0.0490 - val_acc: 0.1659
Epoch 192/200
1712/1712 [==============================] - 1s - loss: 0.0076 - acc: 0.5461 - val_loss: 0.0484 - val_acc: 0.1519
Epoch 193/200
1712/1712 [==============================] - 1s - loss: 0.0074 - acc: 0.5532 - val_loss: 0.0506 - val_acc: 0.1706
Epoch 194/200
1712/1712 [==============================] - 1s - loss: 0.0074 - acc: 0.5502 - val_loss: 0.0493 - val_acc: 0.1939
Epoch 195/200
1712/1712 [==============================] - 1s - loss: 0.0074 - acc: 0.5578 - val_loss: 0.0495 - val_acc: 0.1659
Epoch 196/200
1712/1712 [==============================] - 1s - loss: 0.0077 - acc: 0.5514 - val_loss: 0.0504 - val_acc: 0.1589
Epoch 197/200
1712/1712 [==============================] - 1s - loss: 0.0073 - acc: 0.5567 - val_loss: 0.0510 - val_acc: 0.1565
Epoch 198/200
1712/1712 [==============================] - 1s - loss: 0.0075 - acc: 0.5491 - val_loss: 0.0498 - val_acc: 0.1776
Epoch 199/200
1712/1712 [==============================] - 1s - loss: 0.0073 - acc: 0.5567 - val_loss: 0.0517 - val_acc: 0.1449
Epoch 200/200
1712/1712 [==============================] - 1s - loss: 0.0074 - acc: 0.5456 - val_loss: 0.0493 - val_acc: 0.1612

Let's plot our training curves with this model.

# summarize history for accuracy
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

What we see here is that with this model, the learning quickly gets on a plateau. How can we improve this? There are a lot of options:

• adjust the optimizer settings
  • learning rate
  • batch size
  • momentum
• change the model

However, one things that is pretty clear from the above plot is that our model overfits: the train and test losses are not comparable (the test loss is 3 times higher). Let's see what the results of the net are on some samples from our data.

img = X_train[0, :, :, :].reshape(1, -1)
predictions = model.predict(img)

img

array([[ 0.93333333,  0.9254902 ,  0.92941176, ...,  0.2745098 ,
         0.29411765,  0.35294118]])

xy_predictions = output_pipe.inverse_transform(predictions).reshape(15, 2)

plt.imshow(X_train[0, :, :, 0], cmap='gray')
plt.plot(xy_predictions[:, 0], xy_predictions[:, 1], 'b*')

[<matplotlib.lines.Line2D at 0x11a96cb70>]

def plot_faces_with_keypoints_and_predictions(model, nrows=5, ncols=5, model_input='flat'):
    """Plots sampled faces with their truth and predictions."""
    selection = np.random.choice(np.arange(X.shape[0]), size=(nrows*ncols), replace=False)
    fig, axes = plt.subplots(figsize=(10, 10), nrows=nrows, ncols=ncols)
    for ind, ax in zip(selection, axes.ravel()):
        img = X_train[ind, :, :, 0]
        if model_input == 'flat':
            predictions = model.predict(img.reshape(1, -1))
        else:
            predictions = model.predict(img[np.newaxis, :, :, np.newaxis])
        xy_predictions = output_pipe.inverse_transform(predictions).reshape(15, 2)
        ax.imshow(img, cmap='gray')
        ax.plot(xy_predictions[:, 0], xy_predictions[:, 1], 'bo')
        ax.axis('off')

plot_faces_with_keypoints_and_predictions(model)

Actually, this looks pretty good already. Let's try to train a more complicated model, this time following the initial model description found in Peter Skvarenina's article.

Towards more complicated models

from keras.layers import Dropout, Flatten

model = Sequential()
# input layer
model.add(BatchNormalization(input_shape=(96, 96, 1)))
model.add(Conv2D(24, (5, 5), kernel_initializer='he_normal'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Dropout(0.2))
# layer 2
model.add(Conv2D(36, (5, 5)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Dropout(0.2))
# layer 3
model.add(Conv2D(48, (5, 5)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Dropout(0.2))
# layer 4
model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Dropout(0.2))
# layer 5
model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(Flatten())
# layer 6
model.add(Dense(500, activation="relu"))
# layer 7
model.add(Dense(90, activation="relu"))
# layer 8
model.add(Dense(30))

sgd = optimizers.SGD(lr=0.1, decay=1e-6, momentum=0.95, nesterov=True)
model.compile(optimizer=sgd, loss='mse', metrics=['accuracy'])
epochs = 50
history = model.fit(X_train, y_train, 
                 validation_split=0.2, shuffle=True, 
                 epochs=epochs, batch_size=20)

Train on 1712 samples, validate on 428 samples
Epoch 1/50
1712/1712 [==============================] - 36s - loss: 0.0261 - acc: 0.4112 - val_loss: 0.0544 - val_acc: 0.1332
Epoch 2/50
1712/1712 [==============================] - 35s - loss: 0.0219 - acc: 0.4282 - val_loss: 0.0485 - val_acc: 0.1519
Epoch 3/50
1712/1712 [==============================] - 39s - loss: 0.0198 - acc: 0.4445 - val_loss: 0.0447 - val_acc: 0.1402
Epoch 4/50
1712/1712 [==============================] - 38s - loss: 0.0181 - acc: 0.4609 - val_loss: 0.0426 - val_acc: 0.1449
Epoch 5/50
1712/1712 [==============================] - 37s - loss: 0.0167 - acc: 0.4609 - val_loss: 0.0410 - val_acc: 0.1706
Epoch 6/50
1712/1712 [==============================] - 37s - loss: 0.0156 - acc: 0.4918 - val_loss: 0.0417 - val_acc: 0.1565
Epoch 7/50
1712/1712 [==============================] - 36s - loss: 0.0144 - acc: 0.4912 - val_loss: 0.0389 - val_acc: 0.1589
Epoch 8/50
1712/1712 [==============================] - 36s - loss: 0.0139 - acc: 0.5000 - val_loss: 0.0365 - val_acc: 0.1916
Epoch 9/50
1712/1712 [==============================] - 36s - loss: 0.0131 - acc: 0.5146 - val_loss: 0.0380 - val_acc: 0.1355
Epoch 10/50
1712/1712 [==============================] - 38s - loss: 0.0127 - acc: 0.4942 - val_loss: 0.0347 - val_acc: 0.1916
Epoch 11/50
1712/1712 [==============================] - 38s - loss: 0.0122 - acc: 0.5321 - val_loss: 0.0342 - val_acc: 0.1939
Epoch 12/50
1712/1712 [==============================] - 39s - loss: 0.0117 - acc: 0.5099 - val_loss: 0.0346 - val_acc: 0.2056
Epoch 13/50
1712/1712 [==============================] - 40s - loss: 0.0113 - acc: 0.5397 - val_loss: 0.0347 - val_acc: 0.2009
Epoch 14/50
1712/1712 [==============================] - 39s - loss: 0.0111 - acc: 0.5257 - val_loss: 0.0332 - val_acc: 0.2150
Epoch 15/50
1712/1712 [==============================] - 46s - loss: 0.0106 - acc: 0.5199 - val_loss: 0.0341 - val_acc: 0.1659
Epoch 16/50
1712/1712 [==============================] - 42s - loss: 0.0104 - acc: 0.5251 - val_loss: 0.0321 - val_acc: 0.2056
Epoch 17/50
1712/1712 [==============================] - 41s - loss: 0.0101 - acc: 0.5491 - val_loss: 0.0314 - val_acc: 0.2360
Epoch 18/50
1712/1712 [==============================] - 40s - loss: 0.0099 - acc: 0.5561 - val_loss: 0.0313 - val_acc: 0.1939
Epoch 19/50
1712/1712 [==============================] - 40s - loss: 0.0096 - acc: 0.5555 - val_loss: 0.0309 - val_acc: 0.2290
Epoch 20/50
1712/1712 [==============================] - 41s - loss: 0.0097 - acc: 0.5584 - val_loss: 0.0309 - val_acc: 0.2313
Epoch 21/50
1712/1712 [==============================] - 40s - loss: 0.0095 - acc: 0.5584 - val_loss: 0.0300 - val_acc: 0.2430
Epoch 22/50
1712/1712 [==============================] - 40s - loss: 0.0090 - acc: 0.5631 - val_loss: 0.0304 - val_acc: 0.2313
Epoch 23/50
1712/1712 [==============================] - 40s - loss: 0.0090 - acc: 0.5555 - val_loss: 0.0297 - val_acc: 0.2126
Epoch 24/50
1712/1712 [==============================] - 38s - loss: 0.0090 - acc: 0.5672 - val_loss: 0.0292 - val_acc: 0.2360
Epoch 25/50
1712/1712 [==============================] - 40s - loss: 0.0088 - acc: 0.5683 - val_loss: 0.0293 - val_acc: 0.2453
Epoch 26/50
1712/1712 [==============================] - 38s - loss: 0.0086 - acc: 0.5718 - val_loss: 0.0300 - val_acc: 0.2220
Epoch 27/50
1712/1712 [==============================] - 37s - loss: 0.0084 - acc: 0.5707 - val_loss: 0.0288 - val_acc: 0.2430
Epoch 28/50
1712/1712 [==============================] - 43s - loss: 0.0083 - acc: 0.5742 - val_loss: 0.0288 - val_acc: 0.2103
Epoch 29/50
1712/1712 [==============================] - 39s - loss: 0.0081 - acc: 0.5940 - val_loss: 0.0282 - val_acc: 0.2313
Epoch 30/50
1712/1712 [==============================] - 37s - loss: 0.0082 - acc: 0.5911 - val_loss: 0.0284 - val_acc: 0.2547
Epoch 31/50
1712/1712 [==============================] - 37s - loss: 0.0079 - acc: 0.5911 - val_loss: 0.0279 - val_acc: 0.2617
Epoch 32/50
1712/1712 [==============================] - 44s - loss: 0.0079 - acc: 0.5847 - val_loss: 0.0277 - val_acc: 0.2640
Epoch 33/50
1712/1712 [==============================] - 39s - loss: 0.0079 - acc: 0.5975 - val_loss: 0.0275 - val_acc: 0.2780
Epoch 34/50
1712/1712 [==============================] - 37s - loss: 0.0078 - acc: 0.5864 - val_loss: 0.0266 - val_acc: 0.2500
Epoch 35/50
1712/1712 [==============================] - 39s - loss: 0.0076 - acc: 0.5981 - val_loss: 0.0277 - val_acc: 0.2734
Epoch 36/50
1712/1712 [==============================] - 44s - loss: 0.0076 - acc: 0.5946 - val_loss: 0.0265 - val_acc: 0.2664
Epoch 37/50
1712/1712 [==============================] - 54s - loss: 0.0074 - acc: 0.6016 - val_loss: 0.0267 - val_acc: 0.2453
Epoch 38/50
1712/1712 [==============================] - 41s - loss: 0.0075 - acc: 0.5964 - val_loss: 0.0269 - val_acc: 0.2617
Epoch 39/50
1712/1712 [==============================] - 41s - loss: 0.0072 - acc: 0.6157 - val_loss: 0.0269 - val_acc: 0.2664
Epoch 40/50
1712/1712 [==============================] - 41s - loss: 0.0072 - acc: 0.6162 - val_loss: 0.0267 - val_acc: 0.2500
Epoch 41/50
1712/1712 [==============================] - 37s - loss: 0.0072 - acc: 0.6022 - val_loss: 0.0258 - val_acc: 0.2710
Epoch 42/50
1712/1712 [==============================] - 39s - loss: 0.0071 - acc: 0.6127 - val_loss: 0.0262 - val_acc: 0.2500
Epoch 43/50
1712/1712 [==============================] - 39s - loss: 0.0071 - acc: 0.6256 - val_loss: 0.0265 - val_acc: 0.2477
Epoch 44/50
1712/1712 [==============================] - 37s - loss: 0.0070 - acc: 0.6186 - val_loss: 0.0263 - val_acc: 0.2640
Epoch 45/50
1712/1712 [==============================] - 38s - loss: 0.0068 - acc: 0.6390 - val_loss: 0.0260 - val_acc: 0.2593
Epoch 46/50
1712/1712 [==============================] - 37s - loss: 0.0069 - acc: 0.6180 - val_loss: 0.0263 - val_acc: 0.2383
Epoch 47/50
1712/1712 [==============================] - 37s - loss: 0.0070 - acc: 0.6262 - val_loss: 0.0256 - val_acc: 0.2477
Epoch 48/50
1712/1712 [==============================] - 36s - loss: 0.0067 - acc: 0.6250 - val_loss: 0.0261 - val_acc: 0.2523
Epoch 49/50
1712/1712 [==============================] - 37s - loss: 0.0068 - acc: 0.6098 - val_loss: 0.0265 - val_acc: 0.2757
Epoch 50/50
1712/1712 [==============================] - 39s - loss: 0.0065 - acc: 0.6291 - val_loss: 0.0259 - val_acc: 0.2477

Let's see that in curves:

# summarize history for accuracy
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

How good is the result?

plot_faces_with_keypoints_and_predictions(model, model_input='2d')

If you ask me, that's already pretty good. Even though we didn't reach the performance advertised in Peter Skvarenina's blog post, with 80% validation accuracy. I wonder what he used to reach that level of performance: longer training? better settings?

Let's move on to the last section of this blog post: applications.

Applications

A face mask

A first thing we can do is to apply some sort of mask on top of the detected image. Let's draw a moustache over an image for example.

First, we need an image of a moustache.

import skimage.color
from skimage.filters import median

moustache = plt.imread('http://www.freeiconspng.com/uploads/moustache-png-by-spoonswagging-on-deviantart-1.png')
moustache = skimage.color.rgb2gray(moustache)

moustache = median(moustache, selem=np.ones((3, 3)))

/Users/kappamaki/anaconda/lib/python3.6/site-packages/skimage/util/dtype.py:122: UserWarning: Possible precision loss when converting from float64 to uint8
  .format(dtypeobj_in, dtypeobj_out))

Let's display it.

plt.imshow(moustache, cmap='gray')

<matplotlib.image.AxesImage at 0x1388b2198>

Now, let's extract the boundary of this moustache.

from skimage import measure
moustache_contour = measure.find_contours(moustache, 0.8)[0]
moustache_contour -= np.array([250, 250])

Now, let's write a function that plots a scaled moustache at a given position.

def plot_scaled_moustache(ax, center_xy, dx):
    """Plots a moustache scaled by its width, dx, on current ax."""
    moustache_scaled = moustache_contour.copy()
    moustache_scaled -= moustache_contour.min(axis=0)
    moustache_scaled /= moustache_scaled.max(axis=0)[1]
    deltas = moustache_scaled.max(axis=0) - moustache_scaled.min(axis=0)
    moustache_scaled -= np.array([deltas[0]/2, deltas[1]/2])
    moustache_scaled *= dx
    moustache_scaled += center_xy[::-1]
    ax.fill(moustache_scaled[:, 1], moustache_scaled[:, 0], "g", linewidth=4)

Let's test this:

ax = plt.gca()
plot_scaled_moustache(ax, np.array([2, 3]), dx=3)
ax.invert_yaxis()

Finally, we can integrate this with a function of the predicted points. We will use the mouth location and space the moustache using the size of the mouth.

def draw_moustache(predicted_points, ax):
    """Draws a moustache using the predicted face points."""
    dx = 2 * np.linalg.norm(predicted_points[12, :] - predicted_points[11, :])
    center_xy = predicted_points[13, :]
    plot_scaled_moustache(ax, center_xy, dx)

Let's try this with the first image from the training set.

img = X_train[0, :, :, :][np.newaxis, :, :, :]
predictions = model.predict(img)
xy_predictions = output_pipe.inverse_transform(predictions).reshape(15, 2)

fig, ax = plt.subplots()
ax.imshow(X_train[0, :, :, 0], cmap='gray')
draw_moustache(xy_predictions, ax)

Ok, looks good. Let's apply this to a grid of images.

def plot_faces_with_moustaches(model, nrows=5, ncols=5, model_input='flat'):
    """Plots sampled faces with their truth and predictions."""
    selection = np.random.choice(np.arange(X.shape[0]), size=(nrows*ncols), replace=False)
    fig, axes = plt.subplots(figsize=(10, 10), nrows=nrows, ncols=ncols)
    for ind, ax in zip(selection, axes.ravel()):
        img = X_train[ind, :, :, 0]
        if model_input == 'flat':
            predictions = model.predict(img.reshape(1, -1))
        else:
            predictions = model.predict(img[np.newaxis, :, :, np.newaxis])
        xy_predictions = output_pipe.inverse_transform(predictions).reshape(15, 2)
        ax.imshow(img, cmap='gray')
        draw_moustache(xy_predictions, ax)
        ax.axis('off')

plot_faces_with_moustaches(model, model_input='2d')

This is fun. There's a couple of ways we could better: adjust for face directions (the tilted faces in particular look strange). But that's already pretty nice. Let's make a gallery of famous faces with moustaches.

Famous faces with moustaches

Let's apply the skill of adding automated moustaches to some famous paintings.

portrait_urls = ["https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Mona_Lisa%2C_by_Leonardo_da_Vinci%2C_from_C2RMF_retouched.jpg/1024px-Mona_Lisa%2C_by_Leonardo_da_Vinci%2C_from_C2RMF_retouched.jpg",
                "https://upload.wikimedia.org/wikipedia/commons/thumb/d/d2/Hans_Holbein%2C_the_Younger_-_Sir_Thomas_More_-_Google_Art_Project.jpg/1280px-Hans_Holbein%2C_the_Younger_-_Sir_Thomas_More_-_Google_Art_Project.jpg",
                "https://upload.wikimedia.org/wikipedia/commons/b/b6/The_Blue_Boy.jpg",
                "https://upload.wikimedia.org/wikipedia/commons/thumb/2/2f/Thomas_Kerrich_%281748-1828%29%2C_by_Pompeo_Batoni.jpg/1280px-Thomas_Kerrich_%281748-1828%29%2C_by_Pompeo_Batoni.jpg",
                "https://upload.wikimedia.org/wikipedia/en/d/d6/GertrudeStein.JPG",
                "https://upload.wikimedia.org/wikipedia/commons/thumb/b/b0/Ambrogio_de_Predis_-_Girl_with_Cherries.jpg/1024px-Ambrogio_de_Predis_-_Girl_with_Cherries.jpg",
                "https://upload.wikimedia.org/wikipedia/commons/f/f8/Martin_Luther%2C_1529.jpg",
                "https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Pierre-Auguste_Renoir_110.jpg/1280px-Pierre-Auguste_Renoir_110.jpg"]

portraits = {}
for url in portrait_urls:
    if url not in portraits:
        portraits[url] = imread(url)

from skimage.io import imread
import cv2

face_cascade = cv2.CascadeClassifier('data/haarcascade_frontalface_default.xml')
fig, axes = plt.subplots(nrows=2, ncols=4, figsize=(12, 6))
for img, ax in zip(portraits.values(), axes.ravel()):
    gray = (skimage.color.rgb2gray(img) * 255).astype(dtype='uint8')
    bounding_boxes = face_cascade.detectMultiScale(gray, 1.25, 6)
    for (x,y,w,h) in bounding_boxes:
        roi_gray = gray[y:y+h, x:x+w]
        roi_rescaled = skimage.transform.resize(roi_gray, (96, 96))
        predictions = model.predict(roi_rescaled[np.newaxis, :, :, np.newaxis])
        xy_predictions = output_pipe.inverse_transform(predictions).reshape(15, 2)
        ax.imshow(roi_rescaled, cmap='gray')
        draw_moustache(xy_predictions, ax)
    ax.axis('off')

/Users/kappamaki/anaconda/lib/python3.6/site-packages/skimage/transform/_warps.py:84: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.
  warn("The default mode, 'constant', will be changed to 'reflect' in "

For comparison's sake, here are the original paintings:

face_cascade = cv2.CascadeClassifier('data/haarcascade_frontalface_default.xml')
fig, axes = plt.subplots(nrows=2, ncols=4, figsize=(12, 6))
for img, ax in zip(portraits.values(), axes.ravel()):
    ax.imshow(img)
    ax.axis('off')

Conclusions

Okay, that's it for this blog post. So what steps did we go through? We trained a model using Kaggle data, Keras and a deep convolutional neural network. The model was good enough that we could apply it to images from the internet without major changes.

After doing all this, I still feel that we only scratched the edge of what we could do with this. In particular, the neural network part was not very satisfying since I feel the model we trained could have been better. The reason I did not delve deeper into this (no pun intended) is that I don't own any GPU and hence the training takes quite a long time, which I was not willing to wait for better results.

As a takeaway from this post, I think the claim that a high school genius could do things like these on his own is indeed true. If you have the data, it seems that the machine learning models are powerful and simple enough to allow you to do things that were much more complicated in the past.

If I have time for a next post, I'd love to extend the work we did here but do transfer learning, using features from famous already trained neural networks.

This post was entirely written using the IPython notebook. Its content is BSD-licensed. You can see a static view or download this notebook with the help of nbviewer at 20170914_FacialKeypointsDetection.ipynb.

---

This article has been tagged with the following terms:

Machine Learning Image Processing Kaggle

---

• 
• 
• 
• 
• 

---

Comments