MINST手写数字识别（三）—— 使用antirectifier替换ReLU激活函数

这是一个来自官网的示例：https://github.com/keras-team/keras/blob/master/examples/antirectifier.py

与之前的MINST手写数字识别全连接网络相比，只是本实例使用antirectifier替换ReLU激活函数.

 '''The example demonstrates how to write custom layers for Keras.
 # Keras自定义层编写示范

 We build a custom activation layer called 'Antirectifier',
 建立了一个自定义的激活 'Antirectifier'(反校正)

 which modifies the shape of the tensor that passes through it.
 它修改通过它的张量的形状。

 We need to specify two methods: `compute_output_shape` and `call`.
 需要指定两种方法: `compute_output_shape` and `call`.

 Note that the same result can also be achieved via a Lambda layer.
 注意，同样的结果也可以通过Lambda层来实现

 Because our custom layer is written with primitives from the Keras
 自定义层是用keras底层编写的，

 backend (`K`), our code can run both on TensorFlow and Theano.
 代码可基于TensorFlow and Theano框架运行
 '''

 from __future__ import print_function
 import keras
 from keras.models import Sequential
 from keras import layers
 from keras.datasets import mnist
 from keras import backend as K

 class Antirectifier(layers.Layer):

     '''This is the combination of a sample-wise
     L2 normalization with the concatenation of the
     positive part of the input with the negative part
     of the input. The result is a tensor of samples that are
     twice as large as the input samples.
     这是示例性的L2归一化与输入的正样本与输入的负样本的级联的组合。结果张量样本是输入样本的两倍大。

     It can be used in place of a ReLU.
     可以使用RelU（Rectified Linear Unit,线性整流函数, 激活函数）替换

     # Input shape
     输入形状
         2D tensor of shape (samples, n)
         形状的2维张量

     # Output shape
     输出形状
         2D tensor of shape (samples, 2*n)
         形状的2维张量

     # Theoretical justification
     理论证明
         When applying ReLU, assuming that the distribution
         of the previous output is approximately centered around 0.,
         使用ReLU时，假设前一个输出分布的以0中心分布

         you are discarding half of your input. This is inefficient.
         放弃了一半的输入。这是低效的。

         Antirectifier allows to return all-positive outputs like ReLU,
         without discarding any data.
         反校正返回了所有正样本输出，像ReLU一样，没有丢弃数据。

         Tests on MNIST show that Antirectifier allows to train networks
         with twice less parameters yet with comparable
         classification accuracy as an equivalent ReLU-based network.
         基于MINIST（数据集）训练，展示反校正训练网络和同类ReLU-based网络相比，使用少于2倍的参数参数，但是实现了类似的分类准确度。
     '''

     def compute_output_shape(self, input_shape):
         shape = list(input_shape)
         assert len(shape) == 2  # only valid for 2D tensors
         shape[-1] *= 2
         return tuple(shape)

     def call(self, inputs):
         inputs -= K.mean(inputs, axis=1, keepdims=True)
         inputs = K.l2_normalize(inputs, axis=1)
         pos = K.relu(inputs)
         neg = K.relu(-inputs)
         return K.concatenate([pos, neg], axis=1)

 # global parameters
 # 全局变量
 batch_size = 128
 num_classes = 10
 epochs = 40

 # the data, shuffled and split between train and test sets
 # 筛选（数据顺序打乱）、划分训练集和测试集
 (x_train, y_train), (x_test, y_test) = mnist.load_data()
 x_train = x_train.reshape(60000, 784)
 x_test = x_test.reshape(10000, 784)
 x_train = x_train.astype('float32')
 x_test = x_test.astype('float32')
 x_train /= 255
 x_test /= 255
 print(x_train.shape[0], 'train samples')
 print(x_test.shape[0], 'test samples')

 # convert class vectors to binary class matrices
 # 类别向量转为多分类矩阵
 y_train = keras.utils.to_categorical(y_train, num_classes)
 y_test = keras.utils.to_categorical(y_test, num_classes)

 # build the model
 # 建立模型
 model = Sequential()
 model.add(layers.Dense(256, input_shape=(784,)))
 model.add(Antirectifier())
 model.add(layers.Dropout(0.1))
 model.add(layers.Dense(256))
 model.add(Antirectifier())
 model.add(layers.Dropout(0.1))
 model.add(layers.Dense(num_classes))
 model.add(layers.Activation('softmax'))

 # compile the model
 # 编译模型
 model.compile(loss='categorical_crossentropy',
               optimizer='rmsprop',
               metrics=['accuracy'])

 # train the model
 # 训练模型
 model.fit(x_train, y_train,
           batch_size=batch_size,
           epochs=epochs,
           verbose=1,
           validation_data=(x_test, y_test))

 # next, compare with an equivalent network
 # with2x bigger Dense layers and ReLU
 # 下一步，使用同结构网络比较，该网络有2倍打的全连接层和ReLU激活函数

执行结果：

60000 train samples
10000 test samples
Train on 60000 samples, validate on 10000 samples
Epoch 1/4
60000/60000 [==============================] - 4s 62us/step - loss: 0.6030 - acc: 0.9131 - val_loss: 0.1637 - val_acc: 0.9565
Epoch 2/4
60000/60000 [==============================] - 3s 58us/step - loss: 0.1264 - acc: 0.9652 - val_loss: 0.0910 - val_acc: 0.9730
Epoch 3/4
60000/60000 [==============================] - 3s 57us/step - loss: 0.0822 - acc: 0.9762 - val_loss: 0.0836 - val_acc: 0.9757
Epoch 4/4
60000/60000 [==============================] - 3s 57us/step - loss: 0.0638 - acc: 0.9810 - val_loss: 0.0762 - val_acc: 0.9780
<keras.callbacks.History at 0x7f355fba6c88>

评估模型：

score = model.evaluate(x_test, y_test,
                       verbose=1)
print('Test score:', score[0])
print('Test accuracy:', score[1])

评估结果：

10000/10000 [==============================] - 1s 59us/step
Test score: 0.07624727148264647
Test accuracy: 0.978

参考链接：

1、https://github.com/keras-team/keras/blob/master/examples/antirectifier.py

2、https://blog.csdn.net/wyx100/article/details/80678735