Tensorflow - Implement for a Convolutional Neural Network on MNIST.

Coding according to TensorFlow 官方文档中文版

中文注释源于：tf.truncated_normal与tf.random_normal

　　　　　　　TF-卷积函数 tf.nn.conv2d 介绍

　　　　　　　TensorFlow - tf.nn.conv2d

　　　　　　　tf.nn.max_pool参数含义和用法

 import tensorflow as tf
 from tensorflow.examples.tutorials.mnist import input_data
 mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

 ''' Intro. for this python file.
 Objective:
     Implement for a Convolutional Neural Network on MNIST.
 Operating Environment:
     python = 3.6.4
     tensorflow = 1.5.0
 '''

 # To avoid initializing weight/bias variables repeatedly, we define two functions for initialization.
 ''' tf.truncated_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)
 Explanation:
     Outputs random values from a truncated normal distribution. The generated values follow a normal distribution with
     specified mean and standard deviation, except that values whose magnitude is more than 2 standard deviations from
     the mean are dropped and re-picked.
     从截断的正态分布输出随机值。
 Args:
     shape: A 1-D integer Tensor or Python array. The shape of the output tensor.
     mean: A 0-D Tensor or Python value of type dtype. The mean of the truncated normal distribution.
     stddev: A 0-D Tensor or Python value of type dtype. The standard deviation of the normal distribution, before truncation.
     dtype: The type of the output.
     seed: A Python integer. Used to create a random seed for the distribution. See tf.set_random_seed for behavior.
     name: A name for the operation (optional).
 Returns:
     A tensor of the specified shape filled with random truncated normal values.
 '''

 def weight_variable(shape):
     initial = tf.truncated_normal(shape, stddev=0.1)
     return tf.Variable(initial)

 def bias_variable(shape):
     initial = tf.constant(value=0.1, shape=shape)
     return tf.Variable(initial)

 # Convolution
 ''' tf.nn.conv2d(input, filter, strides, padding, use_cudnn_on_gpu=True, data_format="NHWC", dilations=[1, 1, 1, 1], name=None)
 Explanation:
     Computes a 2-D convolution given 4-D input and filter tensors. Given an input tensor of shape [batch, in_height,
     in_width, in_channels] and a filter / kernel tensor of shape [filter_height, filter_width, in_channels, out_channels],
     this op performs the following:
         1. Flattens the filter to a 2-D matrix with shape [filter_height * filter_width * in_channels, output_channels].
         2. Extracts image patches from the input tensor to form a virtual tensor of shape [batch, out_height, out_width,
            filter_height * filter_width * in_channels].
         3. For each patch, right-multiplies the filter matrix and the image patch vector.
     In detail, with the default NHWC format,
         output[b, i, j, k] = sum_{di, dj, q} input[b, strides[1] * i + di, strides[2] * j + dj, q] * filter[di, dj, q, k]
     Must have strides[0] = strides[3] = 1. For the most common case of the same horizontal and vertices strides,
     strides = [1, stride, stride, 1].
 Args:
     input: A Tensor. Must be one of the following types: half, bfloat16, float32, float64. A 4-D tensor. The dimension
            order is interpreted according to the value of data_format, see below for details.
     filter: A Tensor. Must have the same type as input. A 4-D tensor of shape [filter_height, filter_width, in_channels,
             out_channels]
     strides: A list of ints. 1-D tensor of length 4. The stride of the sliding window for each dimension of input. The
              dimension order is determined by the value of data_format, see below for details.
     padding: A string from: "SAME", "VALID". The type of padding algorithm to use.
     use_cudnn_on_gpu: An optional bool. Defaults to True.
     data_format: An optional string from: "NHWC", "NCHW". Defaults to "NHWC". Specify the data format of the input and
                  output data. With the default format "NHWC", the data is stored in the order of: [batch, height, width,
                  channels]. Alternatively, the format could be "NCHW", the data storage order of: [batch, channels,
                  height, width].
     dilations: An optional list of ints. Defaults to [1, 1, 1, 1]. 1-D tensor of length 4. The dilation factor for each
                dimension of input. If set to k > 1, there will be k-1 skipped cells between each filter element on that
                dimension. The dimension order is determined by the value of data_format, see above for details. Dilations
                in the batch and depth dimensions must be 1.
     name: A name for the operation (optional).
     第一个参数input：指需要做卷积的输入图像，要求是一个Tensor，具有[batch, in_height, in_width, in_channels]这样的shape，
                      具体含义是[训练时一个batch的图片数量，图片高度，图片宽度，图片通道数]，注意这是一个4维的Tensor，要
                      求类型为float32和float64其中之一。
     第二个参数filter：相当于CNN中的卷积核，它要求是一个Tensor，具有[filter_height, filter_width, in_channels, out_channels]
                       这样的shape，具体含义是[卷积核的高度，卷积核的宽度，图像通道数，卷积核个数]，要求类型与参数input相
                       同，此处，第三维in_channels，就是参数input的第四维。
     第三个参数strides：卷积操作在图像每一维上的步长（strides[0]控制batch，strides[1]控制height，strides[2]控制width，
                        strides[3]控制channels，第一个和最后一个跨度参数通常很少修改，因为它们会在该运算中跳过一些数据，
                        从而不将这部分数据考虑在内，如果希望降低输入的维数，可修改height和width参数），这是一个一维向量，
                        长度为4。
     第四个参数padding：string类型的参数，只能是“SAME”和“VALID”中的一个，这个值决定了不同的卷积方式。
     第五个参数use_cudnn_on_gpu：bool类型，是否使用cudnn加速，默认为true。
 Returns:
     A Tensor. Has the same type as input.
     返回一个Tensor，这个输出就是我们常说的feature map，shape依然是[batch, height, width, channels]这种形式。
 '''

 def conv2d(x, W):
     return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding="SAME")

 ''' tf.nn.max_pool(value, ksize, strides, padding, data_format="NHWC", name=None)
 Explanation:
     Performs the max pooling on the input.
 Args:
     value: A 4-D Tensor of the format specified by data_format.
     ksize: A list or tuple of 4 ints. The size of the window for each dimension of the input tensor.
     strides: A list or tuple of 4 ints. The stride of the sliding window for each dimension of the input tensor.
     padding: A string, either 'VALID' or 'SAME'. The padding algorithm. See the comment here
     data_format: A string. 'NHWC', 'NCHW' and 'NCHW_VECT_C' are supported.
     name: Optional name for the operation.
     第一个参数value：池化操作的输入，一般池化层接在卷积层后面，所以输入通常是feature map，依然是[batch, height, width, channels]
                      这样的shape。
     第二个参数ksize：池化窗口的大小，取一个四维向量，一般是[1, height, width, 1]，因为我们不想在batch和channels上做池化，
                      所以将这两个维度设为1。
     第三个参数strides：和卷积类似，窗口在每一个维度上滑动的步长，一般也是[1, stride, stride, 1]。
     第四个参数padding：和卷积类似，可以取“VALID”或者“SAME”。
 Returns:
     A Tensor of format specified by data_format. The max pooled output tensor.
     返回一个Tensor，类型不变，shape仍然是[batch, height, width, channels]这种形式。
 '''

 # Pooling
 def max_pool_2x2(x):
     return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding="SAME")

 # Set a placeholder. We hope arbitrary number of images could be input to this model.
 x = tf.placeholder("float", [None, 784])

 # Set a placeholder 'y_' to accept the ground-truth values.
 y_ = tf.placeholder("float", [None, 10])

 # 1st Convolutional Layer
 W_conv1 = weight_variable([5, 5, 1, 32])
 b_conv1 = bias_variable([32])

 # In order to utilize this layer, we convert x to a 4-D vector.
 x_image = tf.reshape(x, [-1, 28, 28, 1])

 # 1st ReLU & Max-Pooling
 h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
 h_pool1 = max_pool_2x2(h_conv1)

 # 2nd Convolutional Layer
 W_conv2 = weight_variable([5, 5, 32, 64])
 b_conv2 = bias_variable([64])

 h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
 h_pool2 = max_pool_2x2(h_conv2)

 # Fully Connected Layer
 W_fc1 = weight_variable([7 * 7 * 64, 1024])
 b_fc1 = bias_variable([1024])

 h_pool2_flat = tf.reshape(h_pool2, [-1, 7 * 7 * 64])
 h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

 # Dropout Layer
 keep_prob = tf.placeholder("float")
 h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

 # Output Layer
 W_fc2 = weight_variable([1024, 10])
 b_fc2 = bias_variable([10])

 y_conv = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)

 # Training and Evaluation
 cross_entropy = -tf.reduce_sum(y_ * tf.log(y_conv))
 train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
 correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
 accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))

 # Launch the graph in a session.
 sess = tf.Session()
 sess.run(tf.global_variables_initializer())
 for i in range(20000):
     batch = mnist.train.next_batch(50)
     if i % 100 == 0:
         # train_accuracy = accuracy.eval(feed_dict={x: batch[0], y_: batch[1], keep_prob: 1.0})     # ValueError
         train_accuracy = accuracy.eval(session=sess, feed_dict={x: batch[0], y_: batch[1], keep_prob: 1.0})
         print("step %d, training accuracy %g" % (i, train_accuracy))
     # train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})     # ValueError
     # sess.run(train_step, feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})   # Correct
     train_step.run(session=sess, feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
 print("test accuracy %g" % accuracy.eval(sesson=sess, feed_dict={x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))

 # The training accuracy is stable at 1.