TensorFlow NormLization

local_response_normalization

local_response_normalization出现在论文”ImageNet Classification with deep Convolutional Neural Networks”中,论文中说,这种normalization对于泛化是有好处的.

经过了一个conv2d或pooling后,我们获得了[batch_size, height, width, channels]这样一个tensor.现在,将channels称之为层,不考虑batch_size

 tf.nn.local_response_normalization(input, depth_radius=None, bias=None, alpha=None, beta=None, name=None)
 '''
 Local Response Normalization.
 The 4-D input tensor is treated as a 3-D array of 1-D vectors (along the last dimension), and each vector is normalized independently. Within a given vector, each component is divided by the weighted, squared sum of inputs within depth_radius. In detail,
 '''
 """
 input: A Tensor. Must be one of the following types: float32, half. 4-D.
 depth_radius: An optional int. Defaults to 5. 0-D. Half-width of the 1-D normalization window.
 bias: An optional float. Defaults to 1. An offset (usually positive to avoid dividing by 0).
 alpha: An optional float. Defaults to 1. A scale factor, usually positive.
 beta: An optional float. Defaults to 0.5. An exponent.
 name: A name for the operation (optional).
 """

举例子：

 import tensorflow as tf  

 a = tf.constant([
     [[1.0, 2.0, 3.0, 4.0],
      [5.0, 6.0, 7.0, 8.0],
      [8.0, 7.0, 6.0, 5.0],
      [4.0, 3.0, 2.0, 1.0]],
     [[4.0, 3.0, 2.0, 1.0],
      [8.0, 7.0, 6.0, 5.0],
      [1.0, 2.0, 3.0, 4.0],
      [5.0, 6.0, 7.0, 8.0]]
 ])
 #reshape a,get the feature map [batch:1 height:2 width:2 channels:8]
 a = tf.reshape(a, [1, 2, 2, 8])  

 normal_a=tf.nn.local_response_normalization(a,2,0,1,1)
 with tf.Session() as sess:
     print("feature map:")
     image = sess.run(a)
     print (image)
     print("normalized feature map:")
     normal = sess.run(normal_a)
     print (normal)  

feature map:
[[[[ 1.  2.  3.  4.  5.  6.  7.  8.]
   [ 8.  7.  6.  5.  4.  3.  2.  1.]]  

  [[ 4.  3.  2.  1.  8.  7.  6.  5.]
   [ 1.  2.  3.  4.  5.  6.  7.  8.]]]]
normalized feature map:
[[[[ 0.07142857  0.06666667  0.05454545  0.04444445  0.03703704  0.03157895
     0.04022989  0.05369128]
   [ 0.05369128  0.04022989  0.03157895  0.03703704  0.04444445  0.05454545
     0.06666667  0.07142857]]  

  [[ 0.13793103  0.10000001  0.0212766   0.00787402  0.05194805  0.04
     0.03448276  0.04545454]
   [ 0.07142857  0.06666667  0.05454545  0.04444445  0.03703704  0.03157895
     0.04022989  0.05369128]]]]  

　　这里我取了n/2=2，k=0，α=1，β=1，举个例子，比如对于一通道的第一个像素“1”来说，我们把参数代人公式就是1/(1^2+2^2+3^2)=0.07142857，对于四通道的第一个像素“4”来说，公式就是4/（2^2+3^2+4^2+5^2+6^2）=0.04444445，以此类推

　　注意：这里的feature_map为【1,2,2,8】，其中1代表图像的数量，2X2代表图像的长宽，8代表图像的层数（map），NRL主要是利用map去计算，然后计算的值为图像的长宽（像素），与图像的数量无关！

　　　　    我们可以这么理解，feature_map分割为直观的图像，第一个通道[1,8,4,1]，第二个通道[2,7,3,2]，第三个通道[3,6,2,3]，以此类推。。。

　　　　　 那么求解的过程和上面就一一对应了，其中在边角达不到n的时候，那就省略。

　　能感觉到这种方法不好吗？效果肯定有的，因为对像素归一化了，有利于计算。但是对于一整幅图像来说反而没有什么太大的作用，因为归一化的种类不同，造成部分特征体现不出来，有时候反而不好。

batch_normalization

　　为什么有batch？上面结尾已经做了初步分析，也已经大概引出来对批量的图像做归一化，对单个图像做的归一化效果不好！

可以看出,batch_normalization之后,数据的维数没有任何变化,只是数值发生了变化 
OutOut作为下一层的输入 
函数: 
tf.nn.batch_normalization()

def batch_normalization(x,
                        mean,
                        variance,
                        offset,
                        scale,
                        variance_epsilon,
                        name=None):

Args:

• x: Input Tensor of arbitrary dimensionality.
• mean: A mean Tensor.
• variance: A variance Tensor.
• offset: An offset Tensor, often denoted ββ in equations, or None. If present, will be added to the normalized tensor.
• scale: A scale Tensor, often denoted γγ in equations, or None. If present, the scale is applied to the normalized tensor.
• variance_epsilon: A small float number to avoid dividing by 0.
• name: A name for this operation (optional).
• Returns: the normalized, scaled, offset tensor. 
  对于卷积,x:[bathc,height,width,depth] 
  对于卷积,我们要feature map中共享 γiγi 和 βiβi ,所以 γ,βγ,β的维度是[depth]

现在,我们需要一个函数 返回mean和variance, 看下面.

tf.nn.moments()

def moments(x, axes, shift=None, name=None, keep_dims=False):
# for simple batch normalization pass `axes=[0]` (batch only)：

对于卷积的batch_normalization, x 为[batch_size, height, width, depth],axes=[0,1,2],就会输出(mean,variance), mean 与 variance 均为标量。

参考：

　　https://blog.csdn.net/mao_xiao_feng/article/details/53488271

　　https://www.jianshu.com/p/c06aea337d5d

　　https://blog.csdn.net/u012436149/article/details/52985303