tf

第2章 Tensorflow keras实战

2-0 写在课程之前

• 课程代码的Tensorflow版本

• 大部分代码是tensorflow2.0的

• 课程以tf.kerasAPI为主，因而部分代码可以在tf1.3+运行

• 另有少量tensorflow1.*版本代码

  • 方便大家读懂老代码

2-1 tf-keras简介

理论部分

• Tensorflow-keras简介

• 分类问题、回归问题、损失函数

• 神经网络、激活函数、批归一化、Dropout

• Wide&deep模型

• 超参数搜索

实战部分

• Keras搭建分类模型

• Keras回调函数

• Keras搭建回归模型

• Keras搭建深度神经网络

• Keras实现wide&deep模型

• Keras与scikit-learn实现超参数搜索

keras 是什么？

◆基于python的高级神经网络API

◆Francois Chollet于2014-2015年编写Keras

◆以Tensorflow、CNTK或者Theano为后端运行，keras必须有后端才可以运行

◆后端可以切换，现在多用tensorflow

◆极方便于快速实验，帮助用户以最少的时间验证自己的想法

Tensorflow-keras是什么

◆Tensorflow对keras API规范的实现

◆相对于以tensorflow为后端的keras，Tensorflow-keras与Tensorflow结合更加紧密

◆实现在tf.keras空间下

Tf-keras和keras联系

• 基于同一套API

  • keras程序可以通过改导入方式轻松转为tfkeras程序

  • 反之可能不成立，因为tf.keras有其他特性

• 相同的JSON和HDF5模型序列化格式和语义

Tf-keras和keras区别

• Tf.keras全面支持eager mode

  • 只是用keras.Sequential和keras.Model时没影响

  • 自定义Model内部运算逻辑的时候会有影响

    • Tf低层API可以使用keras的model.fit等抽象

    • 适用于研究人员

• Tf.keras支持基于tf.data的模型训练

• Tf.keras支持TPU训练

• Tf.keras支持tf.distribution中的分布式策略

• 其他特性

  • Tf.keras可以与Tensorflow中的estimator集成
  • Tf.keras可以保存为SavedModel

如何选择？

• 如果想用tf.keras的任何一个特性，那么选tfkeras
• 如果后端互换性很重要，那么选keras
• 如果都不重要，那随便

2-2 分类问题和回归问题

分类问题和回归问题

• 分类问题预测的是类别，模型的输出是概率分布

  • 三分类问题输出例子：[0.2，0.7，0.1], 索引表示分类类别

• 回归问题预测的是值，模型的输出是一个实数值

目标函数

为什么需要目标函数？

• 参数是逐步调整的
• 目标函数可以帮助衡量模型的好坏
  • Model A:[0.1，0.4，0.5]
  • Model B:[0.1，0.2，0.7]

分类问题

• 需要衡量目标类别与当前预测的差距
  • 三分类问题输出例子：[0.2，0.7，0.1]
  • 三分类真实类别：2->one_hot->[0，0，1]
• One-hot编码，把正整数变为向量表达
  • 生成一个长度不小于正整数的向量，只有正整数的位置处为1，其余位置都为0

回归问题

• 预测值与真实值的差距
• 平方差损失
• 绝对值损失

目标函数作用

• 模型的训练就是调整参数，使得目标函数逐渐变小的过程

实战

Keras搭建分类模型

Keras回调函数

Keras搭建回归模型

# tf_keras_classification_model
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras

print(tf.__version__)
print(sys.version_info)
for module in mpl, np, pd, sklearn, tf, keras:
    print(module.__name__, module.__version__)

2.4.0
sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0)
matplotlib 3.2.2
numpy 1.19.4
pandas 1.1.5
sklearn 0.22.2.post1
tensorflow 2.4.0
tensorflow.keras 2.4.0

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all, y_train_all), (x_test, y_test) = fashion_mnist.load_data()
# 训练集拆分为训练集和验证集
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

# 查看各数据集shape
print(x_valid.shape, y_valid.shape)
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)

(5000, 28, 28) (5000,)
(55000, 28, 28) (55000,)
(10000, 28, 28) (10000,)

2-3 实战分类模型之数据读取与展示

# 得到数据集之后，一般会看一下数据集的图像，了解数据集

# 展示图像

def show_single_image(img_arr):
    plt.imshow(img_arr, cmap="binary")
    plt.show()
show_single_image(x_train[0])

y_train_all

array([9, 0, 0, ..., 3, 0, 5], dtype=uint8)

# 只显示一行不直观，现在我们现实多行多列

def show_imgs(n_rows, n_cols, x_data, y_data, class_names):
    assert len(x_data) == len(y_data)
    assert n_rows * n_cols < len(x_data)
    plt.figure(figsize = (n_cols*1.4, n_rows * 1.6))

    for row in range(n_rows):
        for col in range(n_cols):
            index = n_cols * row + col
            plt.subplot(n_rows, n_cols, index+1)
            plt.imshow(x_data[index],
                     cmap="binary",
                     interpolation="nearest")
            plt.axis("off")
            plt.title(class_names[y_data[index]])
    plt.show()

class_names = ["T-shirt", "Trouser", "Pullover", "Dress", "Coat",
               "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot"]
show_imgs(3, 5, x_train, y_train, class_names)

2-4 实战分类模型之模型构建

## 下面开始实现tf.keras模型

# tf.keras.models.Sequential()
# 查看其API https://tensorflow.google.cn/api_docs/python/tf/keras/Sequential#attributes

# model = keras.models.Sequential()
# model.add(keras.layers.Flatten(input_shape=[28, 28]))
# model.add(keras.layers.Dense(300, activation="relu"))
# model.add(keras.layers.Dense(100, activation="relu"))
# model.add(keras.layers.Dense(10, activation="softmax"))

# 上述写法亦可以写作列表的形式

model = keras.models.Sequential([
        keras.layers.Flatten(input_shape=[28, 28]),
        keras.layers.Dense(300, activation="relu"),
        keras.layers.Dense(100, activation="relu"),
        keras.layers.Dense(10, activation="softmax")
])
model.compile(loss="sparse_categorical_crossentropy",
        optimizer="Adam",
        metrics=["accuracy"])

## 查看模型有多少层
model.layers

[<tensorflow.python.keras.layers.core.Flatten at 0x7f8f64ead278>,
 <tensorflow.python.keras.layers.core.Dense at 0x7f8fc0081c50>,
 <tensorflow.python.keras.layers.core.Dense at 0x7f8f64f2ceb8>,
 <tensorflow.python.keras.layers.core.Dense at 0x7f8fc00a5160>]

## 查看模型概况
model.summary()

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
flatten_4 (Flatten)          (None, 784)               0
_________________________________________________________________
dense_10 (Dense)             (None, 300)               235500
_________________________________________________________________
dense_11 (Dense)             (None, 100)               30100
_________________________________________________________________
dense_12 (Dense)             (None, 10)                1010
=================================================================
Total params: 266,610
Trainable params: 266,610
Non-trainable params: 0
_________________________________________________________________

model.summary()

Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
flatten_5 (Flatten)          (None, 784)               0
_________________________________________________________________
dense_13 (Dense)             (None, 300)               235500
_________________________________________________________________
dense_14 (Dense)             (None, 100)               30100
_________________________________________________________________
dense_15 (Dense)             (None, 10)                1010
=================================================================
Total params: 266,610
Trainable params: 266,610
Non-trainable params: 0
_________________________________________________________________

## 全连接层参数个数的计算
# 第一层和第二层为例 None 表示样本数，

# [None, 784] ->  [None, 300],   

# y = X * w + b, 

# 则w.shape = [784, 300], b = [300]

history = model.fit(x_train, y_train, epochs=10,
        validation_data=(x_valid, y_valid)) # 每隔一段时间会用验证集验证

Epoch 1/10
1719/1719 [==============================] - 4s 2ms/step - loss: 6.7652 - accuracy: 0.6911 - val_loss: 0.6093 - val_accuracy: 0.7438
Epoch 2/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.5845 - accuracy: 0.7755 - val_loss: 0.5074 - val_accuracy: 0.8312
Epoch 3/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.4766 - accuracy: 0.8285 - val_loss: 0.4982 - val_accuracy: 0.8358
Epoch 4/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.4469 - accuracy: 0.8406 - val_loss: 0.4517 - val_accuracy: 0.8452
Epoch 5/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.4139 - accuracy: 0.8531 - val_loss: 0.4542 - val_accuracy: 0.8540
Epoch 6/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.4083 - accuracy: 0.8568 - val_loss: 0.3902 - val_accuracy: 0.8588
Epoch 7/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3817 - accuracy: 0.8613 - val_loss: 0.4160 - val_accuracy: 0.8604
Epoch 8/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3720 - accuracy: 0.8684 - val_loss: 0.4020 - val_accuracy: 0.8690
Epoch 9/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3551 - accuracy: 0.8727 - val_loss: 0.4674 - val_accuracy: 0.8576
Epoch 10/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3631 - accuracy: 0.8730 - val_loss: 0.3974 - val_accuracy: 0.8686

type(history)

tensorflow.python.keras.callbacks.History

history.history # 存储训练过程中的一些指标值

{'accuracy': [0.7261272668838501,
  0.7946909070014954,
  0.8306000232696533,
  0.8408545255661011,
  0.8514363765716553,
  0.8586909174919128,
  0.8612363338470459,
  0.8690000176429749,
  0.8706181645393372,
  0.8758000135421753],
 'loss': [2.1727123260498047,
  0.5617926120758057,
  0.47968995571136475,
  0.4502112567424774,
  0.42252394556999207,
  0.40437114238739014,
  0.3874453604221344,
  0.3695518672466278,
  0.3600151538848877,
  0.35218536853790283],
 'val_accuracy': [0.7437999844551086,
  0.8312000036239624,
  0.8357999920845032,
  0.8452000021934509,
  0.8539999723434448,
  0.8587999939918518,
  0.8604000210762024,
  0.8690000176429749,
  0.8575999736785889,
  0.8686000108718872],
 'val_loss': [0.6092722415924072,
  0.5074121952056885,
  0.49819114804267883,
  0.4516642689704895,
  0.45421797037124634,
  0.3901795446872711,
  0.4160061180591583,
  0.40199676156044006,
  0.467364639043808,
  0.3974425494670868]}

# 绘制变化过程
def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0,1)
    plt.show()

plot_learning_curves(history)

一个完成的分类模型包含以下步骤：

• 数据处理
• 模型构建
• 模型训练
• 指标可视化

2-5 实战分类模型之数据归一化

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras 

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all, y_train_all), (x_test, y_test) = fashion_mnist.load_data()
# 训练集拆分为训练集和验证集
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

# 查看各数据集shape
print(x_valid.shape, y_valid.shape)
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)

## 数据归一化
# x = (x-u)/ std
from sklearn.preprocessing import StandardScaler 

scaler = StandardScaler()
# x_train:[None, 28, 28] -> [None, 784]
x_train_scaled = scaler.fit_transform(
    x_train.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

x_valid_scaled = scaler.transform(
    x_valid.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

x_test_scaled = scaler.transform(
    x_test.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

# 输出归一化前后训练集的最大和最小值
print(np.max(x_train), np.min(x_train))
print(np.max(x_train_scaled), np.min(x_train_scaled))

(5000, 28, 28) (5000,)
(55000, 28, 28) (55000,)
(10000, 28, 28) (10000,)
255 0
2.0231433 -0.8105136

model = keras.models.Sequential([
        keras.layers.Flatten(input_shape=[28, 28]),
        keras.layers.Dense(300, activation="relu"),
        keras.layers.Dense(100, activation="relu"),
        keras.layers.Dense(10, activation="softmax")
])
model.compile(loss="sparse_categorical_crossentropy",
        optimizer="Adam",
        metrics=["accuracy"])

history = model.fit(x_train_scaled, y_train, epochs=10,
        validation_data=(x_valid_scaled, y_valid)) # 每隔一段时间会用验证集验证

Epoch 1/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.5684 - accuracy: 0.7943 - val_loss: 0.3538 - val_accuracy: 0.8718
Epoch 2/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3485 - accuracy: 0.8704 - val_loss: 0.3592 - val_accuracy: 0.8734
Epoch 3/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3227 - accuracy: 0.8799 - val_loss: 0.3563 - val_accuracy: 0.8738
Epoch 4/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3003 - accuracy: 0.8879 - val_loss: 0.3185 - val_accuracy: 0.8866
Epoch 5/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2742 - accuracy: 0.8963 - val_loss: 0.3146 - val_accuracy: 0.8892
Epoch 6/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2567 - accuracy: 0.9035 - val_loss: 0.3038 - val_accuracy: 0.8942
Epoch 7/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2407 - accuracy: 0.9102 - val_loss: 0.3153 - val_accuracy: 0.8902
Epoch 8/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2202 - accuracy: 0.9178 - val_loss: 0.3043 - val_accuracy: 0.8932
Epoch 9/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2105 - accuracy: 0.9208 - val_loss: 0.3243 - val_accuracy: 0.8908
Epoch 10/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2047 - accuracy: 0.9212 - val_loss: 0.3316 - val_accuracy: 0.8938

# 绘制变化过程
def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0,1)
    plt.show()

plot_learning_curves(history)

model.evaluate(x_test_scaled, y_test)

313/313 [==============================] - 1s 2ms/step - loss: 0.3609 - accuracy: 0.8873

[0.3608780801296234, 0.8873000144958496]

2-6 实战回调函数

https://tensorflow.google.cn/api_docs/python/tf/keras/callbacks

作用：当训练模型时，有些时间可能需要做一些事情，比如

• loss不再下降时，提前停掉迭代，即 tf.keras.callbacks.EarlyStopping

• 保存所有参数的中间状态，在神经网络中每隔段时间保存该checkpoint， 即 tf.keras.callbacks.ModelCheckpoint

• Tensorboard 可视化， 即 tf.keras.callbacks.TensorBoard

代码实战

### 代码实战
# ------------------------------------------原------------------------------------
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras 

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all, y_train_all), (x_test, y_test) = fashion_mnist.load_data()
# 训练集拆分为训练集和验证集
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

# 查看各数据集shape
print(x_valid.shape, y_valid.shape)
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)

## 数据归一化
# x = (x-u)/ std
from sklearn.preprocessing import StandardScaler 

scaler = StandardScaler()
# x_train:[None, 28, 28] -> [None, 784]
x_train_scaled = scaler.fit_transform(
    x_train.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

x_valid_scaled = scaler.transform(
    x_valid.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

x_test_scaled = scaler.transform(
    x_test.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

# 输出归一化前后训练集的最大和最小值
print(np.max(x_train), np.min(x_train))
print(np.max(x_train_scaled), np.min(x_train_scaled))

model = keras.models.Sequential([
        keras.layers.Flatten(input_shape=[28, 28]),
        keras.layers.Dense(300, activation="relu"),
        keras.layers.Dense(100, activation="relu"),
        keras.layers.Dense(10, activation="softmax")
])
model.compile(loss="sparse_categorical_crossentropy",
        optimizer="Adam",
        metrics=["accuracy"])
# ------------------------------------------原------------------------------------

# ------------------------------------------改------------------------------------
# 由于callbacks是在训练过程中做一些侦听，可以加在fit()中
# 添加方式为，定义callback数组,
# Tensorboard, --> 保存至文件夹
# earlystopping,
# ModelCheckpoint --> 保存至文件

# 文件夹定义
logdir = "./callbacks"
if not os.path.exists(logdir):
    os.mkdir(logdir)
# 文件定义
output_model_file = os.path.join(logdir,
            "fashion_mnist_model.h5")
callbacks = [
    keras.callbacks.TensorBoard(logdir),
    keras.callbacks.ModelCheckpoint(output_model_file,
    save_best_only=True # 添加该参数，保存最好的模型，否则，保存最近的模型

            ),
    # earlystopping(
    #     monitor= , # 关注指标，一般关注验证集目标函数的值
    #     min_delta= , # 阈值，本次训练和上次训练的差值与该值比较，如果低于该阈值，提前停止
    #     patience= , # 设置连续patience次差值低于min_delta，停止
    # )
    keras.callbacks.EarlyStopping(
        patience = 5,
        min_delta = 1e-3
    )
]

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz
32768/29515 [=================================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz
26427392/26421880 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz
8192/5148 [===============================================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz
4423680/4422102 [==============================] - 0s 0us/step
(5000, 28, 28) (5000,)
(55000, 28, 28) (55000,)
(10000, 28, 28) (10000,)
255 0
2.0231433 -0.8105136

# fit 中添加callbacks 参数
history = model.fit(x_train_scaled, y_train, epochs=10,
    validation_data=(x_valid_scaled, y_valid), # 每隔一段时间会用验证集验证
    callbacks=callbacks)
# ------------------------------------------改------------------------------------

Epoch 1/10
1719/1719 [==============================] - 6s 3ms/step - loss: 0.5581 - accuracy: 0.7962 - val_loss: 0.3765 - val_accuracy: 0.8618
Epoch 2/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3533 - accuracy: 0.8697 - val_loss: 0.3387 - val_accuracy: 0.8778
Epoch 3/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3236 - accuracy: 0.8799 - val_loss: 0.3310 - val_accuracy: 0.8824
Epoch 4/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2922 - accuracy: 0.8894 - val_loss: 0.3211 - val_accuracy: 0.8816
Epoch 5/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2732 - accuracy: 0.8966 - val_loss: 0.3204 - val_accuracy: 0.8822
Epoch 6/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2476 - accuracy: 0.9057 - val_loss: 0.3019 - val_accuracy: 0.8908
Epoch 7/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2375 - accuracy: 0.9115 - val_loss: 0.3109 - val_accuracy: 0.8908
Epoch 8/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2219 - accuracy: 0.9146 - val_loss: 0.3283 - val_accuracy: 0.8910
Epoch 9/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2153 - accuracy: 0.9200 - val_loss: 0.3627 - val_accuracy: 0.8794
Epoch 10/10
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2073 - accuracy: 0.9208 - val_loss: 0.3470 - val_accuracy: 0.8926

!ls ./callbacks/

fashion_mnist_model.h5	train  validation

# 打开tensorBoard
!tensorboard --logdir=callbacks

2021-01-08 01:32:15.476399: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
W0108 01:32:16.959013 140229050181376 plugin_event_accumulator.py:322] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.4.0 at http://localhost:6006/ (Press CTRL+C to quit)
http://localhost:6006/
http://localhost:6006/
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/manager.py", line 273, in remove_info_file
    os.unlink(_get_info_file_path())
KeyboardInterrupt

colab 无法直接调用tensorboard 解决方法

• 安装ngrok

!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
!unzip ngrok-stable-linux-amd64.zip

• run TensorBoard on Colab

在colab notebook中依次执行以下命令

LOG_DIR = './log'
get_ipython().system_raw(
    'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'
    .format(LOG_DIR)
)
-------------------------------------------------------------------------------------------------
get_ipython().system_raw('./ngrok http 6006 &')
-------------------------------------------------------------------------------------------------
! curl -s http://localhost:4040/api/tunnels | python3 -c \
    "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"

• 打开链接查看效果

!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
!unzip ngrok-stable-linux-amd64.zip

--2021-01-08 01:38:30--  https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
Resolving bin.equinox.io (bin.equinox.io)... 52.2.56.23, 52.44.17.83, 52.200.34.95, ...
Connecting to bin.equinox.io (bin.equinox.io)|52.2.56.23|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13773305 (13M) [application/octet-stream]
Saving to: ‘ngrok-stable-linux-amd64.zip’

ngrok-stable-linux- 100%[===================>]  13.13M  18.7MB/s    in 0.7s    

2021-01-08 01:38:31 (18.7 MB/s) - ‘ngrok-stable-linux-amd64.zip’ saved [13773305/13773305]

Archive:  ngrok-stable-linux-amd64.zip
  inflating: ngrok

logdir = "./callbacks"
get_ipython().system_raw(
    'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'
    .format(logdir)
)

get_ipython().system_raw('./ngrok http 6006 &')

!curl -s http://localhost:4040/api/tunnels | python3 -c \
    "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
# 失败

https://3d8fde90b3b6.ngrok.io

下列方法可用

%reload_ext tensorboard

%tensorboard --logdir "./callbacks"

2-7 实战回归模型

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras 

# 新数据集---房价预测
from sklearn.datasets import fetch_california_housing 

housing = fetch_california_housing()
print(housing.DESCR)
print(housing.data.shape)
print(housing.target.shape)

.. _california_housing_dataset:

California Housing dataset
--------------------------

**Data Set Characteristics:**

    :Number of Instances: 20640

    :Number of Attributes: 8 numeric, predictive attributes and the target

    :Attribute Information:
        - MedInc        median income in block
        - HouseAge      median house age in block
        - AveRooms      average number of rooms
        - AveBedrms     average number of bedrooms
        - Population    block population
        - AveOccup      average house occupancy
        - Latitude      house block latitude
        - Longitude     house block longitude

    :Missing Attribute Values: None

This dataset was obtained from the StatLib repository.
http://lib.stat.cmu.edu/datasets/

The target variable is the median house value for California districts.

This dataset was derived from the 1990 U.S. census, using one row per census
block group. A block group is the smallest geographical unit for which the U.S.
Census Bureau publishes sample data (a block group typically has a population
of 600 to 3,000 people).

It can be downloaded/loaded using the
:func:`sklearn.datasets.fetch_california_housing` function.

.. topic:: References

    - Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,
      Statistics and Probability Letters, 33 (1997) 291-297

(20640, 8)
(20640,)

import pprint
pprint.pprint(housing.data[:5])
pprint.pprint(housing.target[:5])

array([[ 8.32520000e+00,  4.10000000e+01,  6.98412698e+00,
         1.02380952e+00,  3.22000000e+02,  2.55555556e+00,
         3.78800000e+01, -1.22230000e+02],
       [ 8.30140000e+00,  2.10000000e+01,  6.23813708e+00,
         9.71880492e-01,  2.40100000e+03,  2.10984183e+00,
         3.78600000e+01, -1.22220000e+02],
       [ 7.25740000e+00,  5.20000000e+01,  8.28813559e+00,
         1.07344633e+00,  4.96000000e+02,  2.80225989e+00,
         3.78500000e+01, -1.22240000e+02],
       [ 5.64310000e+00,  5.20000000e+01,  5.81735160e+00,
         1.07305936e+00,  5.58000000e+02,  2.54794521e+00,
         3.78500000e+01, -1.22250000e+02],
       [ 3.84620000e+00,  5.20000000e+01,  6.28185328e+00,
         1.08108108e+00,  5.65000000e+02,  2.18146718e+00,
         3.78500000e+01, -1.22250000e+02]])
array([4.526, 3.585, 3.521, 3.413, 3.422])

from sklearn.model_selection import train_test_split 

x_train_all, x_test, y_train_all, y_test = train_test_split(
    housing.data, housing.target, random_state=7, test_size=0.25
) # 默认3:1的比例划分

x_train, x_valid, y_train, y_valid = train_test_split(x_train_all, y_train_all,
                         random_state=11)
print(x_train.shape, y_train.shape)
print(x_valid.shape, y_valid.shape)
print(x_test.shape, y_test.shape)

(11610, 8) (11610,)
(3870, 8) (3870,)
(5160, 8) (5160,)

# 归一化

from sklearn.preprocessing import StandardScaler 

scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train)
# 获取训练集的均值和方差， 在测试集和验证集上都用和训练集一样的均值和方差
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)

# 搭建模型
model = keras.models.Sequential([
    keras.layers.Dense(30, activation="relu",
        input_shape=x_train.shape[1:]), # 8
    keras.layers.Dense(1),
])

model.summary() # model 信息
model.compile(loss="mean_squared_error", optimizer="sgd") # 编译model， 设置损失函数和优化方法
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-3
)]

Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_4 (Dense)              (None, 30)                270
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 31
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________

# 训练模型
history = model.fit(x_train_scaled, y_train,
        validation_data=(x_valid_scaled, y_valid),
        epochs = 100,
        callbacks=callbacks)

Epoch 1/100
363/363 [==============================] - 1s 2ms/step - loss: 1.6214 - val_loss: 0.5763
Epoch 2/100
363/363 [==============================] - 1s 2ms/step - loss: 0.5264 - val_loss: 0.4949
Epoch 3/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4588 - val_loss: 0.4752
Epoch 4/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4485 - val_loss: 0.4457
Epoch 5/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4158 - val_loss: 0.4256
Epoch 6/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4042 - val_loss: 0.4230
Epoch 7/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4001 - val_loss: 0.4183
Epoch 8/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3986 - val_loss: 0.4169
Epoch 9/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4005 - val_loss: 0.4102
Epoch 10/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3910 - val_loss: 0.3958
Epoch 11/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3766 - val_loss: 0.4064
Epoch 12/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3965 - val_loss: 0.3906
Epoch 13/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3726 - val_loss: 0.3879
Epoch 14/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3789 - val_loss: 0.3896
Epoch 15/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3906 - val_loss: 0.3848
Epoch 16/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3817 - val_loss: 0.3761
Epoch 17/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3731 - val_loss: 0.3800
Epoch 18/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3687 - val_loss: 0.3720
Epoch 19/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3705 - val_loss: 0.3678
Epoch 20/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3681 - val_loss: 0.3827
Epoch 21/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3705 - val_loss: 0.3676
Epoch 22/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3408 - val_loss: 0.3653
Epoch 23/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3543 - val_loss: 0.3684
Epoch 24/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3523 - val_loss: 0.3594
Epoch 25/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3586 - val_loss: 0.3602
Epoch 26/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3501 - val_loss: 0.3642
Epoch 27/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3439 - val_loss: 0.3605
Epoch 28/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3365 - val_loss: 0.3535
Epoch 29/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3406 - val_loss: 0.3553
Epoch 30/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3392 - val_loss: 0.3576
Epoch 31/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3478 - val_loss: 0.3519
Epoch 32/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3458 - val_loss: 0.3580
Epoch 33/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3236 - val_loss: 0.3534
Epoch 34/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3406 - val_loss: 0.3579
Epoch 35/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3487 - val_loss: 0.3635
Epoch 36/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3459 - val_loss: 0.3483
Epoch 37/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3353 - val_loss: 0.3469
Epoch 38/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3360 - val_loss: 0.3457
Epoch 39/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3360 - val_loss: 0.3518
Epoch 40/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3286 - val_loss: 0.3548
Epoch 41/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3249 - val_loss: 0.3429
Epoch 42/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3334 - val_loss: 0.3640
Epoch 43/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3244 - val_loss: 0.3402
Epoch 44/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3353 - val_loss: 0.3430
Epoch 45/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3265 - val_loss: 0.3402
Epoch 46/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3250 - val_loss: 0.3381
Epoch 47/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3250 - val_loss: 0.3368
Epoch 48/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3162 - val_loss: 0.3380
Epoch 49/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3201 - val_loss: 0.3399
Epoch 50/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3348 - val_loss: 0.3405
Epoch 51/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3246 - val_loss: 0.3387
Epoch 52/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3218 - val_loss: 0.3337
Epoch 53/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3266 - val_loss: 0.3368
Epoch 54/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3227 - val_loss: 0.3386
Epoch 55/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3273 - val_loss: 0.3372
Epoch 56/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3305 - val_loss: 0.3348
Epoch 57/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3132 - val_loss: 0.3306
Epoch 58/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3413 - val_loss: 0.3344
Epoch 59/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3282 - val_loss: 0.3307
Epoch 60/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3161 - val_loss: 0.3278
Epoch 61/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3282 - val_loss: 0.3312
Epoch 62/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3169 - val_loss: 0.3277
Epoch 63/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3257 - val_loss: 0.3294
Epoch 64/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3131 - val_loss: 0.3344
Epoch 65/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3052 - val_loss: 0.3275

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0, 1)

    plt.show()
plot_learning_curves(history)

# 测试测试集效果
model.evaluate(x_test_scaled, y_test)

162/162 [==============================] - 0s 1ms/step - loss: 0.3329

0.33294564485549927

2-8 神经网络

实战

• Keras搭建分类模型

• Keras回调函数

• Keras搭建回归模型

神经网络训练

• 下山算法
  • 找到方向
  • 走一步
• 梯度下降
  • 求导
  • 更新参数

激活函数

归一化

• min-max 归一化 x = (x-min)/(max-min)
• Z-score 归一化 x = \((x - \mu\))/\(\sigma\)

批归一化

• 每层的激活值都做归一化

dropout

Dropout 作用

• 防止过拟合
  • 训练集上很好，测试集上不好
  • 参数太多，记住样本，不能泛化

2-9 实战深度神经网络

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras 

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all, y_train_all), (x_test, y_test) = fashion_mnist.load_data()
# 训练集拆分为训练集和验证集
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

# 查看各数据集shape
print(x_valid.shape, y_valid.shape)
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)

## 数据归一化
# x = (x-u)/ std
from sklearn.preprocessing import StandardScaler 

scaler = StandardScaler()
# x_train:[None, 28, 28] -> [None, 784]
x_train_scaled = scaler.fit_transform(
    x_train.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

x_valid_scaled = scaler.transform(
    x_valid.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

x_test_scaled = scaler.transform(
    x_test.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

# 输出归一化前后训练集的最大和最小值
print(np.max(x_train), np.min(x_train))
print(np.max(x_train_scaled), np.min(x_train_scaled))

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz
32768/29515 [=================================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz
26427392/26421880 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz
8192/5148 [===============================================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz
4423680/4422102 [==============================] - 0s 0us/step
(5000, 28, 28) (5000,)
(55000, 28, 28) (55000,)
(10000, 28, 28) (10000,)
255 0
2.0231433 -0.8105136

model = keras.models.Sequential()

model.add(keras.layers.Flatten(input_shape=[28, 28]))
for _ in range(20):
    model.add(keras.layers.Dense(10, activation="relu"))
model.add(keras.layers.Dense(10, activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",
        optimizer="Adam",
        metrics=["accuracy"])
# ------------------------------------------改------------------------------------

# ------------------------------------------改------------------------------------
# 由于callbacks是在训练过程中做一些侦听，可以加在fit()中
# 添加方式为，定义callback数组,
# Tensorboard, --> 保存至文件夹
# earlystopping,
# ModelCheckpoint --> 保存至文件

# 文件夹定义
logdir = "./dnn_callbacks"
if not os.path.exists(logdir):
    os.mkdir(logdir)
# 文件定义
output_model_file = os.path.join(logdir,
            "fashion_mnist_model.h5")
callbacks = [
    keras.callbacks.TensorBoard(logdir),
    keras.callbacks.ModelCheckpoint(output_model_file,
    save_best_only=True # 添加该参数，保存最好的模型，否则，保存最近的模型

            ),
    # earlystopping(
    #     monitor= , # 关注指标，一般关注验证集目标函数的值
    #     min_delta= , # 阈值，本次训练和上次训练的差值与该值比较，如果低于该阈值，提前停止
    #     patience= , # 设置连续patience次差值低于min_delta，停止
    # )
    keras.callbacks.EarlyStopping(
        patience = 5,
        min_delta = 1e-3
    )
]

# 训练模型
history = model.fit(x_train_scaled, y_train,
        validation_data=(x_valid_scaled, y_valid),
        epochs = 100,
        callbacks=callbacks)

Epoch 1/100
1719/1719 [==============================] - 10s 5ms/step - loss: 1.9010 - accuracy: 0.2418 - val_loss: 1.2311 - val_accuracy: 0.5884
Epoch 2/100
1719/1719 [==============================] - 7s 4ms/step - loss: 1.1991 - accuracy: 0.5639 - val_loss: 1.0831 - val_accuracy: 0.6364
Epoch 3/100
1719/1719 [==============================] - 7s 4ms/step - loss: 1.0624 - accuracy: 0.6358 - val_loss: 1.0008 - val_accuracy: 0.6814
Epoch 4/100
1719/1719 [==============================] - 7s 4ms/step - loss: 1.0070 - accuracy: 0.6748 - val_loss: 1.0249 - val_accuracy: 0.6260
Epoch 5/100
1719/1719 [==============================] - 7s 4ms/step - loss: 1.0077 - accuracy: 0.6585 - val_loss: 0.8663 - val_accuracy: 0.7020
Epoch 6/100
1719/1719 [==============================] - 8s 4ms/step - loss: 0.8910 - accuracy: 0.6855 - val_loss: 0.8173 - val_accuracy: 0.6992
Epoch 7/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.7930 - accuracy: 0.7046 - val_loss: 0.7331 - val_accuracy: 0.7186
Epoch 8/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.7249 - accuracy: 0.7163 - val_loss: 0.6725 - val_accuracy: 0.7344
Epoch 9/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.6603 - accuracy: 0.7391 - val_loss: 0.6464 - val_accuracy: 0.7504
Epoch 10/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.6401 - accuracy: 0.7479 - val_loss: 0.6380 - val_accuracy: 0.7712
Epoch 11/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.6071 - accuracy: 0.7760 - val_loss: 0.6173 - val_accuracy: 0.7810
Epoch 12/100
1719/1719 [==============================] - 8s 5ms/step - loss: 0.5792 - accuracy: 0.7928 - val_loss: 0.5478 - val_accuracy: 0.8060
Epoch 13/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.5654 - accuracy: 0.7957 - val_loss: 0.5678 - val_accuracy: 0.7952
Epoch 14/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.5420 - accuracy: 0.8031 - val_loss: 0.5327 - val_accuracy: 0.8104
Epoch 15/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.5179 - accuracy: 0.8092 - val_loss: 0.5385 - val_accuracy: 0.8170
Epoch 16/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.5203 - accuracy: 0.8109 - val_loss: 0.5016 - val_accuracy: 0.8182
Epoch 17/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.4946 - accuracy: 0.8167 - val_loss: 0.5325 - val_accuracy: 0.8152
Epoch 18/100
1719/1719 [==============================] - 8s 4ms/step - loss: 0.5141 - accuracy: 0.8145 - val_loss: 0.5069 - val_accuracy: 0.8256
Epoch 19/100
1719/1719 [==============================] - 8s 4ms/step - loss: 0.4930 - accuracy: 0.8211 - val_loss: 0.4899 - val_accuracy: 0.8290
Epoch 20/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.4973 - accuracy: 0.8205 - val_loss: 0.5007 - val_accuracy: 0.8262
Epoch 21/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.4797 - accuracy: 0.8332 - val_loss: 0.5374 - val_accuracy: 0.8174
Epoch 22/100
1719/1719 [==============================] - 8s 4ms/step - loss: 0.4806 - accuracy: 0.8322 - val_loss: 0.5073 - val_accuracy: 0.8194
Epoch 23/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.4972 - accuracy: 0.8181 - val_loss: 0.4662 - val_accuracy: 0.8338
Epoch 24/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.4577 - accuracy: 0.8365 - val_loss: 0.5004 - val_accuracy: 0.8210
Epoch 25/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.4699 - accuracy: 0.8340 - val_loss: 0.4765 - val_accuracy: 0.8382
Epoch 26/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.4584 - accuracy: 0.8397 - val_loss: 0.5015 - val_accuracy: 0.8316
Epoch 27/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.4853 - accuracy: 0.8330 - val_loss: 0.4862 - val_accuracy: 0.8344
Epoch 28/100
1719/1719 [==============================] - 7s 4ms/step - loss: 0.4573 - accuracy: 0.8443 - val_loss: 0.4857 - val_accuracy: 0.8364

model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
flatten (Flatten)            (None, 784)               0
_________________________________________________________________
dense (Dense)                (None, 10)                7850
_________________________________________________________________
dense_1 (Dense)              (None, 10)                110
_________________________________________________________________
dense_2 (Dense)              (None, 10)                110
_________________________________________________________________
dense_3 (Dense)              (None, 10)                110
_________________________________________________________________
dense_4 (Dense)              (None, 10)                110
_________________________________________________________________
dense_5 (Dense)              (None, 10)                110
_________________________________________________________________
dense_6 (Dense)              (None, 10)                110
_________________________________________________________________
dense_7 (Dense)              (None, 10)                110
_________________________________________________________________
dense_8 (Dense)              (None, 10)                110
_________________________________________________________________
dense_9 (Dense)              (None, 10)                110
_________________________________________________________________
dense_10 (Dense)             (None, 10)                110
_________________________________________________________________
dense_11 (Dense)             (None, 10)                110
_________________________________________________________________
dense_12 (Dense)             (None, 10)                110
_________________________________________________________________
dense_13 (Dense)             (None, 10)                110
_________________________________________________________________
dense_14 (Dense)             (None, 10)                110
_________________________________________________________________
dense_15 (Dense)             (None, 10)                110
_________________________________________________________________
dense_16 (Dense)             (None, 10)                110
_________________________________________________________________
dense_17 (Dense)             (None, 10)                110
_________________________________________________________________
dense_18 (Dense)             (None, 10)                110
_________________________________________________________________
dense_19 (Dense)             (None, 10)                110
_________________________________________________________________
dense_20 (Dense)             (None, 10)                110
=================================================================
Total params: 10,050
Trainable params: 10,050
Non-trainable params: 0
_________________________________________________________________

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0, 2)

    plt.show()
plot_learning_curves(history)

2-10 实战批归一化

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras 

fashion_mnist = keras.datasets.fashion_mnist
(x_train_all, y_train_all), (x_test, y_test) = fashion_mnist.load_data()
# 训练集拆分为训练集和验证集
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

# 查看各数据集shape
print(x_valid.shape, y_valid.shape)
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)

## 数据归一化
# x = (x-u)/ std
from sklearn.preprocessing import StandardScaler 

scaler = StandardScaler()
# x_train:[None, 28, 28] -> [None, 784]
x_train_scaled = scaler.fit_transform(
    x_train.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

x_valid_scaled = scaler.transform(
    x_valid.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

x_test_scaled = scaler.transform(
    x_test.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

# 输出归一化前后训练集的最大和最小值
print(np.max(x_train), np.min(x_train))
print(np.max(x_train_scaled), np.min(x_train_scaled))

model = keras.models.Sequential()

model.add(keras.layers.Flatten(input_shape=[28, 28]))
for _ in range(20):
    model.add(keras.layers.Dense(100, activation="relu"))
    model.add(keras.layers.BatchNormalization())
    """
    model.add(keras.layers.Dense(100))
    model.add(keras.layers.BatchNormalization())
    model.add(keras.layers.Activation("relu")
    """
model.add(keras.layers.Dense(10, activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",
        optimizer="Adam",
        metrics=["accuracy"])
# ------------------------------------------改------------------------------------

# 文件夹定义
logdir = "./dnn_bn_callbacks"
if not os.path.exists(logdir):
    os.mkdir(logdir)
# 文件定义
output_model_file = os.path.join(logdir,
            "fashion_mnist_model.h5")
callbacks = [
    keras.callbacks.TensorBoard(logdir),
    keras.callbacks.ModelCheckpoint(output_model_file,
    save_best_only=True # 添加该参数，保存最好的模型，否则，保存最近的模型

            ),
    # earlystopping(
    #     monitor= , # 关注指标，一般关注验证集目标函数的值
    #     min_delta= , # 阈值，本次训练和上次训练的差值与该值比较，如果低于该阈值，提前停止
    #     patience= , # 设置连续patience次差值低于min_delta，停止
    # )
    keras.callbacks.EarlyStopping(
        patience = 5,
        min_delta = 1e-3
    )
]

# 训练模型
history = model.fit(x_train_scaled, y_train,
        validation_data=(x_valid_scaled, y_valid),
        epochs = 100,
        callbacks=callbacks)

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0, 2)

    plt.show()
plot_learning_curves(history)

(5000, 28, 28) (5000,)
(55000, 28, 28) (55000,)
(10000, 28, 28) (10000,)
255 0
2.0231433 -0.8105136
Epoch 1/100
1719/1719 [==============================] - 35s 18ms/step - loss: 1.5903 - accuracy: 0.4364 - val_loss: 0.6456 - val_accuracy: 0.7518
Epoch 2/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.7511 - accuracy: 0.7337 - val_loss: 0.4965 - val_accuracy: 0.8318
Epoch 3/100
1719/1719 [==============================] - 32s 18ms/step - loss: 0.6367 - accuracy: 0.7799 - val_loss: 0.4971 - val_accuracy: 0.8318
Epoch 4/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.5888 - accuracy: 0.8000 - val_loss: 0.5067 - val_accuracy: 0.8274
Epoch 5/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.5674 - accuracy: 0.8067 - val_loss: 0.5105 - val_accuracy: 0.8176
Epoch 6/100
1719/1719 [==============================] - 32s 18ms/step - loss: 0.5449 - accuracy: 0.8132 - val_loss: 0.4530 - val_accuracy: 0.8470
Epoch 7/100
1719/1719 [==============================] - 32s 19ms/step - loss: 0.5033 - accuracy: 0.8300 - val_loss: 0.4640 - val_accuracy: 0.8310
Epoch 8/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.4792 - accuracy: 0.8375 - val_loss: 0.4178 - val_accuracy: 0.8622
Epoch 9/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.4675 - accuracy: 0.8426 - val_loss: 0.4060 - val_accuracy: 0.8708
Epoch 10/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.4392 - accuracy: 0.8499 - val_loss: 0.3845 - val_accuracy: 0.8676
Epoch 11/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.4186 - accuracy: 0.8570 - val_loss: 0.3860 - val_accuracy: 0.8702
Epoch 12/100
1719/1719 [==============================] - 30s 17ms/step - loss: 0.4118 - accuracy: 0.8574 - val_loss: 0.3833 - val_accuracy: 0.8726
Epoch 13/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3956 - accuracy: 0.8631 - val_loss: 0.3827 - val_accuracy: 0.8630
Epoch 14/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3780 - accuracy: 0.8716 - val_loss: 0.3664 - val_accuracy: 0.8712
Epoch 15/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3709 - accuracy: 0.8733 - val_loss: 0.3624 - val_accuracy: 0.8780
Epoch 16/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3634 - accuracy: 0.8768 - val_loss: 0.3510 - val_accuracy: 0.8802
Epoch 17/100
1719/1719 [==============================] - 32s 18ms/step - loss: 0.3601 - accuracy: 0.8740 - val_loss: 0.3877 - val_accuracy: 0.8718
Epoch 18/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3421 - accuracy: 0.8799 - val_loss: 0.3802 - val_accuracy: 0.8688
Epoch 19/100
1719/1719 [==============================] - 30s 18ms/step - loss: 0.3471 - accuracy: 0.8804 - val_loss: 0.3373 - val_accuracy: 0.8754
Epoch 20/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3311 - accuracy: 0.8871 - val_loss: 0.3615 - val_accuracy: 0.8772
Epoch 21/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3230 - accuracy: 0.8894 - val_loss: 0.3563 - val_accuracy: 0.8846
Epoch 22/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3163 - accuracy: 0.8888 - val_loss: 0.3367 - val_accuracy: 0.8792
Epoch 23/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3094 - accuracy: 0.8909 - val_loss: 0.3592 - val_accuracy: 0.8702
Epoch 24/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.3076 - accuracy: 0.8917 - val_loss: 0.3198 - val_accuracy: 0.8910
Epoch 25/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.2984 - accuracy: 0.8962 - val_loss: 0.3445 - val_accuracy: 0.8850
Epoch 26/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.2942 - accuracy: 0.8959 - val_loss: 0.3296 - val_accuracy: 0.8840
Epoch 27/100
1719/1719 [==============================] - 32s 18ms/step - loss: 0.2919 - accuracy: 0.8999 - val_loss: 0.3410 - val_accuracy: 0.8818
Epoch 28/100
1719/1719 [==============================] - 30s 18ms/step - loss: 0.2892 - accuracy: 0.8994 - val_loss: 0.3381 - val_accuracy: 0.8826
Epoch 29/100
1719/1719 [==============================] - 31s 18ms/step - loss: 0.2875 - accuracy: 0.9008 - val_loss: 0.3229 - val_accuracy: 0.8816

2-11 Wide&Deep 模型

第3章 TF 基础API

• 基础API

• 基础API与keras的集成

  • 自定义损失函数
  • 自定义层次

• @tffunction的使用

  • 图结构

• 自定义求导

3-1 tf 基础API引入

@tf.function

• 将python函数编译成图
• 易于将模型导出成为GraphDef+checkpoint或者SavedModel
• 使得eager_execution可以默认打开
• 1.0的代码可以通过tf.function来继续在2.0里使用
  • 替代session

API列表

• 基础数据类型
  • Tf.constant，tf.string
  • tf.ragged.constant，tf.SparseTensor，Tf.Variable
• 自定义损失函数——Tf.reduce_mean
• 自定义层次——Keras.layers.Lambda和继承法
• Tf.function
  • Tf.function，tf.autograph.to_code，get_concrete_function

API列表

• GraphDef
  • get_operations，get_operation_by_name
  • get_tensor_by_name，as_graph_def
• 自动求导
  • Tf.GradientTape
  • Optimzier.apply_gradients

3-2实战tf.constant

# tf_basic_api
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras

t =tf.constant([[1., 2., 3.],
                [4., 5., 6.]])
print(t)
print(t[:, 1:])
print(t[..., 1])

tf.Tensor(
[[1. 2. 3.]
 [4. 5. 6.]], shape=(2, 3), dtype=float32)
tf.Tensor(
[[2. 3.]
 [5. 6.]], shape=(2, 2), dtype=float32)
tf.Tensor([2. 5.], shape=(2,), dtype=float32)

# op
print(t + 10)
print(tf.square(t))
print(t @ tf.transpose(t))

tf.Tensor(
[[11. 12. 13.]
 [14. 15. 16.]], shape=(2, 3), dtype=float32)
tf.Tensor(
[[ 1.  4.  9.]
 [16. 25. 36.]], shape=(2, 3), dtype=float32)
tf.Tensor(
[[14. 32.]
 [32. 77.]], shape=(2, 2), dtype=float32)

# numpy conversion

print(t.numpy())
print(np.square(t))

np_t = np.array([[1., 2., 3.],
                [4., 5., 6.]])
print(tf.constant(np_t))

[[1. 2. 3.]
 [4. 5. 6.]]
[[ 1.  4.  9.]
 [16. 25. 36.]]
tf.Tensor(
[[1. 2. 3.]
 [4. 5. 6.]], shape=(2, 3), dtype=float64)

# Scalars
t = tf.constant(2.718)
print(t.numpy())
print(t.shape)

2.718
()

3-3 实战tf.strings与ragged tensor

# strings

t = tf.constant("cafe")

print(t)
print(tf.strings.length(t))
print(tf.strings.length(t, unit="UTF8_CHAR"))
print(tf.strings.unicode_decode(t, "UTF8"))

tf.Tensor(b'cafe', shape=(), dtype=string)
tf.Tensor(4, shape=(), dtype=int32)
tf.Tensor(4, shape=(), dtype=int32)
tf.Tensor([ 99  97 102 101], shape=(4,), dtype=int32)

# help(tf.strings.length)

# string array
t = tf.constant(["cafe", "coffee", "咖啡"])
print(tf.strings.length(t, unit="UTF8_CHAR"))
r = tf.strings.unicode_decode(t, "UTF8")
print(r)

tf.Tensor([4 6 2], shape=(3,), dtype=int32)
<tf.RaggedTensor [[99, 97, 102, 101], [99, 111, 102, 102, 101, 101], [21654, 21857]]>

不规则张量tf.RaggedTensor

# ragged tensor
r = tf.ragged.constant([[11, 12],
            [21, 22, 23],
            [],
            [41]])
print(r)
print(r[1])
print(r[1:2])

<tf.RaggedTensor [[11, 12], [21, 22, 23], [], [41]]>
tf.Tensor([21 22 23], shape=(3,), dtype=int32)
<tf.RaggedTensor [[21, 22, 23]]>

# ops and ragged tensor
r2 = tf.ragged.constant([[51, 52],
            [],
            [71],
            [81]])
print(tf.concat([r, r2], axis=0)) # 在行的方向拼接

<tf.RaggedTensor [[11, 12], [21, 22, 23], [], [41], [51, 52], [], [71], [81]]>

print(tf.concat([r, r2], axis=1)) # 在列的方向拼接, 行数不同时 报错

<tf.RaggedTensor [[11, 12, 51, 52], [21, 22, 23], [71], [41, 81]]>

print(r.to_tensor())

tf.Tensor(
[[11 12  0]
 [21 22 23]
 [ 0  0  0]
 [41  0  0]], shape=(4, 3), dtype=int32)

3-4 实战sparse tensor与tf.Variable

# sparse  tensor 

# 大部分为零 少部分有值 记录值对应的坐标
s = tf.SparseTensor(indices=[[0, 1], [1, 0], [2 ,3]],
            values=[1., 2., 3.],
            dense_shape = [3, 4])
print(s)
print(tf.sparse.to_dense(s))

SparseTensor(indices=tf.Tensor(
[[0 1]
 [1 0]
 [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))
tf.Tensor(
[[0. 1. 0. 0.]
 [2. 0. 0. 0.]
 [0. 0. 0. 3.]], shape=(3, 4), dtype=float32)

# ops on sparse tensor
# 乘法
s2 = s * 2.0
print(s2)

# 加法
try:
    s3 = s + 1
except TypeError as ex:
    print(ex)

SparseTensor(indices=tf.Tensor(
[[0 1]
 [1 0]
 [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([2. 4. 6.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))
unsupported operand type(s) for +: 'SparseTensor' and 'int'

s4 = tf.constant([[10., 20.],
        [30., 40.],
        [50., 60.],
        [70., 80.]])
print(tf.sparse.sparse_dense_matmul(s, s4))
# 3 * 4 4 * 2 --> 3 * 2

tf.Tensor(
[[ 30.  40.]
 [ 20.  40.]
 [210. 240.]], shape=(3, 2), dtype=float32)

# sparse  tensor 

# 大部分为零 少部分有值 记录值对应的坐标, 定义是indices 必须是排好序的
s = tf.SparseTensor(indices=[[0, 2], [0, 1], [2 ,3]],
            values=[1., 2., 3.],
            dense_shape = [3, 4]) # 报错， 加order
print(s)
print(tf.sparse.to_dense(s))

SparseTensor(indices=tf.Tensor(
[[0 2]
 [0 1]
 [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))

---------------------------------------------------------------------------

InvalidArgumentError                      Traceback (most recent call last)

<ipython-input-30-9827d8053682> in <module>()
      6             dense_shape = [3, 4])
      7 print(s)
----> 8 print(tf.sparse.to_dense(s))

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/sparse_ops.py in sparse_tensor_to_dense(sp_input, default_value, validate_indices, name)
   1647       default_value=default_value,
   1648       validate_indices=validate_indices,
-> 1649       name=name)
   1650
   1651 

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_sparse_ops.py in sparse_to_dense(sparse_indices, output_shape, sparse_values, default_value, validate_indices, name)
   3160       return _result
   3161     except _core._NotOkStatusException as e:
-> 3162       _ops.raise_from_not_ok_status(e, name)
   3163     except _core._FallbackException:
   3164       pass

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in raise_from_not_ok_status(e, name)
   6860   message = e.message + (" name: " + name if name is not None else "")
   6861   # pylint: disable=protected-access
-> 6862   six.raise_from(core._status_to_exception(e.code, message), None)
   6863   # pylint: enable=protected-access
   6864 

/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)

InvalidArgumentError: indices[1] = [0,1] is out of order. Many sparse ops require sorted indices.
    Use `tf.sparse.reorder` to create a correctly ordered copy.

 [Op:SparseToDense]

# sparse  tensor 

# 大部分为零 少部分有值 记录值对应的坐标
s5 = tf.SparseTensor(indices=[[0, 2], [0, 1], [2 ,3]],
            values=[1., 2., 3.],
            dense_shape = [3, 4]) # 报错， 加reorder
print(s5)
s6 = tf.sparse.reorder(s5)
print(tf.sparse.to_dense(s6))

SparseTensor(indices=tf.Tensor(
[[0 2]
 [0 1]
 [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))
tf.Tensor(
[[0. 2. 1. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 3.]], shape=(3, 4), dtype=float32)

变量

# variable
v = tf.Variable([[1., 2., 3.],
            [4., 5., 6.]])
print(v)
print(v.value())
print(v.numpy())

<tf.Variable 'Variable:0' shape=(2, 3) dtype=float32, numpy=
array([[1., 2., 3.],
       [4., 5., 6.]], dtype=float32)>
tf.Tensor(
[[1. 2. 3.]
 [4. 5. 6.]], shape=(2, 3), dtype=float32)
[[1. 2. 3.]
 [4. 5. 6.]]

操作上和常量差不多 但是 变量可以重新被赋值

# assign value
v.assign(2*v)
print(v.numpy())
v[0, 1].assign(44)
print(v.numpy())
v[1].assign([7., 8., 9.])
print(v.numpy())

[[ 2.  4.  6.]
 [ 8. 10. 12.]]
[[ 2. 44.  6.]
 [ 8. 10. 12.]]
[[ 2. 44.  6.]
 [ 7.  8.  9.]]

try:
    v[1] = [7., 8., 9.]
except TypeError as ex:
    print(ex)

'ResourceVariable' object does not support item assignment

3-5 实战自定义损失函数与DenseLayer回顾】

# tf_keras_regression_customized_loss

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras

# 新数据集---房价预测
from sklearn.datasets import fetch_california_housing 

housing = fetch_california_housing()
# print(housing.DESCR)
print(housing.data.shape)
print(housing.target.shape)

(20640, 8)
(20640,)

from sklearn.model_selection import train_test_split 

x_train_all, x_test, y_train_all, y_test = train_test_split(
    housing.data, housing.target, random_state=7, test_size=0.25
) # 默认3:1的比例划分

x_train, x_valid, y_train, y_valid = train_test_split(x_train_all, y_train_all,
                         random_state=11)
print(x_train.shape, y_train.shape)
print(x_valid.shape, y_valid.shape)
print(x_test.shape, y_test.shape)

(11610, 8) (11610,)
(3870, 8) (3870,)
(5160, 8) (5160,)

# 归一化

from sklearn.preprocessing import StandardScaler 

scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train)
# 获取训练集的均值和方差， 在测试集和验证集上都用和训练集一样的均值和方差
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)

# 搭建模型
model = keras.models.Sequential([
    keras.layers.Dense(30, activation="relu",
        input_shape=x_train.shape[1:]), # 8
    keras.layers.Dense(1),
])

# 自定义实现 loss
def customized_mse(y_true, y_pred):
    return tf.reduce_mean(tf.square(y_pred - y_true))

model.summary() # model 信息
model.compile(loss=customized_mse, optimizer="sgd",
              metrics=["mean_squared_error"]) # 编译model， 设置损失函数和优化方法
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-3
)]

Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_4 (Dense)              (None, 30)                270
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 31
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________

# 训练模型
history = model.fit(x_train_scaled, y_train,
        validation_data=(x_valid_scaled, y_valid),
        epochs = 100,
        callbacks=callbacks)

Epoch 1/100
363/363 [==============================] - 1s 2ms/step - loss: 1.4005 - mean_squared_error: 1.4005 - val_loss: 0.8243 - val_mean_squared_error: 0.8243
Epoch 2/100
363/363 [==============================] - 1s 2ms/step - loss: 0.5414 - mean_squared_error: 0.5414 - val_loss: 0.4819 - val_mean_squared_error: 0.4819
Epoch 3/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4678 - mean_squared_error: 0.4678 - val_loss: 0.4576 - val_mean_squared_error: 0.4576
Epoch 4/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4356 - mean_squared_error: 0.4356 - val_loss: 0.4384 - val_mean_squared_error: 0.4384
Epoch 5/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4149 - mean_squared_error: 0.4149 - val_loss: 0.4267 - val_mean_squared_error: 0.4267
Epoch 6/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3944 - mean_squared_error: 0.3944 - val_loss: 0.4151 - val_mean_squared_error: 0.4151
Epoch 7/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4057 - mean_squared_error: 0.4057 - val_loss: 0.4081 - val_mean_squared_error: 0.4081
Epoch 8/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3853 - mean_squared_error: 0.3853 - val_loss: 0.4026 - val_mean_squared_error: 0.4026
Epoch 9/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4058 - mean_squared_error: 0.4058 - val_loss: 0.3977 - val_mean_squared_error: 0.3977
Epoch 10/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3763 - mean_squared_error: 0.3763 - val_loss: 0.4614 - val_mean_squared_error: 0.4614
Epoch 11/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3790 - mean_squared_error: 0.3790 - val_loss: 0.3895 - val_mean_squared_error: 0.3895
Epoch 12/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3818 - mean_squared_error: 0.3818 - val_loss: 0.3892 - val_mean_squared_error: 0.3892
Epoch 13/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3694 - mean_squared_error: 0.3694 - val_loss: 0.3885 - val_mean_squared_error: 0.3885
Epoch 14/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3850 - mean_squared_error: 0.3850 - val_loss: 4.0212 - val_mean_squared_error: 4.0212
Epoch 15/100
363/363 [==============================] - 1s 2ms/step - loss: 1.3281 - mean_squared_error: 1.3281 - val_loss: 0.4364 - val_mean_squared_error: 0.4364
Epoch 16/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4003 - mean_squared_error: 0.4003 - val_loss: 0.4188 - val_mean_squared_error: 0.4188
Epoch 17/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4221 - mean_squared_error: 0.4221 - val_loss: 0.4053 - val_mean_squared_error: 0.4053
Epoch 18/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3889 - mean_squared_error: 0.3889 - val_loss: 0.3960 - val_mean_squared_error: 0.3960

tf_keras_regression_customized_layer

layer = tf.keras.layers.Dense(100)
layer = tf.keras.layers.Dense(100, input_shape=(None, 5))

layer(tf.zeros([10, 5]))

<tf.Tensor: shape=(10, 100), dtype=float32, numpy=
array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.]], dtype=float32)>

layer.variables

[<tf.Variable 'dense_13/kernel:0' shape=(5, 100) dtype=float32, numpy=
 array([[ 1.17993578e-01, -1.56830609e-01, -1.68948278e-01,
         -1.82192534e-01,  6.65033609e-02, -1.16530903e-01,
         -1.95161447e-01,  1.63546160e-01, -3.72667760e-02,
         -2.11995542e-02,  6.36704713e-02, -1.72894925e-01,
          1.50540575e-01, -9.82312113e-02, -1.02308378e-01,
          4.88599390e-02,  2.24954262e-01, -3.72949243e-02,
         -6.01941198e-02, -8.99997205e-02, -4.77534682e-02,
         -5.69917411e-02,  1.30270258e-01, -1.73452944e-02,
         -1.83161646e-02, -2.14328170e-02,  3.06069851e-03,
          2.37779632e-01, -2.07491070e-01,  8.40653330e-02,
          1.27209827e-01, -4.37220931e-03, -1.95200533e-01,
         -2.04393506e-01, -4.98728454e-03,  1.86599448e-01,
         -1.35822102e-01, -5.09840548e-02, -1.41907871e-01,
          7.33199269e-02,  1.16105333e-01,  7.80275017e-02,
          1.39858052e-01,  4.97041196e-02, -1.35599136e-01,
          2.15787813e-01, -6.73650205e-03, -1.62299246e-01,
         -1.10671803e-01, -4.88808006e-02,  2.31273606e-01,
          1.50902942e-01, -1.73787475e-02, -1.44821286e-01,
          5.45063466e-02,  1.29491672e-01, -1.95474163e-01,
         -9.48164761e-02, -1.92742497e-01, -1.07271940e-01,
          8.99470598e-02, -1.46985203e-01,  2.34416828e-01,
         -1.50809363e-01, -8.69280249e-02,  1.96190968e-01,
         -1.76819354e-01, -2.16299266e-01,  2.18500644e-02,
          1.97046891e-01,  7.25797564e-02, -8.77934098e-02,
          1.75517634e-01, -3.91519219e-02, -2.15126351e-01,
          1.78521171e-01,  2.13996023e-02, -2.22161174e-01,
          5.66680878e-02,  2.03519017e-02,  9.29586738e-02,
         -2.26843029e-01,  1.87522843e-01, -3.30755711e-02,
          3.09066474e-03,  5.06575555e-02, -8.58460814e-02,
         -1.94824383e-01, -1.42128766e-01,  1.60556123e-01,
          1.36067107e-01, -3.94089669e-02,  1.44759908e-01,
          1.82055071e-01, -2.00239629e-01,  2.06142664e-04,
         -6.84306324e-02,  9.27842110e-02, -1.89147264e-01,
          5.41335493e-02],
        [-2.35774666e-01,  1.62306562e-01, -1.44630760e-01,
         -8.45796913e-02, -8.72821212e-02,  2.13336393e-01,
         -2.05078036e-01, -2.36002699e-01,  1.14917606e-02,
         -8.91554207e-02, -3.06088030e-02, -3.10562551e-02,
          1.59979239e-01, -2.04142958e-01,  2.33666494e-01,
         -2.12851822e-01, -1.70270622e-01,  2.00299934e-01,
          2.05543056e-01, -1.55235946e-01, -2.28202820e-01,
          2.17619970e-01,  2.18072161e-01,  1.95354074e-02,
         -1.49654686e-01, -1.23770259e-01,  6.42647892e-02,
         -1.08850196e-01,  9.67449993e-02, -8.77622366e-02,
         -1.59778982e-01,  2.36193791e-01, -6.45806938e-02,
         -2.33155161e-01,  2.21592590e-01,  3.21660191e-02,
         -1.54520422e-02,  1.38404980e-01,  1.87509850e-01,
         -1.02676094e-01, -4.30940092e-02,  6.22577816e-02,
          6.93489462e-02, -6.09517097e-03,  8.23376328e-02,
          1.77095875e-01,  3.80660444e-02,  1.94630966e-01,
          2.13605389e-01,  5.98717183e-02,  1.16292104e-01,
          1.80968598e-01,  8.10724050e-02,  1.64164022e-01,
         -2.13646024e-01, -1.89710647e-01, -7.94855505e-02,
         -2.09293514e-02, -2.83155143e-02,  5.62551767e-02,
          2.36149564e-01,  1.81217536e-01, -1.16601914e-01,
          8.40740353e-02,  2.16404483e-01,  1.83589146e-01,
         -1.59670919e-01,  1.21718511e-01, -9.71012712e-02,
         -1.50169045e-01,  6.36996478e-02, -6.35278672e-02,
         -1.94793671e-01,  3.13182026e-02,  1.83542356e-01,
          5.53140491e-02, -2.05091044e-01,  2.21068114e-02,
         -1.81381866e-01,  6.48529977e-02,  2.29394928e-01,
         -1.72381476e-01,  4.47730273e-02,  8.43436271e-02,
         -3.61360312e-02,  6.97026402e-02, -2.35724866e-01,
         -7.47385472e-02, -1.06989443e-02, -2.08874524e-01,
          2.18377754e-01,  4.11485583e-02,  1.33517995e-01,
         -2.23670855e-01,  1.06577322e-01,  7.57554322e-02,
          2.27842227e-01,  2.28776678e-01, -2.28846490e-01,
          1.96350500e-01],
        [-1.80571258e-01, -1.31611973e-01, -1.23560011e-01,
         -7.25644827e-03,  1.68194845e-01,  6.39882535e-02,
          6.49448186e-02, -1.11236885e-01,  1.77815989e-01,
         -1.61307067e-01, -1.19757392e-01, -7.55182654e-02,
          2.20785186e-01,  1.58575580e-01, -6.70485497e-02,
         -2.44060904e-02, -1.02572650e-01,  9.91210490e-02,
         -1.15951687e-01,  2.29302540e-01,  8.92989784e-02,
         -2.97219902e-02,  1.64928883e-02,  7.41528422e-02,
         -4.58508730e-03,  4.78822142e-02,  1.90612510e-01,
         -1.20980740e-01, -1.25501931e-01, -9.63088870e-03,
         -1.38515472e-01, -3.43029052e-02,  1.24467507e-01,
         -2.04019919e-01, -1.58125103e-01,  1.12259522e-01,
         -8.98374617e-03, -5.39632589e-02,  1.38579264e-01,
          1.14018604e-01, -2.27418706e-01, -1.87457830e-01,
         -3.28620225e-02,  1.46933049e-02,  1.15428194e-01,
          1.17602482e-01,  2.14431331e-01, -2.33065113e-01,
          1.73265323e-01, -2.32133225e-01, -4.83830720e-02,
         -1.78886086e-01, -7.13968277e-02, -9.14449543e-02,
          7.23969489e-02,  2.16047019e-02, -1.33315265e-01,
         -1.70755699e-01,  1.27966836e-01,  8.60891491e-02,
         -1.82814598e-01, -1.57531857e-01, -1.55951619e-01,
         -1.41447648e-01,  1.39176831e-01, -5.84817082e-02,
          2.34167427e-02,  1.52700737e-01,  1.73719719e-01,
         -9.27354395e-03,  1.23508707e-01, -2.09955722e-01,
         -1.92015380e-01, -1.34306997e-01,  1.66582316e-02,
          5.39450794e-02, -1.93378761e-01,  1.74318030e-01,
          2.04276726e-01,  1.46198407e-01, -5.38525730e-02,
          1.60146400e-01,  4.74558622e-02, -1.20019168e-02,
          8.51270407e-02, -2.31632307e-01,  1.99092522e-01,
         -1.06971711e-02, -1.39157385e-01, -5.24209440e-03,
         -2.29086205e-01, -1.57549918e-01, -8.20409954e-02,
         -1.32332414e-01, -7.42692649e-02, -1.26496121e-01,
         -8.07453245e-02,  1.90312937e-01, -2.31800556e-01,
          7.24865943e-02],
        [-2.38251865e-01, -8.11809748e-02, -1.37726456e-01,
         -1.83995217e-01,  8.50348324e-02,  1.09083399e-01,
         -9.14118886e-02, -1.49630919e-01, -1.63562000e-01,
         -1.64833844e-01,  1.30269751e-01, -1.89413071e-01,
          2.07567438e-01, -1.62040338e-01,  1.83953390e-01,
          2.37112477e-01, -6.06615841e-03, -3.92208397e-02,
          1.96731701e-01,  2.31520727e-01,  1.60958663e-01,
          7.02184886e-02,  5.88749200e-02,  1.82619542e-02,
         -1.63384855e-01,  1.02474988e-02,  2.17420712e-01,
         -1.44961655e-01, -2.28492171e-02, -5.26894629e-02,
          7.99415559e-02, -1.84868455e-01,  1.62348434e-01,
          1.73505023e-01,  1.00375012e-01, -1.58093065e-01,
          6.42622262e-02, -2.37627506e-01,  2.25016460e-01,
          2.12048039e-01, -9.22474116e-02,  1.91477999e-01,
          1.17883161e-01, -3.78689617e-02, -9.79572535e-03,
          1.36944205e-02, -1.98942423e-02, -9.28505063e-02,
         -7.69308954e-02, -2.62513459e-02, -2.00397670e-01,
          6.66552335e-02, -2.25662813e-01, -1.69921950e-01,
          1.39344946e-01, -7.63480812e-02,  7.96660930e-02,
         -1.55795783e-01, -1.60507798e-01,  1.19389847e-01,
          1.02434680e-01, -2.25314021e-01, -2.31452614e-01,
         -1.20137990e-01,  6.83608800e-02,  9.86010283e-02,
         -1.89073458e-01, -2.05998704e-01,  7.78703243e-02,
          2.14689508e-01, -1.34084448e-01, -1.57677472e-01,
          2.91350335e-02,  1.68535188e-01, -1.82089433e-01,
          1.78327605e-01,  8.48407894e-02, -2.09998935e-01,
          2.70417184e-02,  1.46897361e-01, -9.30094570e-02,
         -9.73228067e-02,  1.46372929e-01, -1.51964039e-01,
         -1.68384731e-01, -2.12973908e-01,  2.20886931e-01,
         -1.92470863e-01,  7.41604716e-02, -6.24230653e-02,
         -2.22013846e-01, -2.22155869e-01,  2.05529436e-01,
         -4.03873622e-02,  2.18250945e-01,  1.45619407e-01,
          6.55782372e-02,  2.90372223e-02, -7.39094764e-02,
          2.33842507e-01],
        [-3.21059525e-02, -2.31029272e-01, -7.09910989e-02,
         -1.42410144e-01, -2.38033593e-01,  1.03846192e-04,
          1.43522993e-01,  1.84639975e-01, -1.14365458e-01,
         -1.46784991e-01,  7.51944333e-02,  1.45526960e-01,
          1.94465563e-01,  1.27261981e-01,  2.14170590e-01,
         -2.32892483e-01,  1.89018622e-01,  1.28891185e-01,
         -9.46170092e-02, -1.20265312e-01,  5.48033863e-02,
          1.21932521e-01,  9.82101113e-02, -1.54365391e-01,
          2.25261077e-01,  8.98155719e-02, -1.74374431e-02,
          2.40292400e-02, -1.73144549e-01,  1.07187942e-01,
         -6.28392845e-02, -6.04035109e-02, -2.25900114e-02,
         -2.07246631e-01,  3.58460397e-02,  1.73451409e-01,
         -2.97844410e-04, -2.09331259e-01,  1.97884634e-01,
         -3.29386741e-02, -2.04553649e-01, -1.17815696e-01,
          8.26779455e-02,  6.88602477e-02, -2.07942799e-01,
          6.44993037e-02,  2.22308889e-01,  1.84164539e-01,
          1.41484395e-01, -1.78974763e-01, -1.72387511e-01,
          2.00336799e-01, -1.69618338e-01, -1.97233930e-01,
          2.36409739e-01,  8.74768645e-02, -3.40589881e-03,
         -3.98442745e-02, -2.22937524e-01, -2.34544247e-01,
          1.20113596e-01, -1.76493973e-02, -5.34559786e-03,
          1.50108486e-02, -4.26822454e-02, -1.42341807e-01,
          4.99777347e-02,  1.44065902e-01, -2.24418148e-01,
         -1.34305060e-02, -2.15131775e-01,  2.36514106e-01,
         -1.04367465e-01, -3.85250002e-02,  3.09597105e-02,
         -2.15378776e-01,  1.44013315e-02, -9.90413129e-02,
          1.30114406e-02,  2.33711973e-01,  1.99497864e-01,
          2.61102766e-02, -1.42305970e-01,  8.29340070e-02,
          1.37307495e-02, -1.73510849e-01,  8.05871636e-02,
          1.65648311e-02,  2.38432735e-02,  1.03347406e-01,
          1.21760681e-01,  2.23188981e-01,  1.13190249e-01,
          1.19871601e-01, -6.25639409e-02, -1.89669833e-01,
          3.46633345e-02, -1.13988906e-01,  4.01383489e-02,
          2.32083932e-01]], dtype=float32)>,
 <tf.Variable 'dense_13/bias:0' shape=(100,) dtype=float32, numpy=
 array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       dtype=float32)>]

layer.trainable_variables

[<tf.Variable 'dense_13/kernel:0' shape=(5, 100) dtype=float32, numpy=
 array([[ 1.17993578e-01, -1.56830609e-01, -1.68948278e-01,
         -1.82192534e-01,  6.65033609e-02, -1.16530903e-01,
         -1.95161447e-01,  1.63546160e-01, -3.72667760e-02,
         -2.11995542e-02,  6.36704713e-02, -1.72894925e-01,
          1.50540575e-01, -9.82312113e-02, -1.02308378e-01,
          4.88599390e-02,  2.24954262e-01, -3.72949243e-02,
         -6.01941198e-02, -8.99997205e-02, -4.77534682e-02,
         -5.69917411e-02,  1.30270258e-01, -1.73452944e-02,
         -1.83161646e-02, -2.14328170e-02,  3.06069851e-03,
          2.37779632e-01, -2.07491070e-01,  8.40653330e-02,
          1.27209827e-01, -4.37220931e-03, -1.95200533e-01,
         -2.04393506e-01, -4.98728454e-03,  1.86599448e-01,
         -1.35822102e-01, -5.09840548e-02, -1.41907871e-01,
          7.33199269e-02,  1.16105333e-01,  7.80275017e-02,
          1.39858052e-01,  4.97041196e-02, -1.35599136e-01,
          2.15787813e-01, -6.73650205e-03, -1.62299246e-01,
         -1.10671803e-01, -4.88808006e-02,  2.31273606e-01,
          1.50902942e-01, -1.73787475e-02, -1.44821286e-01,
          5.45063466e-02,  1.29491672e-01, -1.95474163e-01,
         -9.48164761e-02, -1.92742497e-01, -1.07271940e-01,
          8.99470598e-02, -1.46985203e-01,  2.34416828e-01,
         -1.50809363e-01, -8.69280249e-02,  1.96190968e-01,
         -1.76819354e-01, -2.16299266e-01,  2.18500644e-02,
          1.97046891e-01,  7.25797564e-02, -8.77934098e-02,
          1.75517634e-01, -3.91519219e-02, -2.15126351e-01,
          1.78521171e-01,  2.13996023e-02, -2.22161174e-01,
          5.66680878e-02,  2.03519017e-02,  9.29586738e-02,
         -2.26843029e-01,  1.87522843e-01, -3.30755711e-02,
          3.09066474e-03,  5.06575555e-02, -8.58460814e-02,
         -1.94824383e-01, -1.42128766e-01,  1.60556123e-01,
          1.36067107e-01, -3.94089669e-02,  1.44759908e-01,
          1.82055071e-01, -2.00239629e-01,  2.06142664e-04,
         -6.84306324e-02,  9.27842110e-02, -1.89147264e-01,
          5.41335493e-02],
        [-2.35774666e-01,  1.62306562e-01, -1.44630760e-01,
         -8.45796913e-02, -8.72821212e-02,  2.13336393e-01,
         -2.05078036e-01, -2.36002699e-01,  1.14917606e-02,
         -8.91554207e-02, -3.06088030e-02, -3.10562551e-02,
          1.59979239e-01, -2.04142958e-01,  2.33666494e-01,
         -2.12851822e-01, -1.70270622e-01,  2.00299934e-01,
          2.05543056e-01, -1.55235946e-01, -2.28202820e-01,
          2.17619970e-01,  2.18072161e-01,  1.95354074e-02,
         -1.49654686e-01, -1.23770259e-01,  6.42647892e-02,
         -1.08850196e-01,  9.67449993e-02, -8.77622366e-02,
         -1.59778982e-01,  2.36193791e-01, -6.45806938e-02,
         -2.33155161e-01,  2.21592590e-01,  3.21660191e-02,
         -1.54520422e-02,  1.38404980e-01,  1.87509850e-01,
         -1.02676094e-01, -4.30940092e-02,  6.22577816e-02,
          6.93489462e-02, -6.09517097e-03,  8.23376328e-02,
          1.77095875e-01,  3.80660444e-02,  1.94630966e-01,
          2.13605389e-01,  5.98717183e-02,  1.16292104e-01,
          1.80968598e-01,  8.10724050e-02,  1.64164022e-01,
         -2.13646024e-01, -1.89710647e-01, -7.94855505e-02,
         -2.09293514e-02, -2.83155143e-02,  5.62551767e-02,
          2.36149564e-01,  1.81217536e-01, -1.16601914e-01,
          8.40740353e-02,  2.16404483e-01,  1.83589146e-01,
         -1.59670919e-01,  1.21718511e-01, -9.71012712e-02,
         -1.50169045e-01,  6.36996478e-02, -6.35278672e-02,
         -1.94793671e-01,  3.13182026e-02,  1.83542356e-01,
          5.53140491e-02, -2.05091044e-01,  2.21068114e-02,
         -1.81381866e-01,  6.48529977e-02,  2.29394928e-01,
         -1.72381476e-01,  4.47730273e-02,  8.43436271e-02,
         -3.61360312e-02,  6.97026402e-02, -2.35724866e-01,
         -7.47385472e-02, -1.06989443e-02, -2.08874524e-01,
          2.18377754e-01,  4.11485583e-02,  1.33517995e-01,
         -2.23670855e-01,  1.06577322e-01,  7.57554322e-02,
          2.27842227e-01,  2.28776678e-01, -2.28846490e-01,
          1.96350500e-01],
        [-1.80571258e-01, -1.31611973e-01, -1.23560011e-01,
         -7.25644827e-03,  1.68194845e-01,  6.39882535e-02,
          6.49448186e-02, -1.11236885e-01,  1.77815989e-01,
         -1.61307067e-01, -1.19757392e-01, -7.55182654e-02,
          2.20785186e-01,  1.58575580e-01, -6.70485497e-02,
         -2.44060904e-02, -1.02572650e-01,  9.91210490e-02,
         -1.15951687e-01,  2.29302540e-01,  8.92989784e-02,
         -2.97219902e-02,  1.64928883e-02,  7.41528422e-02,
         -4.58508730e-03,  4.78822142e-02,  1.90612510e-01,
         -1.20980740e-01, -1.25501931e-01, -9.63088870e-03,
         -1.38515472e-01, -3.43029052e-02,  1.24467507e-01,
         -2.04019919e-01, -1.58125103e-01,  1.12259522e-01,
         -8.98374617e-03, -5.39632589e-02,  1.38579264e-01,
          1.14018604e-01, -2.27418706e-01, -1.87457830e-01,
         -3.28620225e-02,  1.46933049e-02,  1.15428194e-01,
          1.17602482e-01,  2.14431331e-01, -2.33065113e-01,
          1.73265323e-01, -2.32133225e-01, -4.83830720e-02,
         -1.78886086e-01, -7.13968277e-02, -9.14449543e-02,
          7.23969489e-02,  2.16047019e-02, -1.33315265e-01,
         -1.70755699e-01,  1.27966836e-01,  8.60891491e-02,
         -1.82814598e-01, -1.57531857e-01, -1.55951619e-01,
         -1.41447648e-01,  1.39176831e-01, -5.84817082e-02,
          2.34167427e-02,  1.52700737e-01,  1.73719719e-01,
         -9.27354395e-03,  1.23508707e-01, -2.09955722e-01,
         -1.92015380e-01, -1.34306997e-01,  1.66582316e-02,
          5.39450794e-02, -1.93378761e-01,  1.74318030e-01,
          2.04276726e-01,  1.46198407e-01, -5.38525730e-02,
          1.60146400e-01,  4.74558622e-02, -1.20019168e-02,
          8.51270407e-02, -2.31632307e-01,  1.99092522e-01,
         -1.06971711e-02, -1.39157385e-01, -5.24209440e-03,
         -2.29086205e-01, -1.57549918e-01, -8.20409954e-02,
         -1.32332414e-01, -7.42692649e-02, -1.26496121e-01,
         -8.07453245e-02,  1.90312937e-01, -2.31800556e-01,
          7.24865943e-02],
        [-2.38251865e-01, -8.11809748e-02, -1.37726456e-01,
         -1.83995217e-01,  8.50348324e-02,  1.09083399e-01,
         -9.14118886e-02, -1.49630919e-01, -1.63562000e-01,
         -1.64833844e-01,  1.30269751e-01, -1.89413071e-01,
          2.07567438e-01, -1.62040338e-01,  1.83953390e-01,
          2.37112477e-01, -6.06615841e-03, -3.92208397e-02,
          1.96731701e-01,  2.31520727e-01,  1.60958663e-01,
          7.02184886e-02,  5.88749200e-02,  1.82619542e-02,
         -1.63384855e-01,  1.02474988e-02,  2.17420712e-01,
         -1.44961655e-01, -2.28492171e-02, -5.26894629e-02,
          7.99415559e-02, -1.84868455e-01,  1.62348434e-01,
          1.73505023e-01,  1.00375012e-01, -1.58093065e-01,
          6.42622262e-02, -2.37627506e-01,  2.25016460e-01,
          2.12048039e-01, -9.22474116e-02,  1.91477999e-01,
          1.17883161e-01, -3.78689617e-02, -9.79572535e-03,
          1.36944205e-02, -1.98942423e-02, -9.28505063e-02,
         -7.69308954e-02, -2.62513459e-02, -2.00397670e-01,
          6.66552335e-02, -2.25662813e-01, -1.69921950e-01,
          1.39344946e-01, -7.63480812e-02,  7.96660930e-02,
         -1.55795783e-01, -1.60507798e-01,  1.19389847e-01,
          1.02434680e-01, -2.25314021e-01, -2.31452614e-01,
         -1.20137990e-01,  6.83608800e-02,  9.86010283e-02,
         -1.89073458e-01, -2.05998704e-01,  7.78703243e-02,
          2.14689508e-01, -1.34084448e-01, -1.57677472e-01,
          2.91350335e-02,  1.68535188e-01, -1.82089433e-01,
          1.78327605e-01,  8.48407894e-02, -2.09998935e-01,
          2.70417184e-02,  1.46897361e-01, -9.30094570e-02,
         -9.73228067e-02,  1.46372929e-01, -1.51964039e-01,
         -1.68384731e-01, -2.12973908e-01,  2.20886931e-01,
         -1.92470863e-01,  7.41604716e-02, -6.24230653e-02,
         -2.22013846e-01, -2.22155869e-01,  2.05529436e-01,
         -4.03873622e-02,  2.18250945e-01,  1.45619407e-01,
          6.55782372e-02,  2.90372223e-02, -7.39094764e-02,
          2.33842507e-01],
        [-3.21059525e-02, -2.31029272e-01, -7.09910989e-02,
         -1.42410144e-01, -2.38033593e-01,  1.03846192e-04,
          1.43522993e-01,  1.84639975e-01, -1.14365458e-01,
         -1.46784991e-01,  7.51944333e-02,  1.45526960e-01,
          1.94465563e-01,  1.27261981e-01,  2.14170590e-01,
         -2.32892483e-01,  1.89018622e-01,  1.28891185e-01,
         -9.46170092e-02, -1.20265312e-01,  5.48033863e-02,
          1.21932521e-01,  9.82101113e-02, -1.54365391e-01,
          2.25261077e-01,  8.98155719e-02, -1.74374431e-02,
          2.40292400e-02, -1.73144549e-01,  1.07187942e-01,
         -6.28392845e-02, -6.04035109e-02, -2.25900114e-02,
         -2.07246631e-01,  3.58460397e-02,  1.73451409e-01,
         -2.97844410e-04, -2.09331259e-01,  1.97884634e-01,
         -3.29386741e-02, -2.04553649e-01, -1.17815696e-01,
          8.26779455e-02,  6.88602477e-02, -2.07942799e-01,
          6.44993037e-02,  2.22308889e-01,  1.84164539e-01,
          1.41484395e-01, -1.78974763e-01, -1.72387511e-01,
          2.00336799e-01, -1.69618338e-01, -1.97233930e-01,
          2.36409739e-01,  8.74768645e-02, -3.40589881e-03,
         -3.98442745e-02, -2.22937524e-01, -2.34544247e-01,
          1.20113596e-01, -1.76493973e-02, -5.34559786e-03,
          1.50108486e-02, -4.26822454e-02, -1.42341807e-01,
          4.99777347e-02,  1.44065902e-01, -2.24418148e-01,
         -1.34305060e-02, -2.15131775e-01,  2.36514106e-01,
         -1.04367465e-01, -3.85250002e-02,  3.09597105e-02,
         -2.15378776e-01,  1.44013315e-02, -9.90413129e-02,
          1.30114406e-02,  2.33711973e-01,  1.99497864e-01,
          2.61102766e-02, -1.42305970e-01,  8.29340070e-02,
          1.37307495e-02, -1.73510849e-01,  8.05871636e-02,
          1.65648311e-02,  2.38432735e-02,  1.03347406e-01,
          1.21760681e-01,  2.23188981e-01,  1.13190249e-01,
          1.19871601e-01, -6.25639409e-02, -1.89669833e-01,
          3.46633345e-02, -1.13988906e-01,  4.01383489e-02,
          2.32083932e-01]], dtype=float32)>,
 <tf.Variable 'dense_13/bias:0' shape=(100,) dtype=float32, numpy=
 array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       dtype=float32)>]

3-6 使子类与lambda分别实战自定义层次(上)

# tf_keras_regression_customized_loss

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras 

# 新数据集---房价预测
from sklearn.datasets import fetch_california_housing 

housing = fetch_california_housing()

from sklearn.model_selection import train_test_split 

x_train_all, x_test, y_train_all, y_test = train_test_split(
    housing.data, housing.target, random_state=7, test_size=0.25
) # 默认3:1的比例划分

x_train, x_valid, y_train, y_valid = train_test_split(x_train_all, y_train_all,
                         random_state=11)

# 归一化

from sklearn.preprocessing import StandardScaler 

scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train)
# 获取训练集的均值和方差， 在测试集和验证集上都用和训练集一样的均值和方差
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)

# 自定义 Dense layer
class CustomizedDenseLayer(keras.layers.Layer):
    def __init__(self, units, activation=None, **kwargs):
        self.units = units
        self.activation = keras.layers.Activation(activation)
        super(CustomizedDenseLayer, self).__init__(**kwargs) # 继承父类

    def build(self, input_shape):
        """构建所需要的的参数"""
        # x*w+b, x_input_shape:[None, a] w: [a, b] output_shape: [None, b]
        self.kernel = self.add_weight(name='kernel',
                        shape=(input_shape[1], self.units),
                        initializer='uniform',
                        trainable=True)
        self.bias = self.add_weight(name="bias",
                            shape=(self.units,),
                            initializer='zeros',
                            trainable=True)
        super(CustomizedDenseLayer, self).build(input_shape)        

    def call(self, x):
        """完成正向计算"""
        return self.activation(x @ self.kernel + self.bias)

# 搭建模型
model = keras.models.Sequential([
    # 使用自定义模型
    CustomizedDenseLayer(30, activation="relu",
        input_shape=x_train.shape[1:]), # 8
    CustomizedDenseLayer(1),
])

# 自定义实现 loss
def customized_mse(y_true, y_pred):
    return tf.reduce_mean(tf.square(y_pred - y_true))

model.summary() # model 信息
model.compile(loss="mean_squared_error", optimizer="sgd",
            #   metrics=["mean_squared_error"]
              ) # 编译model， 设置损失函数和优化方法
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-3
)]

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
customized_dense_layer_2 (Cu (None, 30)                270
_________________________________________________________________
customized_dense_layer_3 (Cu (None, 1)                 31
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________

# 训练模型
history = model.fit(x_train_scaled, y_train,
        validation_data=(x_valid_scaled, y_valid),
        epochs = 100,
        callbacks=callbacks)

Epoch 1/100
363/363 [==============================] - 1s 2ms/step - loss: 1.9831 - val_loss: 0.6361
Epoch 2/100
363/363 [==============================] - 1s 2ms/step - loss: 0.5850 - val_loss: 0.5458
Epoch 3/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4969 - val_loss: 0.4914
Epoch 4/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4590 - val_loss: 0.4655
Epoch 5/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4769 - val_loss: 0.4931
Epoch 6/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4545 - val_loss: 0.4539
Epoch 7/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4269 - val_loss: 0.4403
Epoch 8/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4318 - val_loss: 0.4405
Epoch 9/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4151 - val_loss: 0.4258
Epoch 10/100
363/363 [==============================] - 1s 2ms/step - loss: 0.4172 - val_loss: 0.4235
Epoch 11/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3940 - val_loss: 0.4142
Epoch 12/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3901 - val_loss: 0.4146
Epoch 13/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3998 - val_loss: 0.4065
Epoch 14/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3919 - val_loss: 0.4095
Epoch 15/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3834 - val_loss: 0.4207
Epoch 16/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3896 - val_loss: 0.4005
Epoch 17/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3946 - val_loss: 0.4030
Epoch 18/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3801 - val_loss: 0.3948
Epoch 19/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3827 - val_loss: 0.3895
Epoch 20/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3800 - val_loss: 0.3902
Epoch 21/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3633 - val_loss: 0.3872
Epoch 22/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3735 - val_loss: 0.3972
Epoch 23/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3738 - val_loss: 0.3869
Epoch 24/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3586 - val_loss: 0.3856
Epoch 25/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3814 - val_loss: 0.3863
Epoch 26/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3847 - val_loss: 0.3786
Epoch 27/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3739 - val_loss: 0.3873
Epoch 28/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3533 - val_loss: 0.3825
Epoch 29/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3687 - val_loss: 0.3789
Epoch 30/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3562 - val_loss: 0.3709
Epoch 31/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3634 - val_loss: 0.3731
Epoch 32/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3540 - val_loss: 0.3733
Epoch 33/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3695 - val_loss: 0.3691
Epoch 34/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3617 - val_loss: 0.3683
Epoch 35/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3590 - val_loss: 0.3680
Epoch 36/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3552 - val_loss: 0.3677
Epoch 37/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3615 - val_loss: 0.3633
Epoch 38/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3575 - val_loss: 0.3713
Epoch 39/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3507 - val_loss: 0.3628
Epoch 40/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3616 - val_loss: 0.3602
Epoch 41/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3494 - val_loss: 0.3587
Epoch 42/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3450 - val_loss: 0.3582
Epoch 43/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3341 - val_loss: 0.3628
Epoch 44/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3453 - val_loss: 0.3559
Epoch 45/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3407 - val_loss: 0.3568
Epoch 46/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3479 - val_loss: 0.3579
Epoch 47/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3447 - val_loss: 0.3535
Epoch 48/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3625 - val_loss: 0.3585
Epoch 49/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3438 - val_loss: 0.3572
Epoch 50/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3450 - val_loss: 0.3520
Epoch 51/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3386 - val_loss: 0.3530
Epoch 52/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3554 - val_loss: 0.3494
Epoch 53/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3391 - val_loss: 0.3469
Epoch 54/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3333 - val_loss: 0.3488
Epoch 55/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3352 - val_loss: 0.3556
Epoch 56/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3308 - val_loss: 0.3467
Epoch 57/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3416 - val_loss: 0.3530
Epoch 58/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3401 - val_loss: 0.3450
Epoch 59/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3346 - val_loss: 0.3542
Epoch 60/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3224 - val_loss: 0.3434
Epoch 61/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3327 - val_loss: 0.3414
Epoch 62/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3411 - val_loss: 0.3411
Epoch 63/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3196 - val_loss: 0.3377
Epoch 64/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3287 - val_loss: 0.3360
Epoch 65/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3251 - val_loss: 0.3356
Epoch 66/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3335 - val_loss: 0.3405
Epoch 67/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3318 - val_loss: 0.3373
Epoch 68/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3120 - val_loss: 0.3331
Epoch 69/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3273 - val_loss: 0.3321
Epoch 70/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3355 - val_loss: 0.3318
Epoch 71/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3363 - val_loss: 0.3292
Epoch 72/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3238 - val_loss: 0.3281
Epoch 73/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3234 - val_loss: 0.3329
Epoch 74/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3263 - val_loss: 0.3297
Epoch 75/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3267 - val_loss: 0.3319
Epoch 76/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3172 - val_loss: 0.3298
Epoch 77/100
363/363 [==============================] - 1s 2ms/step - loss: 0.3290 - val_loss: 0.3279

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0, 1.3)
    plt.show()
plot_learning_curves(history)

model.evaluate(x_test_scaled, y_test)

162/162 [==============================] - 0s 1ms/step - loss: 0.3403

0.34025290608406067

3-6 使子类与lambda分别实战自定义层次(下)

# tf.nn.softplus: log(1+e^x)
customized_softplus = keras.layers.Lambda(lambda x: tf.nn.softplus(x))
print(customized_softplus([-10., -5., 0., 5., 10.]))

tf.Tensor([4.5398901e-05 6.7153485e-03 6.9314718e-01 5.0067153e+00 1.0000046e+01], shape=(5,), dtype=float32)

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras 

# 新数据集---房价预测
from sklearn.datasets import fetch_california_housing 

housing = fetch_california_housing()

from sklearn.model_selection import train_test_split 

x_train_all, x_test, y_train_all, y_test = train_test_split(
    housing.data, housing.target, random_state=7, test_size=0.25
) # 默认3:1的比例划分

x_train, x_valid, y_train, y_valid = train_test_split(x_train_all, y_train_all,
                         random_state=11)

# 归一化

from sklearn.preprocessing import StandardScaler 

scaler = StandardScaler()

x_train_scaled = scaler.fit_transform(x_train)
# 获取训练集的均值和方差， 在测试集和验证集上都用和训练集一样的均值和方差
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)

# 自定义 Dense layer
class CustomizedDenseLayer(keras.layers.Layer):
    def __init__(self, units, activation=None, **kwargs):
        self.units = units
        self.activation = keras.layers.Activation(activation)
        super(CustomizedDenseLayer, self).__init__(**kwargs) # 继承父类

    def build(self, input_shape):
        """构建所需要的的参数"""
        # x*w+b, x_input_shape:[None, a] w: [a, b] output_shape: [None, b]
        self.kernel = self.add_weight(name='kernel',
                        shape=(input_shape[1], self.units),
                        initializer='uniform',
                        trainable=True)
        self.bias = self.add_weight(name="bias",
                            shape=(self.units,),
                            initializer='zeros',
                            trainable=True)
        super(CustomizedDenseLayer, self).build(input_shape)        

    def call(self, x):
        """完成正向计算"""
        return self.activation(x @ self.kernel + self.bias)

# 搭建模型
model = keras.models.Sequential([
    # 使用自定义模型
    CustomizedDenseLayer(30, activation="relu",
        input_shape=x_train.shape[1:]), # 8
    CustomizedDenseLayer(1),
    # 加上我们自定一的softplus
    customized_softplus,
    # keras.layers.Dense(1, activation="softplus")
    # keras.layers.Dense(1), keras.layers.Activation("softplus")
])

# 自定义实现 loss
def customized_mse(y_true, y_pred):
    return tf.reduce_mean(tf.square(y_pred - y_true))

model.summary() # model 信息
model.compile(loss="mean_squared_error", optimizer="sgd",
            #   metrics=["mean_squared_error"]
              ) # 编译model， 设置损失函数和优化方法
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-3
)]

Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
customized_dense_layer_4 (Cu (None, 30)                270
_________________________________________________________________
customized_dense_layer_5 (Cu (None, 1)                 31
_________________________________________________________________
lambda (Lambda)              (None, 1)                 0
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________

3-7 tf.function函数转换

# tf.function and auto-graph
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras 

def scaled_elu(z, scale=1.0, alpha=1.0):
    # z > 0 ? scale * z: scale * alpha * tf.nn.elu(z)
    is_positive = tf.greater_equal(z, 0.0)
    return scale * tf.where(is_positive, z, alpha*tf.nn.elu(z))

print(scaled_elu(tf.constant(-3.0)))
print(scaled_elu(tf.constant([-3., -2.5])))

tf.Tensor(-0.95021296, shape=(), dtype=float32)
tf.Tensor([-0.95021296 -0.917915  ], shape=(2,), dtype=float32)

scaled_elu_tf = tf.function(scaled_elu)
print(scaled_elu_tf(tf.constant(-3.0)))
print(scaled_elu_tf(tf.constant([-3., -2.5])))

print(scaled_elu_tf.python_function is scaled_elu)

tf.Tensor(-0.95021296, shape=(), dtype=float32)
tf.Tensor([-0.95021296 -0.917915  ], shape=(2,), dtype=float32)
True

%timeit scaled_elu(tf.random.normal((1000, 1000)))
%timeit scaled_elu_tf(tf.random.normal((1000, 1000)))

The slowest run took 18.35 times longer than the fastest. This could mean that an intermediate result is being cached.
1000 loops, best of 3: 512 µs per loop
The slowest run took 100.58 times longer than the fastest. This could mean that an intermediate result is being cached.
1000 loops, best of 3: 658 µs per loop

TF2默认的即时执行模式（Eager Execution）带来了灵活及易调试的特性，但在特定的场合，例如追求高性能或部署模型时，我们依然希望使用 TensorFlow 1.X 中默认的图执行模式（Graph Execution），将模型转换为高效的 TensorFlow 图模型。此时，TensorFlow 2 为我们提供了 tf.function 模块，结合 AutoGraph 机制，使得我们仅需加入一个简单的 @tf.function 修饰符，就能轻松将模型以图执行模式运行。

tf.function 基础使用方法

在 TensorFlow 2 中，推荐使用 tf.function （而非 1.X 中的 tf.Session ）实现图执行模式，从而将模型转换为易于部署且高性能的 TensorFlow 图模型。只需要将我们希望以图执行模式运行的代码封装在一个函数内，并在函数前加上 @tf.function 即可，如下例所示。关于图执行模式的深入探讨可参考 附录 。

警告！

并不是任何函数都可以被 @tf.function 修饰！@tf.function 使用静态编译将函数内的代码转换成计算图，因此对函数内可使用的语句有一定限制（仅支持 Python 语言的一个子集），且需要函数内的操作本身能够被构建为计算图。建议在函数内只使用 TensorFlow 的原生操作，不要使用过于复杂的 Python 语句，函数参数只包括 TensorFlow 张量或 NumPy 数组，并最好是能够按照计算图的思想去构建函数（换言之，@tf.function 只是给了你一种更方便的写计算图的方法，而不是一颗能给任何函数加速的 银子弹 ）

3-8 @tf.function函数转换

# 1  + 1/2 + 1/2^2 + 。。。 + 1/2^n 

# @tf.function
def converge_to_2(n_iters):
    total = tf.constant(0.)
    increment = tf.constant(1.)
    for _ in range(n_iters):
        total += increment
        increment /= 2.0
    return total 

print(converge_to_2(20))

tf.Tensor(1.9999981, shape=(), dtype=float32)

# 上述两种方式均可以将普通python函数转换为tf中可以生成图结构

def display_tf_code(func):
    code = tf.autograph.to_code(func)
    from IPython.display import display, Markdown
    display(Markdown('```python\n{}\n```'.format(code)))
display_tf_code(scaled_elu)

def tf__scaled_elu(z, scale=None, alpha=None):
    with ag__.FunctionScope('scaled_elu', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:
        do_return = False
        retval_ = ag__.UndefinedReturnValue()
        is_positive = ag__.converted_call(ag__.ld(tf).greater_equal, (ag__.ld(z), 0.0), None, fscope)
        try:
            do_return = True
            retval_ = (ag__.ld(scale) * ag__.converted_call(ag__.ld(tf).where, (ag__.ld(is_positive), ag__.ld(z), (ag__.ld(alpha) * ag__.converted_call(ag__.ld(tf).nn.elu, (ag__.ld(z),), None, fscope))), None, fscope))
        except:
            do_return = False
            raise
        return fscope.ret(retval_, do_return)

display_tf_code(converge_to_2) # 注意注释@tf.function

def tf__converge_to(n_iters):
    with ag__.FunctionScope('converge_to_2', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:
        do_return = False
        retval_ = ag__.UndefinedReturnValue()
        total = ag__.converted_call(ag__.ld(tf).constant, (0.0,), None, fscope)
        increment = ag__.converted_call(ag__.ld(tf).constant, (1.0,), None, fscope)

        def get_state():
            return (total, increment)

        def set_state(vars_):
            nonlocal increment, total
            (total, increment) = vars_

        def loop_body(itr):
            nonlocal increment, total
            _ = itr
            total = ag__.ld(total)
            total += increment
            increment = ag__.ld(increment)
            increment /= 2.0
        _ = ag__.Undefined('_')
        ag__.for_stmt(ag__.converted_call(ag__.ld(range), (ag__.ld(n_iters),), None, fscope), None, loop_body, get_state, set_state, ('total', 'increment'), {'iterate_names': '_'})
        try:
            do_return = True
            retval_ = ag__.ld(total)
        except:
            do_return = False
            raise
        return fscope.ret(retval_, do_return)

# 变量式 

var = tf.Variable(0.)

@tf.function
def add_21():

    return var.assign_add(21)
print(add_21())

tf.Tensor(21.0, shape=(), dtype=float32)

@tf.function
def add_21():
    var = tf.Variable(0.)
    return var.assign_add(21)
print(add_21())

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-24-51cdfc619951> in <module>()
      5     var = tf.Variable(0.)
      6     return var.assign_add(21)
----> 7 print(add_21())

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    826     tracing_count = self.experimental_get_tracing_count()
    827     with trace.Trace(self._name) as tm:
--> 828       result = self._call(*args, **kwds)
    829       compiler = "xla" if self._experimental_compile else "nonXla"
    830       new_tracing_count = self.experimental_get_tracing_count()

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    886         # Lifting succeeded, so variables are initialized and we can run the
    887         # stateless function.
--> 888         return self._stateless_fn(*args, **kwds)
    889     else:
    890       _, _, _, filtered_flat_args = \

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
   2939     with self._lock:
   2940       (graph_function,
-> 2941        filtered_flat_args) = self._maybe_define_function(args, kwargs)
   2942     return graph_function._call_flat(
   2943         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3359
   3360           self._function_cache.missed.add(call_context_key)
-> 3361           graph_function = self._create_graph_function(args, kwargs)
   3362           self._function_cache.primary[cache_key] = graph_function
   3363 

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3204             arg_names=arg_names,
   3205             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3206             capture_by_value=self._capture_by_value),
   3207         self._function_attributes,
   3208         function_spec=self.function_spec,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    988         _, original_func = tf_decorator.unwrap(python_func)
    989
--> 990       func_outputs = python_func(*func_args, **func_kwargs)
    991
    992       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    632             xla_context.Exit()
    633         else:
--> 634           out = weak_wrapped_fn().__wrapped__(*args, **kwds)
    635         return out
    636 

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    975           except Exception as e:  # pylint:disable=broad-except
    976             if hasattr(e, "ag_error_metadata"):
--> 977               raise e.ag_error_metadata.to_exception(e)
    978             else:
    979               raise

ValueError: in user code:

    <ipython-input-24-51cdfc619951>:5 add_21  *
        var = tf.Variable(0.)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py:262 __call__  **
        return cls._variable_v2_call(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py:256 _variable_v2_call
        shape=shape)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py:67 getter
        return captured_getter(captured_previous, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py:731 invalid_creator_scope
        "tf.function-decorated function tried to create "

    ValueError: tf.function-decorated function tried to create variables on non-first call.

在定义神经网络时 转化为function之前 Variable 需要初始化

但constant可直接放于内部

@tf.function

def cube(z):
    return tf.pow(z, 3)

print(cube(tf.constant([1, 2, 3])))
print(cube(tf.constant([1., 2., 3.])))

tf.Tensor([ 1  8 27], shape=(3,), dtype=int32)
tf.Tensor([ 1.  8. 27.], shape=(3,), dtype=float32)

3-9 函数签名与图结构

@tf.function(input_signature=[tf.TensorSpec([None], tf.int32, name="x")])
# 指定签名，限定函数输入数据类型
def cube(z):
    return tf.pow(z, 3)

print(cube(tf.constant([1, 2, 3])))
print("_______")
try:
    print(cube(tf.constant([1., 2., 3.])))
except ValueError as ex:
    print(ex)

tf.Tensor([ 1  8 27], shape=(3,), dtype=int32)
_______
Python inputs incompatible with input_signature:
  inputs: (
    tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32))
  input_signature: (
    TensorSpec(shape=(None,), dtype=tf.int32, name='x'))

tf 中 ， 一个函数有了 input_signature 后才能够被保存为model 即 saved_model()

# @tf.function : py func -> graph 

# get_concreate_function -> add input_signature -> SavedModel

cube_func_int32 = cube.get_concrete_function(
    tf.TensorSpec([None], tf.int32))
print(cube_func_int32)

ConcreteFunction cube(z)
  Args:
    z: int32 Tensor, shape=(None,)
  Returns:
    int32 Tensor, shape=(None,)

print(cube_func_int32 is cube.get_concrete_function(
    tf.TensorSpec([5], tf.int32)))

print(cube_func_int32 is cube.get_concrete_function(
    tf.constant([1, 2, 3])))

# resilt is same input_signature is same

True
True

cube_func_int32.graph

<tensorflow.python.framework.func_graph.FuncGraph at 0x7efb83e64b00>

# 查看操作
cube_func_int32.graph.get_operations()

[<tf.Operation 'x' type=Placeholder>,
 <tf.Operation 'Pow/y' type=Const>,
 <tf.Operation 'Pow' type=Pow>,
 <tf.Operation 'Identity' type=Identity>]

pow_op = cube_func_int32.graph.get_operations()[2]
print(pow_op)

name: "Pow"
op: "Pow"
input: "x"
input: "Pow/y"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}

​

print(list(pow_op.inputs))
print("________________________")
print(list(pow_op.outputs))

[<tf.Tensor 'x:0' shape=(None,) dtype=int32>, <tf.Tensor 'Pow/y:0' shape=() dtype=int32>]
________________________
[<tf.Tensor 'Pow:0' shape=(None,) dtype=int32>]

## graph 中 也可以通过名字 进行对应的操作

cube_func_int32.graph.get_operation_by_name("x")

<tf.Operation 'x' type=Placeholder>

Placeholder 即放入一个输入 才能得到输出

cube_func_int32.graph.get_tensor_by_name("x:0")

<tf.Tensor 'x:0' shape=(None,) dtype=int32>

cube_func_int32.graph.as_graph_def()

node {
  name: "x"
  op: "Placeholder"
  attr {
    key: "_user_specified_name"
    value {
      s: "x"
    }
  }
  attr {
    key: "dtype"
    value {
      type: DT_INT32
    }
  }
  attr {
    key: "shape"
    value {
      shape {
        dim {
          size: -1
        }
      }
    }
  }
}
node {
  name: "Pow/y"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_INT32
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_INT32
        tensor_shape {
        }
        int_val: 3
      }
    }
  }
}
node {
  name: "Pow"
  op: "Pow"
  input: "x"
  input: "Pow/y"
  attr {
    key: "T"
    value {
      type: DT_INT32
    }
  }
}
node {
  name: "Identity"
  op: "Identity"
  input: "Pow"
  attr {
    key: "T"
    value {
      type: DT_INT32
    }
  }
}
versions {
  producer: 561
}

# 保存模型
# 保存模型后，如何载入，载入后的操作

3-10 近似求导

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras 

def f(x):
    return 3. * x ** 2 + 2. * x - 1 

def approximate_derivative(f, x, eps=1e-3):
    return (f(x+eps) - f(x-eps))/(2.*eps)

print(approximate_derivative(f, 1.))

7.999999999999119

def g(x1, x2):
    return (x1 + 5) * (x2 ** 2)

def approximate_gradient(g, x1, x2, eps=1e-3):
    dg_x1 = approximate_derivative(lambda x: g(x, x2), x1, eps)
    dg_x2 = approximate_derivative(lambda x: g(x1, x), x2, eps)
    return dg_x1, dg_x2 

print(approximate_gradient(g, 2., 3.))

(8.999999999993236, 41.999999999994486)

3-11 tf.GradientTape基本使用方法

x1 = tf.Variable(2.)
x2 = tf.Variable(3.)

with tf.GradientTape() as tape:
    z = g(x1, x2)

dz_x1 = tape.gradient(z, x1)
print(dz_x1)

try:
    dz_x2 = tape.gradient(z, x2)
except RuntimeError as ex:
    print(ex)
# tape的结果只能应用一次

tf.Tensor(9.0, shape=(), dtype=float32)
A non-persistent GradientTape can only be used tocompute one set of gradients (or jacobians)

x1 = tf.Variable(2.)
x2 = tf.Variable(3.)

# 指定参数 即可一直使用
with tf.GradientTape(persistent=True) as tape:
    z = g(x1, x2)

dz_x1 = tape.gradient(z, x1)
print(dz_x1)

try:
    dz_x2 = tape.gradient(z, x2)
except RuntimeError as ex:
    print(ex)
print(dz_x2)

tf.Tensor(9.0, shape=(), dtype=float32)
tf.Tensor(42.0, shape=(), dtype=float32)

注意 使用完后 需要自己手动释放内存

del tape

## 同时求导

x1 = tf.Variable(2.)
x2 = tf.Variable(3.)

with tf.GradientTape() as tape:
    z = g(x1, x2)

dz_x1x2 = tape.gradient(z, [x1, x2])
print(dz_x1x2)

[<tf.Tensor: shape=(), dtype=float32, numpy=9.0>, <tf.Tensor: shape=(), dtype=float32, numpy=42.0>]

# 常量是否可以求导
## 同时求导

x1 = tf.constant(2.)
x2 = tf.constant(3.)

with tf.GradientTape() as tape:
    z = g(x1, x2)

dz_x1x2 = tape.gradient(z, [x1, x2])
print(dz_x1x2)

[None, None]

x1 = tf.constant(2.)
x2 = tf.constant(3.)

with tf.GradientTape() as tape:
    tape.watch(x1)
    tape.watch(x2)
    z = g(x1, x2)

dz_x1x2 = tape.gradient(z, [x1, x2])
print(dz_x1x2)

[<tf.Tensor: shape=(), dtype=float32, numpy=9.0>, <tf.Tensor: shape=(), dtype=float32, numpy=42.0>]