【读书笔记】2_增强学习中的Q-Learning

本文为Thomas Simonini增强学习系列文章笔记或读后感，原文可以直接跳转到medium系列文章。

主要概念为：

Q-Learning，探讨其概念以及用Numpy实现

我们可以将二维游戏想象成平面格子，每个格子代表一个状态，并且对应了不同的动作，例如下图：

Q函数接收状态和动作两个参数并输出Q值，即在一个状态下各种动作各自未来的期望奖励。公式如下：

这里的未来期望奖励，就是当前状态下一直到结束状态（成功或失败）所获取的奖励。

Q-learning算法伪代码：

其中，更新Q值为bellman等式，如下描述：

这篇文章总体来说，非常简单，各种步骤也特别详细，告诉了我们如何计算Q-table的算法过程。但是为什么能迭代到最优，并没有给出一个比较明确的证明过程。主要也是因为采用的EE平衡问题，这个过程采用了greedy episolon的启发式算法，每次直接选取的是最大概率的action，而多次重复episode, 其实计算的是对未来的奖励积累的期望。所以从bellman等式到Q(s, a) state-value function 定义如何连接的呢？推导的公式如下：

Numpy具体实现

# -*- coding: utf-8 -*-

# pkg need
import numpy as np
import gym
import random
import time

# step 1. create the environment
env = gym.make("Taxi-v2")
env.render()  # tick and run it to see

# this game enviroment could be found detail documented at:
# https://gym.openai.com/envs/#toy_text
# pick and drop off the passenger right for -20 points,
# fail for either one will lose 10 points
# every step will decrease 1 points

# step 2. create the q-table and initialize it.
state_size = env.observation_space.n
action_size = env.action_space.n
qtable = np.zeros((state_size, action_size))
print("state size: %d, action size: %d" % qtable.shape)   # tick and run

# step 3. create the hyperparameters
total_episodes = 50000
total_test_episodes = 100
max_steps = 99

learning_rate = 0.7
gamma = 0.618

# exploration parameter
epsilon = 1.0                # exploration rate
max_epsilon = 1.0            # exploration probability at start
min_epsilon = 0.01           # minumum exploration probability
decay_rate = 0.01            # exponential rate to decay exploration rate

# step 4. The Q learning algorithm
# 2 For life or until learning is stopped
for episode in range(total_episodes):
    # reset the environment
    state = env.reset()
    step = 0
    done = False 

    # start the game
    for step in range(max_steps):
        # 3 choose an action a in the current world state (s)
        # first random a number
        ee_tradeoff = random.uniform(0, 1)

        # exploitation, taking the biggest Q value for this state
        if ee_tradeoff > epsilon:
            action = np.argmax(qtable[state, :])
        else:
            action = env.action_space.sample()    # exploration, randomly sample a action

        # take action and observe the outcome
        new_state, reward, done, info = env.step(action)

        # Update the Q(s, a)
        qtable[state, action] += learning_rate * (
            reward + gamma * np.max(qtable[new_state, :]) - qtable[state, action])

        # update state
        state = new_state

        # if done: finish episode
        if done:
            break 

    # reduce epsilon -> we want less and less exploration
    epsilon = min_epsilon + (max_epsilon - min_epsilon) * np.exp(-decay_rate * episode)

print(qtable)

# use q table to play taxi

def play(env, qtable, show=True, sec=None):
    state = env.reset()
    step = 0
    done = False
    total_rewards = 0

    for step in range(max_steps):
        # see agent to play
        if show:
            env.render()
        action = np.argmax(qtable[state, :])

        new_state, reward, done, info = env.step(action)
        total_rewards += reward 

        if done:
            break

        if sec:
            time.sleep(sec)

        state = new_state
    return total_rewards

# play one test episode
play(env, qtable)

env.reset()
rewards = []
for episode in range(total_test_episodes):
    total_rewards = play(env, qtable, show=False)
    rewards.append(total_rewards)

env.close()
print("Score over time: " + str(sum(rewards) / total_test_episodes))

参考文献：

1. bellman equation to state value function，berkeley的增强学习课程，讲的真详细。