初始 DQN 程序所遇到的问题

初始 DQN 程序所遇到的问题

　　最近在看 DQN，但是想试试别人放出来的 code，但是发现，额，各种问题，在此记录，以备不时之需！

　　问题1.

wangxiao@GTX980:~/Documents/DRL/DQN-tensorflow-master$ python main.py --env_name=Breakout-v0 --is_train=True
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:99] Couldn't open CUDA library libcudnn.so. LD_LIBRARY_PATH: /home/wangxiao/torch/install/lib:/home/wangxiao/torch/install/lib:/home/wangxiao/torch/install/lib:/home/wangxiao/torch/install/lib:
I tensorflow/stream_executor/cuda/cuda_dnn.cc:1562] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
[*] GPU : 1.0000
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX 980
major: 5 minor: 2 memoryClockRate (GHz) 1.329
pciBusID 0000:01:00.0
Total memory: 4.00GiB
Free memory: 3.58GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:755] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0)
[2016-07-03 09:14:02,576] Making new env: Breakout-v0
{'_save_step': 50000,
'_test_step': 10000,
'action_repeat': 4,
'backend': 'tf',
'batch_size': 32,
'cnn_format': 'NCHW',
'discount': 0.99,
'display': False,
'env_name': 'Breakout-v0',
'env_type': 'simple',
'ep_end': 0.1,
'ep_end_t': 1000000,
'ep_start': 1.0,
'history_length': 4,
'learn_start': 50000.0,
'learning_rate': 0.00025,
'learning_rate_decay': 0.96,
'learning_rate_decay_step': 50000,
'learning_rate_minimum': 0.00025,
'max_delta': 1,
'max_reward': 1.0,
'max_step': 50000000,
'memory_size': 1000000,
'min_delta': -1,
'min_reward': -1.0,
'model': 'm2',
'random_start': 30,
'scale': 10000,
'screen_height': 84,
'screen_width': 84,
'target_q_update_step': 10000,
'train_frequency': 4}

E tensorflow/stream_executor/cuda/cuda_driver.cc:932] failed to allocate 3.58G (3844833280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
[*] Loading checkpoints...
[!] Load FAILED: checkpoints/Breakout-v0/min_delta--1/max_delta-1/history_length-4/train_frequency-4/target_q_update_step-10000/memory_size-1000000/action_repeat-4/ep_end_t-1000000/min_reward--1.0/backend-tf/random_start-30/scale-10000/env_type-simple/learning_rate_decay_step-50000/ep_start-1.0/screen_width-84/learn_start-50000.0/cnn_format-NCHW/learning_rate-0.00025/batch_size-32/discount-0.99/max_step-50000000/max_reward-1.0/learning_rate_decay-0.96/learning_rate_minimum-0.00025/env_name-Breakout-v0/ep_end-0.1/model-m2/screen_height-84/
0%| | 49970/50000000 [01:06<18:20:10, 756.70it/s]F tensorflow/stream_executor/cuda/cuda_dnn.cc:204] could not find cudnnCreate in cudnn DSO; dlerror: /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so: undefined symbol: cudnnCreate
Aborted

那么就搜到了如下的答案： link：http://stackoverflow.com/questions/35702403/tensorflow-0-7-1-with-cuda-toolkit-7-5-and-cudnn-7-0

export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"

export CUDA_HOME=/usr/local/cuda

或者，copy the cuDNN libraries to /usr/local/cuda/lib64. 我两个同时执行的，所以不知道哪个起作用了，好吧，但是再执行，确实是变成了另一个问题，即：

E tensorflow/stream_executor/cuda/cuda_driver.cc:932] failed to allocate 3.58G (3844702208 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
[*] Loading checkpoints...
[!] Load FAILED: checkpoints/Breakout-v0/min_delta--1/max_delta-1/history_length-4/train_frequency-4/target_q_update_step-10000/memory_size-1000000/action_repeat-4/ep_end_t-1000000/min_reward--1.0/backend-tf/random_start-30/scale-10000/env_type-simple/learning_rate_decay_step-50000/ep_start-1.0/screen_width-84/learn_start-50000.0/cnn_format-NCHW/learning_rate-0.00025/batch_size-32/discount-0.99/max_step-50000000/max_reward-1.0/learning_rate_decay-0.96/learning_rate_minimum-0.00025/env_name-Breakout-v0/ep_end-0.1/model-m2/screen_height-84/
0%| | 49983/50000000 [01:04<18:01:04, 770.06it/s]F tensorflow/stream_executor/cuda/cuda_dnn.cc:220] could not find cudnnConvolutionBackwardFilter_v2 in cudnn DSO; dlerror: /usr/local/cuda/lib64/libcudnn.so: undefined symbol: cudnnConvolutionBackwardFilter_v2
Aborted
wangxiao@GTX980:~/Documents/DRL/DQN-tensorflow-master$

　　然后就发现，cudnn 这么多问题！

　　于是乎，我就将 cudnn 6.5 换成了 cudnn 7.0 版本，重新配置了一下，然后在执行：

　　仍然在加载检查点 (checkpoint)，不知道什么时候会断掉？

　　额貌似可以了。。。

　　Ok 大家发现了吧原来主要原因在于 cndnn 的版本问题。

Question 2:

　 when running the code from: https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner

　　It shown me a mistake, i.e. no module named AleWrap

　　 Don't worry, just run the following operation:

　　luarocks install https://raw.githubusercontent.com/lake4790k/xitari/master/xitari-0-0.rockspec　　

　　luarocks install https://raw.githubusercontent.com/Kaixhin/alewrap/master/alewrap-0-0.rockspec

　　luarocks install https://raw.githubusercontent.com/Kaixhin/rlenvs/master/rocks/rlenvs-scm-1.rockspec

　　everything will be ok and you will you see something like this:


Question ３:　
　
利用lua 读取和显示图像:

　　可以发现上述过程, 其实只能一张一张的展示图像, 怎么把多张图像放到一个窗口中进行显示呢? 见下图:

　　　效果如下: