caffe solver configuration

(用到一个加一个, 并非完整的介绍)

---

# lr_policy
基本的learning rate 在`solver.prototxt`中由参数`base_lr`配置.
配合`lr_policy`和其余的一些参数制定learning rate的变化策略.
## lr_policy="fixed"
在整个训练过程中learning rate不变.
## lr_policy="step"
需要另外几个参数配合:
```
base_lr: 0.01 # begin training at a learning rate of 0.01 = 1e-2

lr_policy: "step" # learning rate policy: drop the learning rate in "stepsize"

# by a factor of gamma every stepsize iterations

gamma: 0.1 # drop the learning rate by a factor of 10

# (i.e., multiply it by a factor of gamma = 0.1)

stepsize: 100000 # drop the learning rate every 100K iterations

# average_loss
相当于做了一个平滑. 控制台打印训练loss时, 当前loss为最近20个iteration的loss的平均数. 仅仅是为了显示好看, 不影响训练.
* http://stackoverflow.com/questions/40190377/what-is-average-loss-field-in-caffe-solver-for

# iter_size
在显存不够用时很管用.
它产生的效果是`forward` `iter_size`次后才`backpropogate`一次, 相当于将`batch_size `增大了`iter_size`倍.

简单来说, real batch_size = batch_size * iter_size.
每执行一次`solver.step(1)`, 会执行batch_size * iter_size次forward与1次backward.

* https://www.zhihu.com/question/37270367

# max_iter
最大iteration次数. 但如果是通过`solver.step(n)`来forward-backward, 这个配置是无效的.
例如以下代码, total iterations = 100 * 10 = 1000

for _ in xrange(100):

solver.step(10)

<hr>

* http://caffe.berkeleyvision.org/tutorial/solver.html