利用caffe生成 lmdb 格式的文件，并对网络进行FineTuning

利用caffe生成 lmdb 格式的文件，并对网络进行FineTuning

数据的组织格式为：

首先，所需要的脚本指令路径为：

/home/wangxiao/Downloads/caffe-master/examples/imagenet/

其中，生成lmdb的文件为： create_imagenet.sh

接下来的主要任务就是修改自己的data的存放路径了。

 #!/usr/bin/env sh
 # Create the imagenet lmdb inputs
 # N.B. set the path to the imagenet train + val data dirs

 EXAMPLE=../wangxiao
 DATA=../fine_tuning_data
 TOOLS=../build/tools

 TRAIN_DATA_ROOT=../fine_tuning_data/training/data/
 VAL_DATA_ROOT=../fine_tuning_data/validation/data/

 #TRAIN_DATA_ROOT=/media/yukai/247317a3-e6b5-45d4-81d1-/---------------/Attribute reconginition/final_PETA_dataset/whole_benchmark/用于微调网络的数据/training/data/
 #VAL_DATA_ROOT=/media/yukai/247317a3-e6b5-45d4-81d1-/---------------/Attribute reconginition/final_PETA_dataset/whole_benchmark/用于微调网络的数据/validation/data/

 # Set RESIZE=true to resize the images to 256x256. Leave as false if images have
 # already been resized using another tool.

 # RESIZE=false  default parameter and wangxiao modify it in 2015.10. :

 RESIZE=true
 if $RESIZE; then
   RESIZE_HEIGHT=
   RESIZE_WIDTH=
 else
   RESIZE_HEIGHT=
   RESIZE_WIDTH=
 fi

 if [ ! -d "$TRAIN_DATA_ROOT" ]; then
   echo "Error: TRAIN_DATA_ROOT is not a path to a directory: $TRAIN_DATA_ROOT"
   echo "Set the TRAIN_DATA_ROOT variable in create_imagenet.sh to the path" \
        "where the ImageNet training data is stored."
   exit
 fi

 if [ ! -d "$VAL_DATA_ROOT" ]; then
   echo "Error: VAL_DATA_ROOT is not a path to a directory: $VAL_DATA_ROOT"
   echo "Set the VAL_DATA_ROOT variable in create_imagenet.sh to the path" \
        "where the ImageNet validation data is stored."
   exit
 fi

 echo "Creating train lmdb..."

 GLOG_logtostderr= $TOOLS/convert_imageset \
     --resize_height=$RESIZE_HEIGHT \
     --resize_width=$RESIZE_WIDTH \
     --shuffle \
     $TRAIN_DATA_ROOT \
     $DATA/training/final_train_data.txt \
     $EXAMPLE/PETA_train_lmdb

 #echo "Creating val lmdb..."

 #GLOG_logtostderr= $TOOLS/convert_imageset \
 #    --resize_height=$RESIZE_HEIGHT \
 #    --resize_width=$RESIZE_WIDTH \
 #    --shuffle \
 #    $VAL_DATA_ROOT \
 #    $DATA/validation/final_test_data.txt \
 #    $EXAMPLE/PETA_val_lmdb

 echo "Done."

都修改完成后，在终端执行：create_imagenet.sh，然后会有如此的提示，表示正在生成lmdb文件：

然后完成后，提示： Done. 然后可以看到已经生成了所需要的文件：

然后利用 make_imagenet_mean.sh 生成所需要的 mean file :

caffe-master$:  sh ./make_imagenet_mean.sh

 #!/usr/bin/env sh
 # Compute the mean image from the imagenet training lmdb
 # N.B. this is available in data/ilsvrc12

 EXAMPLE=../wangxiao
 DATA=./data
 TOOLS=../build/tools

 #echo $TOOLS/compute_image_mean $EXAMPLE/HAT_train_lmdb \
 #    $DATA/HAT_mean.binaryproto
 $TOOLS/compute_image_mean $EXAMPLE/HAT_train_lmdb \
     $DATA/HAT_mean.binaryproto
 echo "Done."

然后就生成了 HAT_mean.binaryproto

其中，caffe路径下：/home/wangxiao/Downloads/caffe-master/examples/imagenet/readme.md 对这个过程有一个详细的解释。

然后就是查看 solver.prototxt:

 net: "models/bvlc_reference_caffenet/train_val.prototxt"
 test_iter:
 test_interval:
 base_lr: 0.01
 lr_policy: "step"
 gamma: 0.1
 stepsize:
 display:
 max_iter:
 momentum: 0.9
 weight_decay: 0.0005
 snapshot:
 snapshot_prefix: "models/bvlc_reference_caffenet/caffenet_train"
 solver_mode: GPU

打开 models/bvlc_reference_caffenet/train_val.prototxt

需要修改的也就到第55行：

 name: "CaffeNet"
 layer {
   name: "data"
   type: "Data"
   top: "data"
   top: "label"
   include {
     phase: TRAIN
   }
   transform_param {
     mirror: true
     crop_size:
     mean_file: "wangxiao/HAT_data/HAT_mean.binaryproto"
   }
 # mean pixel / channel-wise mean instead of mean image
 #  transform_param {
 #    crop_size:
 #    mean_value:
 #    mean_value:
 #    mean_value:
 #    mirror: true
 #  }
   data_param {
     source: "wangxiao/HAT_data/HAT_train_lmdb"
     batch_size:
     backend: LMDB
   }
 }
 layer {
   name: "data"
   type: "Data"
   top: "data"
   top: "label"
   include {
     phase: TEST
   }
   transform_param {
     mirror: false
     crop_size:
     mean_file: "wangxiao/HAT_data/HAT_mean.binaryproto"
   }
 # mean pixel / channel-wise mean instead of mean image
 #  transform_param {
 #    crop_size:
 #    mean_value:
 #    mean_value:
 #    mean_value:
 #    mirror: true
 #  }
   data_param {
     source: "wangxiao/HAT_data/HAT_val_lmdb"
     batch_size:
     backend: LMDB
   }
 }

然后执行：

终端会有显示：

 I0103 ::21.027832   net.cpp:] Network initialization done.
 I0103 ::21.027839   net.cpp:] Memory required for data:
 I0103 ::21.027928   solver.cpp:] Solver scaffolding done.
 I0103 ::21.028312   caffe.cpp:] Starting Optimization
 I0103 ::21.028326   solver.cpp:] Solving CaffeNet
 I0103 ::21.028333   solver.cpp:] Learning Rate Policy: step
 I0103 ::22.012593   solver.cpp:] Iteration , loss = 7.52783
 I0103 ::22.012660   solver.cpp:]     Train net output #: loss = 7.52783 (*  = 7.52783 loss)
 I0103 ::22.012687   solver.cpp:] Iteration , lr = 0.01
 I0103 ::41.812361   solver.cpp:] Iteration , loss = 3.9723
 I0103 ::41.812413   solver.cpp:]     Train net output #: loss = 3.9723 (*  = 3.9723 loss)
 I0103 ::41.812428   solver.cpp:] Iteration , lr = 0.01
 I0103 ::01.553021   solver.cpp:] Iteration , loss = 2.9715
 I0103 ::01.553104   solver.cpp:]     Train net output #: loss = 2.9715 (*  = 2.9715 loss)
 I0103 ::01.553119   solver.cpp:] Iteration , lr = 0.01
 I0103 ::21.574745   solver.cpp:] Iteration , loss = 2.91547
 I0103 ::21.574798   solver.cpp:]     Train net output #: loss = 2.91547 (*  = 2.91547 loss)
 I0103 ::21.574811   solver.cpp:] Iteration , lr = 0.01
 I0103 ::41.247493   solver.cpp:] Iteration , loss = 2.96451
 I0103 ::41.247627   solver.cpp:]     Train net output #: loss = 2.96451 (*  = 2.96451 loss)
 I0103 ::41.247642   solver.cpp:] Iteration , lr = 0.01
 I0103 ::00.941267   solver.cpp:] Iteration , loss = 2.85887
 I0103 ::00.941318   solver.cpp:]     Train net output #: loss = 2.85887 (*  = 2.85887 loss)
 I0103 ::00.941332   solver.cpp:] Iteration , lr = 0.01
 I0103 ::20.628329   solver.cpp:] Iteration , loss = 2.91318
 I0103 ::20.628463   solver.cpp:]     Train net output #: loss = 2.91318 (*  = 2.91318 loss)
 I0103 ::20.628476   solver.cpp:] Iteration , lr = 0.01
  I0103 ::40.621937   solver.cpp:] Iteration , loss = 3.06499
 I0103 ::40.621989   solver.cpp:]     Train net output #: loss = 3.06499 (*  = 3.06499 loss)
 I0103 ::40.622004   solver.cpp:] Iteration , lr = 0.01
 I0103 ::00.557921   solver.cpp:] Iteration , loss = 2.9818
 I0103 ::00.558048   solver.cpp:]     Train net output #: loss = 2.9818 (*  = 2.9818 loss)
 I0103 ::00.558063   solver.cpp:] Iteration , lr = 0.01

因为设置的迭代次数为： 450000次，所以，接下来就是睡觉了。。。O(∩_∩)O~ 感谢木得兄刚刚的帮助。

------ 未完待续------

另外就是，当loss 后期变化不大的时候，可以试着调整学习率， 在Solver.prototext中：

 train_net: "models/bvlc_reference_caffenet/train_val.prototxt"
 # test_iter:
 # test_interval:
 base_lr: 0.0001
 lr_policy: "step"
 gamma: 0.1
 stepsize:
 display:
 max_iter:
 momentum: 0.9
 weight_decay: 0.0005
 snapshot:
 snapshot_prefix: "models/bvlc_reference_caffenet/caffenet_train"
 solver_mode: GPU

base_lr: 0.0001  每次可以改为0.1×base_lr， 这里的 0.0001 是我两次调整之后的数值。
然后运行 resume_training.sh 

 #!/usr/bin/env sh

 ./build/tools/caffe train \
     --solver=models/bvlc_reference_caffenet/solver.prototxt \
     --snapshot=models/bvlc_reference_caffenet/caffenet_train_iter_88251.solverstate

将snapshot改为之前中断时的结果即可，即： caffenet_train_iter_88251.solverstate

继续看loss是否降低。。。

--------------------------------- 未完待续 ---------------------------------