1Nginx+fastdfs分布式文件存储



准备，将所需的软件传到服务器上，服务器的列表如下：

fastdfs-nginx-module_v1.15.tar.gz

FastDFS_v4.06.tar.gz

libevent-2.0.21-stable.tar.gz

nginx-1.5.6.tar.gz

openssl-1.0.1c.tar.gz

pcre-8.36.tar.gz

zlib-1.2.7.tar.gz

1. 首先切换到root用户，命令是：

su root

1. 删除系统自带的低版本的libevent ,
   注意：这里的tracker和storage端都做相同的操作。

查看系统上是否安装了libevent

rpm -qa | grep libevent

libevent-1.4.13-4.el6.x86_64

删除已经安装过的libevent

yum remove libevent-1.4.13-4.el6.x86_64

3 
安装libevent2.0.21,可以通过.wget的方式进行下来所需要的tar包

1>.wget https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz

2> tar -zxvf libevent-2.0.21-stable.tar.gz

3> cd libevent-2.0.21-stable

4> ./configure --prefix=/usr/local/libevent

5> make && make install

6>
为libevent创建软链接到/lib库下，64位系统对应/lib64 
（注意下面的的只选择其一）

ln -s /usr/local/libevent/lib/libevent-2.0.so.5 /usr/lib/libevent-2.0.so.5

ln -s /usr/local/libevent/lib/libevent-2.0.so.5 /usr/lib64/libevent-2.0.so.5

4
安装Fastdfs

1>
下载Fastdfs，需要FQ来。

wget http://fastdfs.googlecode.com/files/FastDFS_v4.06.tar.gz

2> tar -zxvf FastDFS_v4.06.tar.gz

3> cd FastDFS

4>
由于定义/usr/local/fastdfs为fastdfs安装目录,所以需要修改make.sh

vim make.sh

#内容更改如下

TARGET_PREFIX=/usr/local 修改为 /usr/local/fastdfs

将/etc/fdfs
全部替换为 /usr/local/fastdfs/conf

vim替换命令,进入末行模式(ESC 
再按shift :)

%s/\/etc\/fdfs/\/usr\/local\/fastdfs\/conf/g
 (注意，这里”\/”表示转义’/’ ,g表示全部替换)

5>
安装

./make.sh C_INCLUDE_PATH=/usr/local/libevent/include LIBRARY_PATH=/usr/local/libevent/lib

./make.sh install

===================以上是Tracker和Storage都需要安装的部分======================

5
配置Tracker（它是单独一台电脑上的，并且在使用storage服务器的时候，此端要先启动）

注意Tracker服务器IP地址不能是127.0.0.1，Tacker不能和Storage在一台电脑上。

1>
创建tracker目录保存运行日志

mkdir -m 777 -p /home/fastdfs/tracker

2>
修改tracker.conf配置

vim /usr/local/fastdfs/conf/tracker.conf

3>
修改内容如下所示

# the tracker server port

port=22122

# the base path to store data and log files

base_path=/home/yuqing/fastdfs -> base_path=/home/fastdfs/tracker #日志目录

#开启自定义server ID取代ip形式,方便内部网络服务器更换ip#**此方式要重点理解，4.0以后新特性

use_storage_id = true #使用server ID作为storage
server标识

storage_ids_filename = storage_ids.conf #<id> <group_name> <ip_or_hostname>

id_type_in_filename = id #文件名反解析中包含server ID,以前是ip

复制代码

4>
移动storage_ids.conf文件

cp  <FastDFS源码目录>/conf/storage_ids.conf /usr/local/fastdfs/conf/

5>
编辑storage服务器ID与IP地址的对应关系

vim /usr/local/fastdfs/conf/storage_ids.conf

修改内容如下所示

#<id> <group_name> <ip_or_hostname>

100001          group1          
192.168.1.12

#100002          group2          
192.168.1.13

#100003          group2          
192.168.1.14

6>
编辑tracker服务器启动脚本

vim /etc/init.d/fdfs_trackerd

#启动脚本内容如下

#!/bin/bash

#

# fdfs_trackerd Starts fdfs_trackerd

#

#

# chkconfig: 2345 99 01

# description: FastDFS tracker server

### BEGIN INIT INFO

# Provides: $fdfs_trackerd

### END INIT INFO

# Source function library.

. /etc/init.d/functions

FastDfs='/usr/local/fastdfs'

CONF="$FastDfs/conf/tracker.conf"

if [ ! -f $CONF ]; then

echo "file $CONF does not exist!"

exit 2

fi

PRG="$FastDfs/bin/fdfs_trackerd"

if [ ! -f $PRG ]; then

echo "file $PRG does not exist!"

exit 2

fi

Stop="$FastDfs/bin/stop.sh"

if [ ! -f $Stop ]; then

echo "file $Stop does not exist!"

exit 2

fi

Restart="$FastDfs/bin/restart.sh"

if [ ! -f $Restart ]; then

echo "file $Restart does not exist!"

exit 2

fi

RETVAL=0

start() {

echo -n $"Starting FastDFS tracker server: "

$PRG $CONF &

RETVAL=$?

echo

return $RETVAL

}

stop() {

echo -n $"Stop FastDFS tracker server: "

$Stop $PRG $CONF

RETVAL=$?

return $RETVAL

}

rhstatus() {

status fdfs_trackerd

}

restart() {

$Restart $PRG $CONF &

}

case "$1" in

start)

start

;;

stop)

stop

;;

status)

rhstatus

;;

restart|reload)

restart

;;

condrestart)

restart

;;

*)

echo $"Usage: $0 {start|stop|status|restart|condrestart}"

exit 1

esac

exit $?

7>
给启动脚本增加权限

chmod 777 /etc/init.d/fdfs_trackerd

8>
启动tracker

service fdfs_trackerd restart

9>
启动成功，加入开机启动

vim /etc/rc.d/rc.local

#加入内容如下

service fdfs_trackerd start

10>
防火墙开启tracker端口22122

vim /etc/sysconfig/iptables

#加入内容如下

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22122 -j ACCEPT

11>
重启防火墙

service iptables restart

注意的时候，最后运行Tracker的时候，要关闭防火墙，也就是说要执行service
iptables stop

===============================================================================

6.
配置storage服务器，注意的是storage和Nginx是在另外一台电脑上的。

1>
创建Storage目录保存运行日志及其data数据

mkdir-m 777 -p /home/fastdfs/storage

2>
修改storage.conf配置

vim/usr/local/fastdfs/conf/storage.conf

#修改内容如下所示

#the name of the group this storage server belongs to

group_name=group1#设置组名

#the name of the group this storage server belongs to

#the storage server port #the storage server port

port=23000

#the base path to store data and log files #日志目录

base_path=/home/yuqing/fastdfs-> /home/fastdfs/storage

#store_path#, based 0, if store_path0 not exists, it's value is base_path #data数据存储目录

#the paths must be exist

store_path0=/home/fastdfs/storage

#tracker_server can ocur more than once, and tracker_server format is

#  "host:port", host can be hostnameor ip address

tracker_server=192.168.209.121:22122->192.168.1.11:22122 
这里192.168.1.11:22122表示的是tracker所在电脑的ip地址

3>
编辑启动脚本

vim/etc/init.d/fdfs_storaged

#启动脚本内容如下

#!/bin/bash

#

#fdfs_storaged Starts fdfs_storaged

#

#

#chkconfig: 2345 99 01

#description: FastDFS storage server

###BEGIN INIT INFO

#Provides: $fdfs_storaged

###END INIT INFO

#Source function library.

./etc/init.d/functions

FastDfs='/usr/local/fastdfs'

CONF="$FastDfs/conf/storage.conf"

if[ ! -f $CONF ]; then

echo "file $CONF does not exist!"

exit 2

fi

PRG="$FastDfs/bin/fdfs_storaged"

if[ ! -f $PRG ]; then

echo "file $PRG does not exist!"

exit 2

fi

Stop="$FastDfs/bin/stop.sh"

if[ ! -f $Stop ]; then

echo "file $Stop does not exist!"

exit 2

fi

Restart="$FastDfs/bin/restart.sh"

if[ ! -f $Restart ]; then

echo "file $Restart does notexist!"

exit 2

fi

RETVAL=0

start(){

echo -n $"Starting FastDFS storageserver: "

$PRG $CONF &

RETVAL=$?

echo

return $RETVAL

}

stop(){

echo -n $"Stop FastDFS storage server:"

$Stop $PRG $CONF

RETVAL=$?

return $RETVAL

}

rhstatus(){

status fdfs_storaged

}

restart(){

$Restart$PRG $CONF &

}

case"$1" in

start)

start

;;

stop)

stop

;;

status)

rhstatus

;;

restart|reload)

restart

;;

condrestart)

restart

;;

*)

echo $"Usage: $0{start|stop|status|restart|condrestart}"

exit 1

esac

exit$?

4>
给启动脚本增加权限

chmod777 /etc/init.d/fdfs_storaged

5>
启动storage 
（注意的是，启动storage的时候要先启动storage）

servicefdfs_storaged restart

#正常情况，接下来会出现很多mkdirdata path，这是系统在创建数据目录

6>
启动成功，加入开机启动

vim/etc/rc.d/rc.local

#加入内容如下

servicefdfs_storaged start

7.
安装nginx（仅正在有Storage的电脑上安装）

1>
创建nginx日志目录

mkdir-m 777 -p /home/www/logs

2>
安装nginx必需的库：zlib-developenssl-devel pcre

yum -y install zlib-devel openssl-devel

3>
手动安装pcre

1> wgetftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.36.tar.gz

2> tar -zxvf pcre-8.36.tar.gz

3> cd pcre-8.36

4> ./configure

5> make && make install

6> ln -s /usr/local/lib/libpcre.so.1 /lib

4>
安装nginx

1> wget http://nginx.org/download/nginx-1.5.6.tar.gz

2> tar -zxvf nginx-1.5.6.tar.gz

3> cd nginx-1.5.6

4> ./configure --prefix=/usr/local/nginx --with-http_stub_status_module--with-http_ssl_module  （注意：这里的--with-http_ssl_module可以不加）

5> make && make install

5>
检查nginx配置是否正确

/usr/local/nginx/sbin/nginx -t

6>
查看nginx编译选项

/usr/local/nginx/sbin/nginx -V

7>
编辑启动脚本

vim /etc/init.d/nginxd

#启动脚本内容如下

#!/bin/sh

#

#nginx - this script starts and stops the nginx daemin

#

#description:  Nginx is an HTTP(S) server,HTTP(S) reverse \

#               proxy and IMAP/POP3 proxy server

#processname: nginx

#config:     /usr/local/nginx/conf/nginx.conf

#pidfile:     /usr/local/nginx/nginx.pid

#Source function library.

./etc/rc.d/init.d/functions

#Source networking configuration.

./etc/sysconfig/network

#Check that networking is up.

["$NETWORKING" = "no" ] && exit 0

#必填

nginx="/usr/local/nginx/sbin/nginx"

prog=$(basename$nginx)

#必填

NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"

[-f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/lock/subsys/nginx

start(){

[ -x $nginx ] || exit 5

[ -f $NGINX_CONF_FILE ] || exit 6

echo -n $"Starting $prog: "

daemon $nginx -c $NGINX_CONF_FILE

retval=$?

echo

#service php-fpm start

[ $retval -eq 0 ] && touch$lockfile

return $retval

}

stop(){

echo -n $"Stopping $prog: "

$nginx -s stop

echo_success

retval=$?

echo

#service php-fpm stop

[ $retval -eq 0 ] && rm -f$lockfile

return $retval

}

restart(){

stop

start

}

reload(){

configtest || return $?

echo -n $"Reloading $prog: "

$nginx -s reload

RETVAL=$?

echo

}

force_reload(){

restart

}

configtest(){

$nginx -t -c $NGINX_CONF_FILE

}

version(){

$nginx -V

}

rh_status(){

status $prog

}

rh_status_q(){

rh_status >/dev/null 2>&1

}

case"$1" in

start)

rh_status_q&& exit 0

$1

;;

stop)

rh_status_q|| exit 0

$1

;;

restart|configtest|version)

$1

;;

reload)

rh_status_q|| exit 7

$1

;;

force-reload)

force_reload

;;

status)

rh_status

;;

condrestart|try-restart)

rh_status_q|| exit 0

;;

*)

echo$"Usage: $0{start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest|version}"

exit2

esac

8>
给启动脚本增加权限

chmod777 /etc/init.d/nginxd

9>
启动nginx

servicenginxd restart

如果启动失败提示找不到库，ldconfig更新库路径

10>
启动成功，加入开机启动

vim/etc/rc.d/rc.local

#加入内容如下

servicenginxd start

8.
安装nginx-module模块（仅在有Storage的电脑上）

1>
下载fastdfs-nginx-module，google被禁了，需要FQ

wgethttp://fastdfs.googlecode.com/files/fastdfs-nginx-module_v1.15.tar.gz

2> tar -zxvffastdfs-nginx-module_v1.15.tar.gz;

3>
修改插件配置文件,主要是跟安装fastdfs目录相关

vim<fastdfs-nginx-module源码目录>/src/config

#修改内容如下

ngx_addon_name=ngx_http_fastdfs_module

HTTP_MODULES="$HTTP_MODULESngx_http_fastdfs_module"

NGX_ADDON_SRCS="$NGX_ADDON_SRCS$ngx_addon_dir/ngx_http_fastdfs_module.c"

CORE_INCS="$CORE_INCS/usr/local/fastdfs/include/fastdfs /usr/local/fastdfs/include/fastcommon/"

CORE_LIBS="$CORE_LIBS-L/usr/local/fastdfs/lib -lfastcommon -lfdfsclient"

CFLAGS="$CFLAGS-D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024'-DFDFS_MOD_CONF_FILENAME='\"/usr/local/fastdfs/conf/mod_fastdfs.conf\"'"

4>
复制mod_fastdfs.conf到/usr/local/fastdfs/conf/目录下

cp<fastdfs-nginx-module源码目录>/src/mod_fastdfs.conf /usr/local/fastdfs/conf/

5>
将/usr/local/fastdfs/lib加入系统文件/etc/ld.so.conf中（编译时使用的动态链接库)

vim/etc/ld.so.conf

#修改内容如下

/usr/local/fastdfs/lib

6>
更新库文件缓存ld.so.cache

ldconfig

7>
编译fastdfs-nginx-module模块

1> cd nginx-1.5.6/

2> ./configure
此处加上nginx之前的编译参数（使用 /usr/local/nginx/sbin/nginx -V
命令查看）--add-module=<fastdfs-nginx-module源码目录>/src

3> make && make install

8>
修改mod_fastdfs.conf配置

vim/usr/local/fastdfs/conf/mod_fastdfs.conf

#修改内容如下

#if load FastDFS parameters from tracker server

#since V1.12

#default value is false

load_fdfs_parameters_from_tracker=true

#FastDFS tracker_server can ocur more than once, and tracker_server format is

#  "host:port", host can be hostnameor ip address

#valid only when load_fdfs_parameters_from_tracker is true

tracker_server=192.168.25.11:22122

#the port of the local storage server

#the default value is 23000

storage_server_port=23000

#the group name of the local storage server

group_name=group1#当前storage机器组名

#if the url / uri including the group name

#set to false when uri like /M00/00/00/xxx

#set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx

#default value is false

url_have_group_name= true

#path(disk or mount point) count, default value is 1

#must same as storage.conf

store_path_count=1

#store_path#, based 0, if store_path0 not exists, it's value is base_path

#the paths must be exist

#must same as storage.conf

store_path0=/home/fastdfs/storage

#set the log filename, such as /usr/local/apache2/logs/mod_fastdfs.log

#empty for output to stderr (apache and nginx error_log file)

log_filename=/home/www/logs/mod_fastdfs.log

9.
修改nginx.conf配置

vim /usr/local/nginx/conf/nginx.conf

#修改内容如下（下面的可以替换原来文件中的所有内容）

worker_processes  2;

error_log  /home/www/logs/error.log 
notice;

pid        /home/www/logs/nginx.pid;

worker_rlimit_nofile5120;

events{

use epoll;

worker_connections  5120;

}

http{

include       mime.types;

default_type  application/octet-stream;

sendfile        on;

tcp_nopush     on;

keepalive_timeout  60;

tcp_nodelay on;

server {

listen       80;

server_name  localhost;

location/group1/M00 {

alias /home/fastdfs/storage/data;

ngx_fastdfs_module;

}

}

}

10.
测试（注意：如果不成功，还要修改client.config中的内容）

1.创建测试文件

echo "hello nginx fastdfs" >test.html

2.查看fastdfs自带测试命令使用方法

/usr/local/fastdfs/bin/fdfs_test -?

3.上传测试文件，通过tracker转存到storage服务器里

/usr/local/fastdfs/bin/fdfs_test/usr/local/fastdfs/conf/client.conf upload test.html

得到以下信息

tracker_query_storage_store_list_without_group:

server1. group_name=, ip_addr=172.16.63.13, port=23000

group_name=group1, ip_addr=172.16.63.13,port=23000

storage_upload_by_filename

group_name=group1,remote_filename=M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45.html

source ip address: 172.16.63.13

file timestamp=2014-12-18 20:18:51

file size=6

file crc32=1998380320

file url:http://172.16.63.12:8080/group1/M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45.html

storage_upload_slave_by_filename

group_name=group1, remote_filename=M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45_big.html

source ip address: 172.16.63.13

file timestamp=2014-12-18 20:18:51

file size=6

file crc32=1998380320

file url:http://172.16.63.12:8080/group1/M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45_big.html

4.打开浏览器，查看上传文件

http://172.16.63.13/group1/M00/00/00/oYYBAFSSwuKAO89DAAAABncc3SA04.html

看到"hello nginx fastdfs"则成功

备注：

1>
由于没有设置好防火墙，同组group到storage之间备份数据失败，关闭防火墙：service
iptables stop

2>
至少有一个storage到主机安装nginx

3> mod_fastdfs.conf此配置文件安装路径

4> tracker服务器ip不能是127.0.0.1