1.单机部署hadoop测试环境

之前看了很多理论上的知识，感觉云里雾里的，所以赶紧着手搭建个单机版的hadoop跑一跑，开启自学大数据技术的第一步~~

1.在开源的世界里，我就是个土豪，要啥有啥，所以首先你得有个jdk，有钱所以用最新的java8，hadoop使用的是hadoop2.6.0。

2.配置好java后，可以在/etc/profile里配置好环境变量，方便之后使用，紧接着解压hadoop2.6.0.tar.gz。

3.接下来配置hadoop,所有的配置文件都在hadoop文件夹下的etc/hadoop中：

　（1）hadoop-env.sh ：这个脚本只需要修改最上面的JavaHome即可，修改为自己的java路径

　（2）core-site.xml，mapred-site.xml，hdfs-site.xml这几个配置完事再补上吧~~~，网上挺多的，不过要找自己对应的版本，不然会出很多奇怪的问题。

4.配置好之后就要启动了

　　（1）启动之前首先要把namenode格式化一下，这是第一次启动hadoop需要做的动作，他会把hdfs中所有的东西全部清空掉的，所以要慎用~~

[qiang@localhost hadoop-2.6.]$  bin/hadoop namenode -format

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.6.0
.....
.....
.....
15/08/11 08:25:46 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/

　　格式化会出现一大堆信息，如果没有报错，那么说明之前的配置应该是可以滴~~~

　　（2）启动的时候,可以直接使用sbin/start-all.sh，但是这种方式太low，如果集群启动出现错误，那么不会知道是那一部分的问题，不便于问题的排查，所以我们来一个一个启动它

启动namenode:

[qiang@localhost hadoop-2.6.]$ sbin/hadoop-daemon.sh start namenode

starting namenode, logging to /home/qiang/hadoop-2.6./logs/hadoop-qiang-namenode-localhost.localdomain.out

启动datanode:

[qiang@localhost hadoop-2.6.]$ sbin/hadoop-daemon.sh start datanode

starting datanode, logging to /home/qiang/hadoop-2.6./logs/hadoop-qiang-datanode-localhost.localdomain.out

可以用jps命令查看是否启动

[qiang@localhost ~]$ jps
 Jps
 NameNode
 DataNode

当然也可以使用开放的端口在web浏览器上查看：（hdfs开放的端口为50070）

开了当然要用用他了，看看是不是唬人的，所以我们向hdfs中上传点东西试试：

[qiang@localhost hadoop-2.6.]$ bin/hadoop fs -mkdir /home
[qiang@localhost hadoop-2.6.]$ bin/hadoop fs -mkdir /home/qiangweikang
[qiang@localhost hadoop-2.6.]$ bin/hadoop fs -put README.txt /home/qiangweikang

点击uitilites中的system source会看到我们之前传进去的东东：

好开森~~

完事我们继续启动yarn

[qiang@localhost hadoop-2.6.]$ sbin/start-yarn.sh

在web上就可以看到传说中的那只大象....  ，而且我们可以看到有一个活动的节点（yarn的ResourceManager的默认端口号是8088）

接下来我们再跑一个demo，看看hadoop是怎么去运行的（在share下有自带的demo可供测试）这个pi的计算很有意思，是对一个圆做投掷飞镖的动作，第一个参数是map操作的次数

第二个参数是每次投掷多少个飞镖，好高大上啊，pi还可以这样算~~~，难道这就是传说中的概率统计？

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6..jar pi  

Number of Maps  =
Samples per Map =
Wrote input for Map #
Wrote input for Map #
Starting Job
// :: INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1439308289430_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1439308289430_0001
// :: INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1439308289430_0001/
// :: INFO mapreduce.Job: Running job: job_1439308289430_0001
// :: INFO mapreduce.Job: Job job_1439308289430_0001 running in uber mode : false
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job: Job job_1439308289430_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
    File System Counters
        FILE: Number of bytes read=
        FILE: Number of bytes written=
        FILE: Number of read operations=
        FILE: Number of large read operations=
        FILE: Number of write operations=
        HDFS: Number of bytes read=
        HDFS: Number of bytes written=
        HDFS: Number of read operations=
        HDFS: Number of large read operations=
        HDFS: Number of write operations=
    Job Counters
        Launched map tasks=
        Launched reduce tasks=
        Data-local map tasks=
        Total time spent by all maps in occupied slots (ms)=
        Total time spent by all reduces in occupied slots (ms)=
        Total time spent by all map tasks (ms)=
        Total time spent by all reduce tasks (ms)=
        Total vcore-seconds taken by all map tasks=
        Total vcore-seconds taken by all reduce tasks=
        Total megabyte-seconds taken by all map tasks=
        Total megabyte-seconds taken by all reduce tasks=
    Map-Reduce Framework
        Map input records=
        Map output records=
        Map output bytes=
        Map output materialized bytes=
        Input split bytes=
        Combine input records=
        Combine output records=
        Reduce input groups=
        Reduce shuffle bytes=
        Reduce input records=
        Reduce output records=
        Spilled Records=
        Shuffled Maps =
        Failed Shuffles=
        Merged Map outputs=
        GC time elapsed (ms)=
        CPU time spent (ms)=
        Physical memory (bytes) snapshot=
        Virtual memory (bytes) snapshot=
        Total committed heap usage (bytes)=
    Shuffle Errors
        BAD_ID=
        CONNECTION=
        IO_ERROR=
        WRONG_LENGTH=
        WRONG_MAP=
        WRONG_REDUCE=
    File Input Format Counters
        Bytes Read=
    File Output Format Counters
        Bytes Written=
Job Finished in 42.318 seconds
Estimated value of Pi is 3.12000000000000000000

最后记得把yarn关掉~~

[qiang@localhost hadoop-2.6.]$ sbin/stop-yarn.sh