Hadoop Single Node Setup（hadoop本地模式和伪分布式模式安装-官方文档翻译 2.7.3）

Purpose（目标）

This document describes how to set up and configure a single-node Hadoop installation so that you can quickly perform simple operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS).

这个文档描述了如何安装和配置一个单节点的Hadoop安装，这样很快的通过用Hadoop MapReduce 和Haddop分布式文件系统(HDFS)执行一些简单的操作。

Prerequisites（先决条件）

Supported Platforms（支持平台）

GNU/Linux is supported as a development and production platform. Hadoop has been demonstrated on GNU/Linux clusters with 2000 nodes.

Windows is also a supported platform but the followings steps are for Linux only. To set up Hadoop on Windows, see wiki page.

GNU/Linux可以被支持为开发或生产平台。Hadoop被在2000节点的GNU/Linux检验测试过。

Windows是在linux平台之后唯一被支持的平台，对于hadoop在windows上的安装，看wiki。

Required Software（软件需求）

Required software for Linux include:

1、Java™ must be installed. Recommended Java versions are described at Hadoop Java Versions.

2、ssh must be installed and sshd must be running to use the Hadoop scripts that manage remote Hadoop daemons.

在linux上安装需要的软件包括：

1、java必须被安装，推荐的java版本在hadoop java 版本中被描述。

2、ssh 一定要安装并且 sshd 一定要处于运行状态，从而使Hadoop scripts可以管理远程Hadoop实例

Installing Software（安装软件）

If your cluster doesn’t have the requisite software you will need to install it.

For example on Ubuntu Linux:

$ sudo apt-get install ssh

$ sudo apt-get install rsync

如果你的集群没有安装必要的软件，请安装他们.

以ubuntu linux为例：

$ sudo apt-get install ssh

$ sudo apt-get install rsync

Download

To get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.

为了下载一个Hadoop分布式，从Apache的下载镜像一个下载地址下载一个最近的稳定的发布版本。

Prepare to Start the Hadoop Cluster（准备启动hadoop集群）

Unpack the downloaded Hadoop distribution. In the distribution, edit the file etc/hadoop/hadoop-env.sh to define some parameters as follows:

# set to the root of your Java installation

export JAVA_HOME=/usr/java/latest

Try the following command:

$ bin/hadoop

This will display the usage documentation for the hadoop script.

Now you are ready to start your Hadoop cluster in one of the three supported modes:

Local (Standalone) Mode

Pseudo-Distributed Mode

Fully-Distributed Mode

解压下载的Hadoop分布式软件，在这个分布式软件中，编辑文件 conf/hadoop-env.sh 定义一些如下参数：

# set to the root of your Java installation

export JAVA_HOME=/usr/java/latest

尝试如下命令：

$ bin/hadoop

这将显示hadoop脚本的使用帮助。

现在，你已经准备好启动Hadoop集群中的三种支持的模式之一

本地(独立)模式

伪分布式模式

完全分布模式

Standalone Operation（本地模式操作）

By default, Hadoop is configured to run in a non-distributed mode, as a single Java process. This is useful for debugging.

The following example copies the unpacked conf directory to use as input and then finds and displays every match of the given regular expression. Output is written to the given output directory.

$ mkdir input

$ cp etc/hadoop/*.xml input

$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep input output 'dfs[a-z.]+'

$ cat output/*

默认情况，hadoop的配置可以运行一个非分布式模式，作为一个单独的java进程，这通常应用在调试上。

就是把加压的conf目录的一个拷贝作为输入目录，然后查找显示所给正则表达的每一个匹配，输出到所给出的输出目录

$ mkdir input

$ cp etc/hadoop/*.xml input

$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep input output 'dfs[a-z.]+'

$ cat output/*

Pseudo-Distributed Operation（伪分布式模式）

Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

Hadoop可以运行在伪分布式模式下运行在一个单一的节点，此时每一个Hadoop守护进程作为一个单独的Java进程运行。

Configuration（配置）

Use the following:

应用如下配置：

etc/hadoop/core-site.xml:

<name>fs.defaultFS</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

etc/hadoop/hdfs-site.xml:

<name>dfs.replication</name>

</property>

</configuration>

Setup passphraseless ssh（设置ssh免密码登录）

Now check that you can ssh to the localhost without a passphrase:

$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

$ chmod 0600 ~/.ssh/authorized_keys

现在可以在本地检查ssh的免密码登录：

$ssh localhost

如果不能进行免密码登录，执行如下命令：

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

$ chmod 0600 ~/.ssh/authorized_keys

Execution（执行）

The following instructions are to run a MapReduce job locally. If you want to execute a job on YARN, see YARN on Single Node.

以下命令在本地运行一个mapreduce job，如果你想要执行job在yarn上，看yarn on single node。

Format the filesystem:

格式化文件系统：

$ bin/hdfs namenode -format

Start NameNode daemon and DataNode daemon:

启动namenode进程和datanode进程。

$ sbin/start-dfs.sh

The hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).

Hadoop守护进程日志被写入到$HADOOP_LOG_DIR目录下（默认是$HADOOP_HOME/logs目录下）。

Browse the web interface for the NameNode; by default it is available at:

Namenode的浏览器web接口，通过如下地址登录：

NameNode - http://localhost:50070/

Make the HDFS directories required to execute MapReduce jobs:

创建hdfs目录需要运行mapreduce jobs：

$ bin/hdfs dfs -mkdir /user

$ bin/hdfs dfs -mkdir /user/<username>

Copy the input files into the distributed filesystem:

拷贝输入文件到分布式文件系统上：

$ bin/hdfs dfs -put etc/hadoop input

Run some of the examples provided:

运行如下的例子：

$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep input output 'dfs[a-z.]+'

Examine the output files: Copy the output files from the distributed filesystem to the local filesystem and examine them:

检查输出文件：拷贝输出文件从分布式文件系统到本地文件系统执行他们：

$ bin/hdfs dfs -get output output

$ cat output/*

Or 或

View the output files on the distributed filesystem:

查看输出文件在分布式文件系统上：

$ bin/hdfs dfs -cat output/*

When you’re done, stop the daemons with:

当你不使用时，通过如下命令停止守护进程：

$ sbin/stop-dfs.sh

YARN on a Single Node

You can run a MapReduce job on YARN in a pseudo-distributed mode by setting a few parameters and running ResourceManager daemon and NodeManager daemon in addition.

你可以在伪分布式模式下通过设置一些很少的参数开启resourcemanager和nodemanager守护进程，并且运行一个mapreduce job在yarn上。

The following instructions assume that 1. ~ 4. steps of the above instructions are already executed.

去执行executed章节的1~4步的指令。

Configure parameters as follows:etc/hadoop/mapred-site.xml:

配置如下参数在etc/hadoop/mapred-site.xml文件：

<name>mapreduce.framework.name</name>

</property>

</configuration>

etc/hadoop/yarn-site.xml:

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

Start ResourceManager daemon and NodeManager daemon:

启动resourcemanager和nodemanager守护进程：

$ sbin/start-yarn.sh

Browse the web interface for the ResourceManager; by default it is available at:

Resourcemanager的浏览器访问接口通过如下地址：

ResourceManager - http://localhost:8088/

Run a MapReduce job.

运行一个mapreduce job。

When you’re done, stop the daemons with:

当你不使用时，如下命令停止守护进程：

$ sbin/stop-yarn.sh

Fully-Distributed Operation（完全分布式操作）

For information on setting up fully-distributed, non-trivial clusters see Cluster Setup.

为了详细说明完全分布式下的操作步骤，请看cluster setup章节。