cygwin下安装hadoop0.20

安装时选择软件包：

editor-两个VIM、base-SED、Net-OPENSSH、OPENSSL、libs-libintl3和libintl8

环境变量：

path=%JAVA_HOME%\bin;%JAVA_HOME%\jre6\bin;D:\Program Files\MySQL\MySQL Server 5.5\bin;D:\Program Files\Python2.;D:\Program Files\MongoDB\bin;D:\Program Files\TortoiseSVN\bin;%MAVEN_HOME%\bin;D:\cygwin64\bin;D:\cygwin64\usr\bin

和

CYGWIN=ntsec tty

然后开始安装，安装完开始启动sshd服务：

进入终端执行：

tree@treePC ~
$ ssh-host-config

*** Query: Overwrite existing /etc/ssh_config file? (yes/no) yes
*** Info: Creating default /etc/ssh_config file
*** Query: Overwrite existing /etc/sshd_config file? (yes/no) yes
*** Info: Creating default /etc/sshd_config file
*** Info: Privilege separation is set to yes by default since OpenSSH 3.3.
*** Info: However, this requires a non-privileged account called 'sshd'.
*** Info: For more info on privilege separation read /usr/share/doc/openssh/README.privsep.
*** Query: Should privilege separation be used? (yes/no) no
*** Info: Updating /etc/sshd_config file

*** Query: Do you want to install sshd as a service?
*** Query: (Say "no" if it is already installed as a service) (yes/no) yes
*** Query: Enter the value of CYGWIN for the daemon: [] ntsec
*** Info: On Windows Server , Windows Vista, and above, the
*** Info: SYSTEM account cannot setuid to other users -- a capability
*** Info: sshd requires.  You need to have or to create a privileged
*** Info: account.  This script will help you do so.

*** Info: You appear to be running Windows XP 64bit, Windows  Server,
*** Info: or later.  On these systems, it's not possible to use the LocalSystem
*** Info: account for services that can change the user id without an
*** Info: explicit password (such as passwordless logins [e.g. public key
*** Info: authentication] via sshd).

*** Info: If you want to enable that functionality, it's required to create
*** Info: a new account with special privileges (unless a similar account
*** Info: already exists). This account is then used to run these special
*** Info: servers.

*** Info: Note that creating a new user requires that the current account
*** Info: have Administrator privileges itself.

*** Info: No privileged account could be found.

*** Info: This script plans to use 'cyg_server'.
*** Info: 'cyg_server' will only be used by registered services.
*** Query: Do you want to use a different name? (yes/no) no
*** Query: Create new privileged user account 'cyg_server'? (yes/no) yes
*** Info: Please enter a password for new user cyg_server.  Please be sure
*** Info: that this password matches the password rules given on your system.
*** Info: Entering no password will exit the configuration.
*** Query: Please enter the password:
*** Query: Reenter:

*** Info: User 'cyg_server' has been created with password 'lee**jan'.
*** Info: If you change the password, please remember also to change the
*** Info: password for the installed services which use (or will soon use)
*** Info: the 'cyg_server' account.

*** Info: Also keep in mind that the user 'cyg_server' needs read permissions
*** Info: on all users' relevant files for the services running as 'cyg_server'.
*** Info: In particular, for the sshd server all users' .ssh/authorized_keys
*** Info: files must have appropriate permissions to allow public key
*** Info: authentication. (Re-)running ssh-user-config for each user will set
*** Info: these permissions correctly. [Similar restrictions apply, for
*** Info: instance, for .rhosts files if the rshd server is running, etc].

*** Info: The sshd service has been installed under the 'cyg_server'
*** Info: account.  To start the service now, call `net start sshd' or
*** Info: `cygrunsrv -S sshd'.  Otherwise, it will start automatically
*** Info: after the next reboot.

*** Info: Host configuration finished. Have fun!

OK，启动SSH：

配置无密码SSH：

在终端执行

tree@treePC ~/.ssh
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tree/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tree/.ssh/id_rsa.
Your public key has been saved in /home/tree/.ssh/id_rsa.pub.
The key fingerprint is:
a0:4c:cd::e4::6f:c4::::eb:d6::c9: tree@treePC
The key's randomart image is:
+--[ RSA ]----+
|    .+=          |
|    .B +         |
|    +.O o  .     |
|   o.+ =  E      |
|   .o.  So .     |
|    o ... +      |
|   .   . .       |
|                 |
|                 |
+-----------------+

将公钥加入公钥授权文件中：

tree@treePC ~/.ssh
$ pwd
/home/tree/.ssh

tree@treePC ~/.ssh
$ ls
id_rsa  id_rsa.pub

tree@treePC ~/.ssh
$ cp id_rsa.pub authorized_keys

tree@treePC ~/.ssh
$ ls
authorized_keys  id_rsa  id_rsa.pub

弄完之后重启cygWin，ssh localhost试试能不能连接。

如果报错，重启sshd服务。然后重启cygWin试试。

创建连接，将C盘的jdk链接进来：

tree@treePC ~
$ ln -s "/cygdrive/c/Program Files/Java/jdk1.6.0_43" "/usr/local/jdk"

配置文件：

hadoop-env.sh文件加上JDK

# The java implementation to use.  Required.
export JAVA_HOME=/usr/local/jdk

core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/hadoop-0.20.2/temp</value>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.name.dir</name>
        <value>/hadoop-0.20.2/hdfs/name</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/hadoop-0.20.2/hdfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9001</value>
    </property>
    <property>
        <name>mapred.local.dir</name>
        <value>/hadoop-0.20.2/mapred</value>
    </property>
</configuration>

配置完之后，格式化namenode：

tree@treePC /hadoop-0.20./bin
$ ./hadoop namenode -format
// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = treePC/192.168.0.157
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
// :: INFO namenode.FSNamesystem: fsOwner=treepc\tree,None,root,Administrators,Users,HomeUsers,ora_dba
// :: INFO namenode.FSNamesystem: supergroup=supergroup
// :: INFO namenode.FSNamesystem: isPermissionEnabled=true
// :: INFO common.Storage: Image file of size  saved in  seconds.
// :: INFO common.Storage: Storage directory \hadoop-0.20.\hdfs\name has been successfully formatted.
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at treePC/192.168.0.157
************************************************************/

启动：

tree@treePC /hadoop-0.20./bin
$ ./start-all.sh
starting namenode, logging to /hadoop-0.20./bin/../logs/hadoop-tree-namenode-treePC.out
localhost: starting datanode, logging to /hadoop-0.20./bin/../logs/hadoop-tree-datanode-treePC.out
localhost: starting secondarynamenode, logging to /hadoop-0.20./bin/../logs/hadoop-tree-secondarynamenode-treePC.out
starting jobtracker, logging to /hadoop-0.20./bin/../logs/hadoop-tree-jobtracker-treePC.out
localhost: starting tasktracker, logging to /hadoop-0.20./bin/../logs/hadoop-tree-tasktracker-treePC.out

运行一个圆周率DEMO：

tree@treePC /hadoop-0.20./bin
$ ./hadoop jar ../hadoop-0.20.-examples.jar pi
Number of Maps  =
Samples per Map =
Wrote input for Map #
Wrote input for Map #
Starting Job
// :: INFO mapred.FileInputFormat: Total input paths to process :
// :: INFO mapred.JobClient: Running job: job_201402281919_0001
// :: INFO mapred.JobClient:  map % reduce %
// :: INFO mapred.JobClient:  map % reduce %
// :: INFO mapred.JobClient:  map % reduce %
// :: INFO mapred.JobClient: Job complete: job_201402281919_0001
// :: INFO mapred.JobClient: Counters:
// :: INFO mapred.JobClient:   Job Counters
// :: INFO mapred.JobClient:     Launched reduce tasks=
// :: INFO mapred.JobClient:     Launched map tasks=
// :: INFO mapred.JobClient:     Data-local map tasks=
// :: INFO mapred.JobClient:   FileSystemCounters
// :: INFO mapred.JobClient:     FILE_BYTES_READ=
// :: INFO mapred.JobClient:     HDFS_BYTES_READ=
// :: INFO mapred.JobClient:     FILE_BYTES_WRITTEN=
// :: INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=
// :: INFO mapred.JobClient:   Map-Reduce Framework
// :: INFO mapred.JobClient:     Reduce input groups=
// :: INFO mapred.JobClient:     Combine output records=
// :: INFO mapred.JobClient:     Map input records=
// :: INFO mapred.JobClient:     Reduce shuffle bytes=
// :: INFO mapred.JobClient:     Reduce output records=
// :: INFO mapred.JobClient:     Spilled Records=
// :: INFO mapred.JobClient:     Map output bytes=
// :: INFO mapred.JobClient:     Map input bytes=
// :: INFO mapred.JobClient:     Combine input records=
// :: INFO mapred.JobClient:     Map output records=
// :: INFO mapred.JobClient:     Reduce input records=
Job Finished in 24.927 seconds
Estimated value of Pi is 3.80000000000000000000

上浏览器看看：

mapreduce：

http://localhost:50030

HDFS:

http://localhost:50070

OK，运行成功就证明安装成功了。