https://blog.csdn.net/wsh596823919/article/details/80774805

hadoop3.1.0 window win7 基础环境搭建

前言:在windows上部署hadoop默认都是安装了java环境的哈。

1、下载hadoop3.1.0

https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common

2、下载之后解压到某个目录

3、配置hadoop_home

新建HADOOP_HOME,指向hadoop解压目录,如:D:/hadoop。path环境变量中增加:%HADOOP_HOME%\bin;。

4、配置hadoop相关文件

hadoop基本文件配置:hadoop配置文件位于:hadoop/etc/hadoop下

hadoop-env.cmd / core-site.xml / hdfs-site.xml / mapred-site.xml

hadoop-env.cmd,主要是在文件末尾添加了红色的字

@echo off
@rem Licensed to the Apache Software Foundation (ASF) under one or more
@rem contributor license agreements. See the NOTICE file distributed with
@rem this work for additional information regarding copyright ownership.
@rem The ASF licenses this file to You under the Apache License, Version 2.0
@rem (the "License"); you may not use this file except in compliance with
@rem the License. You may obtain a copy of the License at
@rem
@rem http://www.apache.org/licenses/LICENSE-2.0
@rem
@rem Unless required by applicable law or agreed to in writing, software
@rem distributed under the License is distributed on an "AS IS" BASIS,
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@rem See the License for the specific language governing permissions and
@rem limitations under the License. @rem Set Hadoop-specific environment variables here. @rem The only required environment variable is JAVA_HOME. All others are
@rem optional. When running a distributed configuration it is best to
@rem set JAVA_HOME in this file, so that it is correctly defined on
@rem remote nodes. @rem The java implementation to use. Required.
set JAVA_HOME=%JAVA_HOME% @rem The jsvc implementation to use. Jsvc is required to run secure datanodes.
@rem set JSVC_HOME=%JSVC_HOME% @rem set HADOOP_CONF_DIR= @rem Extra Java CLASSPATH elements. Automatically insert capacity-scheduler.
if exist %HADOOP_HOME%\contrib\capacity-scheduler (
if not defined HADOOP_CLASSPATH (
set HADOOP_CLASSPATH=%HADOOP_HOME%\contrib\capacity-scheduler\*.jar
) else (
set HADOOP_CLASSPATH=%HADOOP_CLASSPATH%;%HADOOP_HOME%\contrib\capacity-scheduler\*.jar
)
) @rem The maximum amount of heap to use, in MB. Default is 1000.
@rem set HADOOP_HEAPSIZE=
@rem set HADOOP_NAMENODE_INIT_HEAPSIZE="" @rem Extra Java runtime options. Empty by default.
@rem set HADOOP_OPTS=%HADOOP_OPTS% -Djava.net.preferIPv4Stack=true @rem Command specific options appended to HADOOP_OPTS when specified
if not defined HADOOP_SECURITY_LOGGER (
set HADOOP_SECURITY_LOGGER=INFO,RFAS
)
if not defined HDFS_AUDIT_LOGGER (
set HDFS_AUDIT_LOGGER=INFO,NullAppender
) set HADOOP_NAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_NAMENODE_OPTS%
set HADOOP_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS %HADOOP_DATANODE_OPTS%
set HADOOP_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_SECONDARYNAMENODE_OPTS% @rem The following applies to multiple commands (fs, dfs, fsck, distcp etc)
set HADOOP_CLIENT_OPTS=-Xmx512m %HADOOP_CLIENT_OPTS%
@rem set HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData %HADOOP_JAVA_PLATFORM_OPTS%" @rem On secure datanodes, user to run the datanode as after dropping privileges
set HADOOP_SECURE_DN_USER=%HADOOP_SECURE_DN_USER% @rem Where log files are stored. %HADOOP_HOME%/logs by default.
@rem set HADOOP_LOG_DIR=%HADOOP_LOG_DIR%\%USERNAME% @rem Where log files are stored in the secure data environment.
set HADOOP_SECURE_DN_LOG_DIR=%HADOOP_LOG_DIR%\%HADOOP_HDFS_USER% @rem
@rem Router-based HDFS Federation specific parameters
@rem Specify the JVM options to be used when starting the RBF Routers.
@rem These options will be appended to the options specified as HADOOP_OPTS
@rem and therefore may override any similar flags set in HADOOP_OPTS
@rem
@rem set HADOOP_DFSROUTER_OPTS=""
@rem @rem The directory where pid files are stored. /tmp by default.
@rem NOTE: this should be set to a directory that can only be written to by
@rem the user that will run the hadoop daemons. Otherwise there is the
@rem potential for a symlink attack.
set HADOOP_PID_DIR=%HADOOP_PID_DIR%
set HADOOP_SECURE_DN_PID_DIR=%HADOOP_PID_DIR% @rem A string representing this instance of hadoop. %USERNAME% by default.
set HADOOP_IDENT_STRING=%USERNAME% set HADOOP_PREFIX=D:\study\bigdata\hadoop\hadoop-3.1.0
set HADOOP_CONF_DIR=%HADOOP_PREFIX%\etc\hadoop
set YARN_CONF_DIR=%HADOOP_CONF_DIR%
set PATH=%PATH%;%HADOOP_PREFIX%\bin

core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

hdsf-site.xml

<!--这里是单机版所以是1-->
<!--注意这里的路径写法D前面有斜杠-->
<property>
    
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
  
<property>
<name>dfs.namenode.name.dir</name>
<value>/D:/study/bigdata/hadoop/hadoop-3.1.0/data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/D:/study/bigdata/hadoop/hadoop-3.1.0/data/datanode</value>
</property>

mapred-site.xml

<description></description>标签中的内容可以删除
<configuration>
<property>
<description>CLASSPATH for MR applications. A comma-separated list
of CLASSPATH entries. If mapreduce.application.framework is set then this
must specify the appropriate classpath for that archive, and the name of
the archive must be present in the classpath.
If mapreduce.app-submission.cross-platform is false, platform-specific
environment vairable expansion syntax would be used to construct the default
CLASSPATH entries.
For Linux:
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*.
For Windows:
%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,
%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*. If mapreduce.app-submission.cross-platform is true, platform-agnostic default
CLASSPATH for MR applications would be used:
{{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*,
{{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/*
Parameter expansion marker will be replaced by NodeManager on container
launch based on the underlying OS accordingly.
</description>
<name>mapreduce.application.classpath</name>
<value>/D:\study\bigdata\hadoop\hadoop-3.1.0/share/hadoop/mapreduce/*, /D:\study\bigdata\hadoop\hadoop-3.1.0/share/hadoop/mapreduce/lib/*</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

yarn-site.xml

<configuration>

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

        <value>org.apache.hadoop.mapred.ShuffleHandler</value>

    </property>

</configuration>

5、winutils相关,hadoop在windows上运行需要winutils支持和hadoop.dll等文件

注意

  a、hadoop.dll等文件不要与hadoop冲突。为了不出现依赖性错误可以将hadoop.dll放到c:/windows/System32下一份。

  b、这是什么版本就是找什么版本的,不然会出现各种未知问题

下面是3.1.0的下载地址

https://download.csdn.net/download/cntpro/10497884#comment

下下来之后替换到D:\study\bigdata\hadoop\hadoop-3.1.0\bin里面的内容

6、到D:\study\bigdata\hadoop\hadoop-3.1.0\etc\hadoop路径下用管理员执行hadoop-env.cmd初始化环境(路径是自己的解压路径)

也可以用cmd窗口执行

7、到D:\study\bigdata\hadoop\hadoop-3.1.0\bin路径下执行

hdfs namenode -format

注意这里只执行一次,我因为之前没配好,出错了,所以这里执行了多次,导致后面datanode启动老是报各种各样的错,我是删掉文件,重新解压配了一遍才成功的

8、到D:\study\bigdata\hadoop\hadoop-3.1.0\sbin路径下执行

start-dfs.cmd启动dfs

start-all.cmd是启动全部程序(推荐)

9、启动之后没报错就是成功了

使用jps验证是否成功,下面就是成功了

D:\study\bigdata\hadoop\hadoop-3.1.0\sbin>jps
16032 NameNode
15956 ResourceManager
16996 NodeManager
17268 DataNode
19160 Jps

10、通过http://127.0.0.1:8088/即可查看集群所有节点状态

  访问http://localhost:9870/即可查看文件管理页面:

11、后续操作自行百度

hadoop3.1.0 window win7 基础环境搭建的更多相关文章

  1. Hadoop window win10 基础环境搭建(2.8.1)

    下面运行步骤除了配置文件有部分改动,其他都是参照hadoop下载解压的share/doc/index.html. hadoop下载:http://apache.opencas.org/hadoop/c ...

  2. Hadoop window win10 基础环境搭建(2.8.1)(转)

    下面运行步骤除了配置文件有部分改动,其他都是参照hadoop下载解压的share/doc/index.html. hadoop下载:http://apache.opencas.org/hadoop/c ...

  3. Selenium win7+selenium2.0+python+JetBrains PyCharm环境搭建

    win7+selenium2.0+python+JetBrains PyCharm环境搭建 by:授客 QQ:1033553122 步骤1:下载python 担心最新版的支持不太好,这里我下载的是py ...

  4. Spark入门实战系列--2.Spark编译与部署(上)--基础环境搭建

    [注] 1.该系列文章以及使用到安装包/测试数据 可以在<倾情大奉送--Spark入门实战系列>获取: 2.Spark编译与部署将以CentOS 64位操作系统为基础,主要是考虑到实际应用 ...

  5. 【转】android 最新 NDK r8 在window下开发环境搭建 安装配置与使用 详细图文讲解,完整实际配置过程记录(原创)

    原文网址:http://www.cnblogs.com/zdz8207/archive/2012/11/27/android-ndk-install.html android 最新 NDK r8 在w ...

  6. Spark环境搭建(上)——基础环境搭建

    Spark摘说 Spark的环境搭建涉及三个部分,一是linux系统基础环境搭建,二是Hadoop集群安装,三是Spark集群安装.在这里,主要介绍Spark在Centos系统上的准备工作--linu ...

  7. android 最新 NDK r8 在window下开发环境搭建 安装配置与使用 详细图文讲解,完整实际配置过程记录(原创)

      android 最新 NDK r8 在window下开发环境搭建 安装配置与使用 详细图文讲解,完整实际配置过程记录(原创) 一直想搞NDK开发却一直给其他事情耽搁了,参考了些网上的资料今天终于把 ...

  8. Android NDK r8 Cygwin CDT 在window下开发环境搭建 安装配置与使用 具体图文解说

    版权声明:本博客全部文章均为原创.欢迎交流.欢迎转载:转载请勿篡改内容,而且注明出处,谢谢! https://blog.csdn.net/waldmer/article/details/3272500 ...

  9. EXT 基础环境搭建

    EXT 基础环境搭建使用 Sencha CMD 下载地址 https://www.sencha.com/products/extjs/cmd-download/ Sencha CMD 常用命令 API ...

随机推荐

  1. Avoid strong reference cycles

    转自:http://masteringios.com/blog/2014/03/06/avoid-strong-reference-cycles/ With the introduction of A ...

  2. 黑客编程教程(十三)多线程DOS程序

    DOS基本原理相信大家都已经很熟悉了,DOS工具大家也用的很熟悉.在群里 经常有人说什么时候去DOS什么东西. 现在我们就自己编写一个DOS工具. #include <winsock2.h> ...

  3. Python 里 and、or 的计算规则

    引子: >>> print '' or 5 or 05>>> print 5 and 44>>> print '' or 5 or 0 and 4 ...

  4. AAuto 快速开发win32小程序

    AAuto编程语言 AAuto是专用于桌面软件快速开发的新一代混合型编程语言 -  具有动态语言轻便.灵活.快速开发的特性,而且又可以同时支持静态类型开发,象静态语言那样使用.AAuto可以直接支持原 ...

  5. shell关于日期的加减

    $cat sh.sh #!/bin/bash ] ; then ydate=`date -d -1days "+%Y%m%d"` ydateother=`date -d -1day ...

  6. 基于jquery的可拖动div

    昨天给大家介绍了一款基于jquery ui漂亮的可拖动div实例,今天要给大家分享一款基于jquery的可拖动div.这款可拖动div只要引用jquery就可以,无需引用jquery ui.还实时记录 ...

  7. redis、mysql、和php原生array数组效率对比

    最近要做一个屏蔽词的功能,屏蔽词是保存在配置文件里,最开始是用php数组存储,然后查找. 后来随着屏蔽词的增多,速度明显的变慢了,于是考虑到用redis或mysql做. 于是写了一个测试,测试结果如下 ...

  8. 【Unity】用代码给按钮动态添加点击事件

    问题:多数情况下用UGUI的Button控件身上的OnClick()列表可以指明该按钮点击后触发的回调.现在想要调用自定义脚本里的方法,当这个脚本挂在Button所属的Canvas身上时,传入Canv ...

  9. sql server拼接一列字段

    有一表,名曰IPSlot,欲取IP整列字段. sql语句,利用sql server的xml auto将表数据转换成xml=> select name= STUFF( REPLACE( REPLA ...

  10. iOS边练边学--UINavigationController导航条的使用

    一.使用UINavigationController的步骤以及代码 // 程序加载完成后执行的代码 - (BOOL)application:(UIApplication *)application d ...