Install hadoop on windows(non-virtual machine, such cygwin)
Download
Before starting make sure you have this two softwares
Extract downloaded tar file
Configuration
Step 1 – Windows path configuration
set HADOOP_HOME path in enviornment variable for windows
Right click on my computer > properties > advanced system settings > advance tab > environment variables > click on new
Set hadoop bin directory path
Find path variable in system variable > click on edit > at the end insert ‘; (semicolon)’ and paste path upto hadoop bin directory in my case it’s a
F:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/bin
Step 2 – Hadoop configuration
Edit hadoop-2.7.1/etc/hadoop/core-site.xml, paste the following lines and save it.
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
Edit hadoop-2.7.1/etc/hadoop/mapred-site.xml, paste the following lines and save it.
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Edit hadoop-2.7.1/etc/hadoop/hdfs-site.xml, paste the following lines and save it, please create data folder somewhere and in my case i have created it in myHADOOP_HOME directory
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/f:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/f:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/data/datanode</value>
</property>
</configuration>
OR
Edit hadoop-2.7.1/etc/hadoop/yarn-site.xml, paste the following lines and save it.
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
Edit hadoop-2.7.1/etc/hadoop/hadoop-env.cmd, comment existing%JAVA_HOME% using @rem at start, give proper path and save it. (my jdk is in program files to avoid spaces i gave PROGRA~1)
Demo
Step 3 – Start everything
Very Important step!!!!
3.1) Before starting everything you need to add some [dot].dll and [dot].exe files of windows please download bin folder from my github repository –sardetushar_gitrepo_download
or download from https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin
3.2) copy all the files dowloaded into the %HADOOP_HOME%\bin folder (The download file contains .dll and .exe file (winutils.exe for hadoop 2.7.1) which adapte for windows os).
3.3) Open cmd and type ‘hdfs namenode -format’ – after execution you will see below logs
3.4) Open cmd and point to sbin directory and type ‘start-all.cmd’
C:\UserDefined\BigData\hadoop-2.7.1\sbin>start-all.cmd
It will start following process
Namenode
Datanode
YARN resourcemanager
YARN nodemanager
JPS – to see services are running
open cmd and type – jps (for jps make sure your java path is set properly)
GUI
Step 4 – namenode GUI, resourcemanager GUI
Resourcemanager GUI address – http://localhost:8088
Namenode GUI address – http://localhost:50070
In next tutorial we will see how to run mapreduce programs in windows using eclipse and this hadoop setup
Install hadoop on windows(non-virtual machine, such cygwin)的更多相关文章
- [New Portal]Windows Azure Virtual Machine (11) 在本地使用Hyper-V制作虚拟机模板,并上传至Azure (1)
<Windows Azure Platform 系列文章目录> 本章介绍的内容是将本地Hyper-V的VHD,上传到Azure数据中心,作为自定义的虚拟机模板. 注意:因为在制作VHD的最 ...
- [New Portal]Windows Azure Virtual Machine (12) 在本地使用Hyper-V制作虚拟机模板,并上传至Azure (2)
<Windows Azure Platform 系列文章目录> 本章介绍的内容是将本地Hyper-V的VHD,上传到Azure数据中心,作为自定义的虚拟机模板. 注意:因为在制作VHD的最 ...
- [New Portal]Windows Azure Virtual Machine (13) 在本地使用Hyper-V制作虚拟机模板,并上传至Azure (3)
<Windows Azure Platform 系列文章目录> 本章介绍的内容是将本地Hyper-V的VHD,上传到Azure数据中心,作为自定义的虚拟机模板. 注意:因为在制作VHD的最 ...
- [New Portal]Windows Azure Virtual Machine (14) 在本地制作数据文件VHD并上传至Azure(1)
<Windows Azure Platform 系列文章目录> 之前的内容里,我介绍了如何将本地的Server 2012中文版 VHD上传至Windows Azure,并创建基于该Serv ...
- [New Portal]Windows Azure Virtual Machine (16) 使用Azure PowerShell创建Azure Virtual Machine
<Windows Azure Platform 系列文章目录> 注:本章内容和之前的[New Portal]Windows Azure Virtual Machine (12) 在本地制作 ...
- [New Portal]Windows Azure Virtual Machine (18) Azure Virtual Machine内部IP和外部IP
<Windows Azure Platform 系列文章目录> 在开始本章内容之前,请读者熟悉以下2篇博文: [New Portal]Windows Azure Virtual ...
- [New Portal]Windows Azure Virtual Machine (19) 关闭Azure Virtual Machine与VIP Address,Internal IP Address的关系(1)
<Windows Azure Platform 系列文章目录> 默认情况下,通过Azure Management Portal创建的Public IP和Private IP都是随机分配的. ...
- [New Portal]Windows Azure Virtual Machine (21) 将本地Hyper-V的VM上传至Windows Azure Virtual Machine
<Windows Azure Platform 系列文章目录> 本章介绍的内容是将本地Hyper-V的VHD,上传到Azure数据中心,并且保留OS中的内容. 注意:笔者没有执行Syspr ...
- [New Portal]Windows Azure Virtual Machine (22) 使用Azure PowerShell,设置Virtual Machine Endpoint
<Windows Azure Platform 系列文章目录> 我们可以通过Windows Azure Management Portal,打开Virtual Machine的Endpoi ...
- [New Portal]Windows Azure Virtual Machine (23) 使用Storage Space,提高Virtual Machine磁盘的IOPS
<Windows Azure Platform 系列文章目录> 注意:如果使用Azure Virtual Machine,虚拟机所在的存储账号建议使用Local Redundant.不建议 ...
随机推荐
- vue 登录验证引擎
1.router配置: 路由元信息 const router = new VueRouter({ routes: [ { path: '/foo', component: Foo, children: ...
- CentOS7安装OpenStack(Rocky版)-05.安装一个nova计算节点实例
上一篇文章分享了控制节点的nova计算服务的安装方法,在实际生产环境中,计算节点通常会安装一些单独的节点提供服务,本文分享单独的nova计算节点的安装方法 ---------------- 完美的分 ...
- Linux Framebuffer save as picture
/********************************************************************************* * Linux Framebuff ...
- 【opencv基础】测量运行时间的函数getTickCount/getCPUTickCount/getTickFrequency
函数的计算结果类型是double,单位是秒. 要使用更精确的计时,就需要使用getCPUTickCount(),不过现代计算机CPU的频率会随着负载而变化所以没大有必要使用该函数,可以参看函数的介绍[ ...
- c++ 字符数组-print and 写入文件
1.print ][] = { }; method1: 为了打印出unsigned char数据所对应的数值,可以将其强制转换成int类型,并做打印输出. std::cout << ][] ...
- codeforces315Div1 B Symmetric and Transitive
http://codeforces.com/contest/568/problem/B 题意就是给一个有n个元素的集合,现在需要求有多少个A的二元关系p,使得p是对称的,是传递的,但不是自反的. 首先 ...
- matplotlib的颜色及线条控制
refer to: https://www.cnblogs.com/darkknightzh/p/6117528.html
- M端计算rem方法
(function(){var a=document.documentElement.clientWidth||document.body.clientWidth;if(a>460){a=460 ...
- gRPC-Web is going GA
原文: https://www.cncf.io/blog/2018/10/24/grpc-web-is-going-ga On behalf of the Cloud Native Computing ...
- 用zcat查看压缩日志中百度抓取的量
比如查看124.251.44.85这一台服务器的07-13,07-14,07-15的日志中百度抓取http://www.baidu.com/search/spider.html 的量 wc命令参考博客 ...