一,准备环境

  • PC基本配置如下:
处理器:Intel(R) Core(TM) i5-3230M CPU @ 2.6GHz 2.60GHz
安装内存(RAM): 12.0GB
系统类型:64位操作系统

  • 初始化四台Ubuntu-14.04_x64虚拟机,配置如下:
内存:2GB
处理器:1
硬盘:40G
网络适配器:NAT
系统:Linux ubuntu 4.4.0-142-generic #168~14.04.1-Ubuntu SMP Sat Jan 19 11:26:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

  • 修改系统时区
 ~ sudo timedatectl set-timezone "Asia/Shanghai"
  • 为方便使用建议如下配置:

安装oh-my-zsh插件;

设置VIM行号;

安装SSH插件服务;

安装vsftpd插件服务并加以配置,方便文件上传下载;

PC上安装XSHELL客户端;

PC上安装FTP客户端。

  • 需要的软件:
jdk-7u51-linux-x64.gz  链接:http://pan.baidu.com/s/1dFFT1GP  密码:cc5t

hadoop-2.2.0-x64.tar.gz https://download.csdn.net/download/wwyymmddbb/10203840
  • 在虚拟机做如下步骤:创建目录,存储工具包/home/zhangbocheng,并利用FTP上传相关软件包。



二,安装单机环境

  • 安装Java1.7.0
~ mkdir java
~ cd java
➜ java tar -xf /home/zhangbocheng/jdk-7u51-linux-x64.gz
➜ java ln -s jdk1.7.0_51 JDK
➜ java vi ~/.bashrc
➜ java vi ~/.zshrc
export JAVA_HOME=$HOME/java/java
export PATH=$JAVA_HOME/bin:$PATH :wq
➜ java source ~/.bashrc
➜ java source ~/.zshrc
➜ java java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

  • 安装Hadoop2.2.0
➜  ~ tar -xf /home/zhangbocheng/hadoop-2.2.0-x64.tar.gz
➜ ~ mv hadoop-2.2.0 hadoop2.2.0
➜ ~ mkdir hadoop2.2.0/hdfs
➜ ~ mkdir hadoop2.2.0/hdfs/name
➜ ~ mkdir hadoop2.2.0/hdfs/data
➜ ~ mkdir hadoop2.2.0/logs
➜ ~ mkdir hadoop2.2.0/tmp
  • 配置环境
➜  ~ vi .zshrc
export JAVA_HOME=$HOME/java/jdk
export CLASSPATH=$JAVA_HOME/lib/tool.jar:$JAVA_HOME/lib/dt.jar
export PATH=$JAVA_HOME/bin:$PATH
➜ ~ source .zshrc
➜ ~ echo $CLASSPATH
/home/zhangbc/java/jdk/lib/tool.jar:/home/zhangbc/java/jdk/lib/dt.jar
➜ ~ vi .zshrc
export HADOOP_HOME=$HOME/hadoop2.2.0
export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$JAVA_HOME/bin:$PATH
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export YARN_LOG_DIR=$HADOOP_LOG_DIR
➜ ~ source .zshrc
➜ ~ hadoop version
Hadoop 2.2.0
Subversion Unknown -r Unknown
Compiled by root on 2014-09-21T22:41Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4
This command was run using /home/zhangbc/hadoop2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar

  • 修改Hadoop2.2.0配置文件
# 检查并修改以下三个文件中JAVA_HOME的值
➜ ~ vi hadoop2.2.0/etc/hadoop/hadoop-env.sh
➜ ~ vi hadoop2.2.0/etc/hadoop/yarn-env.sh
➜ ~ vi hadoop2.2.0/etc/hadoop/mapred-env.sh # 只需要去掉注释加以修改
export JAVA_HOME=${JAVA_HOME} # 错误
export JAVA_HOME=/home/zhangbc/java/jdk # 正确 # 添加集群的slave节点
➜ ~ vi hadoop2.2.0/etc/hadoop/slaves
# localhost
slave_1
slave_2
slave_3 ➜ ~ vi hadoop2.2.0/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000/</value>
<descrption>设定namenode的主机名及其端口</descrption>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/zhangbc/hadoop2.2.0/tmp/hadoop-${user.name}</value>
<descrption>存储临时文件</descrption>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
</configuration> ➜ ~ vi hadoop2.2.0/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.http-address</name>
<value>master:50070</value>
<descrption>设定NameNode地址及其端口</descrption>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>slave_1:50090</value>
<descrption>设定SecondNameNode地址及其端口</descrption>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
<descrption>设定HDFS存储文件的副本个数,默认为3</descrption>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/zhangbc/hadoop2.2.0/hdfs/name</value>
<descrption>设置NameNode用来持续存储命名空间和交换日志的本地文件系统路径</descrption>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/zhangbc/hadoop2.2.0/hdfs/data</value>
<descrption>设置DataNode在本地存储文件的目录列表</descrption>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///home/zhangbc/hadoop2.2.0/hdfs/namesecondary</value>
<descrption>设置SecondaryNameNode存储临时镜像的本地文件系统路径,
若这是一个用逗号分隔的列表,则镜像会冗余复制到所有目录</descrption>
</property>
<property>
<name>dfs.webhdfs.enable</name>
<value>true</value>
<descrption>是否允许网页浏览HDFS文件</descrption>
</property>
<property>
<name>dfs.stream-buffer-size</name>
<value>131072</value>
<descrption>默认为4KB,作为Hadoop的缓冲区,用于Hapdoop读写HDFS的文件,
还有map的输出都用到了这个缓冲区容量,131072=128KB</descrption>
</property>
</configuration> ➜ ~ vi hadoop2.2.0/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration> ➜ ~ vi hadoop2.2.0/etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
  • 修改主机名称(千万不要含有下划线_
➜  ~ sudo hostname master   # 只对当前状态生效
[sudo] password for zhangbc:
➜ ~ hostname
master
➜ ~ sudo vi /etc/hostname # 永久修改 master
:wq
  • 关闭防火墙
➜  ~ service ufw status
ufw start/running
➜ ~ sudo service ufw stop
[sudo] password for zhangbc:
ufw stop/waiting
➜ ~ service ufw status
ufw stop/waiting

三,克隆VM

通过 VMware Workstation工具,关闭当前虚拟机,对其克隆三台虚拟机作为从机使用。

克隆方法选择“创建完整克隆(F)”,如图所示:

四,搭建集群

  • 修改三台从机slavehost,并再重启使之生效。
➜  ~ sudo vi /etc/hostname
➜ ~ sudo vi /etc/hosts
➜ ~ sudo reboot
  • 对所有集群中的服务器进行检查,关闭防火墙并禁止掉。
➜  ~ sudo service ufw status
[sudo] password for zhangbc:
ufw start/running
➜ ~ sudo service ufw stop
ufw stop/waiting
➜ ~ sudo service ufw status
ufw stop/waiting
➜ ~ sudo ufw disable
Firewall stopped and disabled on system startup
  • 对所有集群中的服务器绑定hostnameIP
➜  ~ sudo vi /etc/hosts
192.168.71.128 master
192.168.71.129 slave_1
192.168.71.130 slave_2
192.168.71.131 slave_3
  • 对所有集群中的服务器创建SSH密钥,完成相关验证,注意保留原有的其他密钥,以备他用
➜  .ssh mv id_rsa id_rsa_git
➜ .ssh mv id_rsa.pub id_rsa_git.pub
➜ .ssh ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
# 生成authorized_keys
➜ .ssh cat id_rsa.pub >> authorized_keys
# 设置权限
➜ .ssh sudo chmod 600 authorized_keys
# ssh登录本机,并退出
➜ .ssh ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is b6:fa:8d:2b:2d:0d:e4:fd:4f:44:ed:37:3f:79:b6:ce.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.6 LTS (GNU/Linux 4.4.0-142-generic x86_64) * Documentation: https://help.ubuntu.com/ New release '16.04.6 LTS' available.
Run 'do-release-upgrade' to upgrade to it. Your Hardware Enablement Stack (HWE) is supported until April 2019.
Last login: Wed Nov 13 20:17:41 2019 from 192.168.71.1
➜ ~ exit
Connection to localhost closed.
➜ .ssh
  • 配置各个节点之间免密登录
# 将slave_1节点rsa通过ssh-copy-id分别复制到master,slave_2,slave_3
➜ ~ ssh-copy-id -i ~/.ssh/id_rsa.pub master
➜ ~ ssh-copy-id -i ~/.ssh/id_rsa.pub slave_2
➜ ~ ssh-copy-id -i ~/.ssh/id_rsa.pub slave_3
# 验证登录
➜ ~ ssh master
➜ ~ ssh slave_2
➜ ~ ssh slave_3
# 其他节点同步骤

五,Hadoop启动与测试

  • 格式化文件系统
➜  ~ hdfs namenode -format
19/11/13 21:57:48 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master/192.168.71.128
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.2.0
.........
19/11/13 21:57:55 INFO util.ExitUtil: Exiting with status 0 # 表示成功
.........
  • 启动HDFS
zhangbc@master:~$ start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /home/zhangbc/hadoop2.2.0/logs/hadoop-zhangbc-namenode-master.out
slave_1: starting datanode, logging to /home/zhangbc/hadoop2.2.0/logs/hadoop-zhangbc-datanode-slave_1.out
slave_3: starting datanode, logging to /home/zhangbc/hadoop2.2.0/logs/hadoop-zhangbc-datanode-slave_3.out
slave_2: starting datanode, logging to /home/zhangbc/hadoop2.2.0/logs/hadoop-zhangbc-datanode-slave_2.out zhangbc@master:~$ jps
6524 Jps
5771 NameNode
zhangbc@slave_1:~$ jps
4919 Jps
4818 DataNode
zhangbc@slave_2:~$ jps
4919 Jps
4801 DataNode
zhangbc@slave_3:~$ jps
4705 DataNode
4800 Jps

WEB验证:http://192.168.71.128:50070

  • 启动Yarn
zhangbc@master:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/zhangbc/hadoop2.2.0/logs/yarn-zhangbc-resourcemanager-master.out
slave_2: starting nodemanager, logging to /home/zhangbc/hadoop2.2.0/logs/yarn-zhangbc-nodemanager-slave_2.out
slave_1: starting nodemanager, logging to /home/zhangbc/hadoop2.2.0/logs/yarn-zhangbc-nodemanager-slave_1.out
slave_3: starting nodemanager, logging to /home/zhangbc/hadoop2.2.0/logs/yarn-zhangbc-nodemanager-slave_3.out zhangbc@master:~$ jps
5771 NameNode
6642 Jps
zhangbc@slave_1:~$ jps
5099 Jps
4818 DataNode
5011 NodeManager
zhangbc@slave_2:~$ jps
5101 Jps
5016 NodeManager
4801 DataNode
zhangbc@slave_2:~$ jps
5101 Jps
5016 NodeManager
4801 DataNode

WEB验证:http://192.168.71.128:8088

  • 管理JobHistory Server
zhangbc@master:~$ mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /home/zhangbc/hadoop2.2.0/logs/mapred-zhangbc-historyserver-master.out
zhangbc@master:~$ mr-jobhistory-daemon.sh stop historyserver

WEB验证:http://192.168.71.128:19888

  • 集群验证
# 创建目录
zhangbc@master:~$ hdfs dfs -mkdir -p /data/wordscount
zhangbc@master:~$ hdfs dfs -mkdir -p /output # 查看数据目录
zhangbc@master:~$ hdfs dfs -ls /data # 上传本地文件
zhangbc@master:~$ hdfs dfs -put hadoop2.2.0/etc/hadoop/core-site.xml /data/wordscount # 执行
zhangbc@master:~$ hadoop jar hadoop2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /data/wordscount /output/wordscount
...............................
19/11/14 13:04:45 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1573705994579_0001/
19/11/14 13:04:45 INFO mapreduce.Job: Running job: job_1573705994579_0001
19/11/14 13:04:59 INFO mapreduce.Job: Job job_1573705994579_0001 running in uber mode : false
19/11/14 13:04:59 INFO mapreduce.Job: map 0% reduce 0%
19/11/14 13:05:14 INFO mapreduce.Job: map 100% reduce 0%
19/11/14 13:05:27 INFO mapreduce.Job: map 100% reduce 100%
19/11/14 13:05:27 INFO mapreduce.Job: Job job_1573705994579_0001 completed successfully
19/11/14 13:05:27 INFO mapreduce.Job: Counters: 43
............................................ # 查看运行结果
zhangbc@master:~$ hdfs dfs -cat /output/wordscount/part-r-00000

六,安装过程中遇到的问题及其解决方案

  • 问题1:上传文件报错
zhangbc@master:~$ hdfs dfs -put hadoop2.2.0/etc/hadoop/core-site.xml /data/wordcount
19/11/14 10:13:24 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOExcept、ion): File /data/wordcount/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
put: File /data/wordcount/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
19/11/14 10:13:24 ERROR hdfs.DFSClient: Failed to close file /data/wordcount/core-site.xml._COPYING_
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /data/wordcount/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)

主要原因是重新格式化文件系统,导致master节点下的hadoop2.2.0/hdfs/name/current/VERSION中的clusterIDSlave节点下的hadoop2.2.0/hdfs/data/current/VERSION中的clusterID不一致。在浏览器输入master:50070可以发现Live Nodes为0。

解决方案是修改master节点下的clusterID使之与Slave节点下的clusterID一致,然后重启服务即可。

  • 问题2:执行JAR报错问题
Container launch failed for container_1573700741821_0001_01_000007 : java.lang.IllegalArgumentException: Does not contain a valid host:port authority: slave_1:33775

主要原因Hadoop nodemanager结点主机名不能带下划线_

解决方案:修改主机名称。

  • 问题3:绑定主机名引起的问题:sudo: unable to resolve host master

解决方案如下:

➜  ~ sudo vi /etc/hosts

  1 127.0.0.1       localhost
2 127.0.1.1 ubuntu
3 127.0.1.1 master
:wq!

通过本次实验,对集群概念有个基本的认识,在搭建过程中遇到问题不算太多,主要是对局域网组建缺乏认识深度,本集群环境可以进一步扩展,如动态增减节点,借助Zookeeper技术加以融合等在企业中是比较常见的做法。

分享一份关于Hadoop2.2.0集群环境搭建文档的更多相关文章

  1. Linux下Hadoop2.6.0集群环境的搭建

    本文旨在提供最基本的,可以用于在生产环境进行Hadoop.HDFS分布式环境的搭建,对自己是个总结和整理,也能方便新人学习使用. 基础环境 JDK的安装与配置 现在直接到Oracle官网(http:/ ...

  2. 搭建hadoop2.6.0集群环境

    一.规划 (一)硬件资源 10.171.29.191 master 10.171.94.155  slave1 10.251.0.197 slave3 (二)基本资料 用户:  jediael 目录: ...

  3. 搭建hadoop2.6.0集群环境 分类: A1_HADOOP 2015-04-20 07:21 459人阅读 评论(0) 收藏

    一.规划 (一)硬件资源 10.171.29.191 master 10.171.94.155  slave1 10.251.0.197 slave3 (二)基本资料 用户:  jediael 目录: ...

  4. Hadoop-2.7.1集群环境搭建

    摘自:http://blog.csdn.net/u014039577/article/details/49813531 由于日志数据量越来越大,数据处理的逻辑越来越复杂,同时还涉及到大量日志需要批处理 ...

  5. ElasticSearch 5.0.0 集群安装部署文档

    1.  搭建环境 3台物理机 操作系统 centos7 es1   192.168.31.141   4g内存   2核 es2   192.168.31.142   4g内存   2核 es3    ...

  6. Hadoop 2.6.0-cdh5.4.0集群环境搭建和Apache-Hive、Sqoop的安装

    搭建此环境主要用来hadoop的学习,因此我们的操作直接在root用户下,不涉及HA. Software: Hadoop 2.6.0-cdh5.4.0 Apache-hive-2.1.0-bin Sq ...

  7. Ubuntu12.04-x64编译Hadoop2.2.0和安装Hadoop2.2.0集群

      本文Blog地址:http://www.cnblogs.com/fesh/p/3766656.html   本文对Hadoop-2.2.0源码进行重新编译(64位操作系统下不重新编译会有版本问题) ...

  8. Kafka:ZK+Kafka+Spark Streaming集群环境搭建(二)安装hadoop2.9.0

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  9. Linux基于Hadoop2.8.0集群安装配置Hive2.1.1及基础操作

    前言 安装Apache Hive前提是要先安装hadoop集群,并且hive只需要在hadoop的namenode节点集群里安装即可,安装前需保证Hadoop已启(动文中用到了hadoop的hdfs命 ...

随机推荐

  1. B-微积分-sign(符号)函数

    目录 sign(符号)函数 一.sign函数概述 二.python实现sign函数 更新.更全的<机器学习>的更新网站,更有python.go.数据结构与算法.爬虫.人工智能教学等着你:h ...

  2. POJ1017&&UVA311 Packets(中文题面版)

    感谢有道翻译--- Description A工厂生产的产品是用相同高度h的方形包装,尺寸为1* 1,2 * 2,3 * 3,4 * 4,5 * 5,6 6.这些产品总是以与产品高度h相同,尺寸为66 ...

  3. java高并发_博客-网址-资料 推荐

    大概说一下自己作为入门学习java高并发的博客地址,很不错在自己的博客里记录一下:如果能有刷到我的博客的骚年,又刚好想了解java高并发,强烈推荐看看 地址:http://www.itsoku.com ...

  4. Linux重要配置文件

    目录 简介 0x01 系统重要文件 0x02 用户重要文件 0x03 重要日志文件 本教程概述 本课时学习Linux系统中重要的配置文件. 用到的工具 SshClient Ubuntu 标签 Linu ...

  5. POJ2828 Buy Tickets 树状数组

    Description Railway tickets were difficult to buy around the Lunar New Year in China, so we must get ...

  6. 详解AJAX工作原理以及实例讲解(通俗易懂)

    什么是 AJAX ? AJAX = 异步 JavaScript 和 XML. AJAX 是一种用于创建快速动态网页的技术. 通过在后台与服务器进行少量数据交换,AJAX 可以使网页实现异步更新.这意味 ...

  7. opencv::两张图片的线性融合

    理论-线性混合操作 g(x) 表示 融合图片中的像素点,f0(x) 和 f1(x) 分别表示背景和前景图片中的像素点. //参数1:输入图像Mat – src1 //参数2:输入图像src1的alph ...

  8. WebSocket学习简书

    1.什么是Websocket? WebSocket 是 HTML5 开始提供的一种在单个 TCP 连接上进行全双工通讯的协议. 2.单工,半双工和全双工通信? 在单工通信中,通信的信道是单向的,发送端 ...

  9. Linux shell脚本笔记

    shell 命令解释器 是用来解释用户对系统的操作 使用 cat /etc/shells 可以查看 系统安装的shell Linux 启动过程: BIOS -> MBR -> BootLo ...

  10. Qt乱码的问题

    1.在启动应用程序前加入以下代码: //配置字符编码环境,让应用程序支持中文. QTextCodec *codec = QTextCodec::codecForName("System&qu ...