Hadoop Ecosystem related ports
本文总结了Hadoop生态系统中各个组件使用的端口,包括了HDFS,Map Reduce,HBase,Hive,Spark,WebHCat,Impala,Alluxio,Sqoop等,后续会持续更新。
HDFS Ports:
Service |
Servers |
Default Ports Used |
Protocol |
Description |
Need End User Access? |
Configuration Parameters |
NameNode WebUI |
Master Nodes (NameNode and any back-up NameNodes) |
http |
Web UI to look at current status of HDFS, explore file system |
Yes (Typically admins, Dev/Support teams) |
dfs.http.address |
|
https |
Secure http service |
dfs.https.address |
||||
NameNode metadata service |
Master Nodes (NameNode and any back-up NameNodes) |
8020/9000 |
IPC |
File system metadata operations |
Yes (All clients who directly need to interact with the HDFS) |
Embedded in URI specified by fs.default.name |
DataNode |
All Slave Nodes |
http |
DataNode WebUI to access the status, logs etc. |
Yes (Typically admins, Dev/Support teams) |
dfs.datanode.http.address |
|
https |
Secure http service |
dfs.datanode.https.address |
||||
Data transfer |
dfs.datanode.address |
|||||
IPC |
Metadata operations |
No |
dfs.datanode.ipc.address |
|||
Secondary NameNode |
Secondary NameNode and any backup Secondary NameNode |
http |
Checkpoint for NameNode metadata |
No |
dfs.secondary.http.address |
Map Reduce Ports:
Service |
Servers |
Default Ports Used |
Protocol |
Description |
Need End User Access? |
Configuration Parameters |
JobTracker WebUI |
Master Nodes (JobTracker Node and any back-up JobTracker node ) |
http |
Web UI for JobTracker |
Yes |
mapred.job.tracker.http.address |
|
JobTracker |
Master Nodes (JobTracker Node) |
IPC |
For job submissions |
Yes (All clients who need to submit the MapReduce jobs including Hive, Hive server, Pig) |
Embedded in URI specified by mapred.job.tracker |
|
TaskTracker Web UI and Shuffle |
All Slave Nodes |
http |
DataNode Web UI to access status, logs, etc. |
Yes (Typically admins, Dev/Support teams) |
mapred.task.tracker.http.address |
|
History Server WebUI |
http |
Web UI for Job History |
Yes |
mapreduce.history.server.http.address |
HBase Ports:
Service |
Servers |
Default Ports Used |
Protocol |
Description |
Need End User Access? |
Configuration Parameters |
HMaster |
Master Nodes (HBase Master Node and any back-up HBase Master node) |
Yes |
hbase.master.port |
|||
HMaster Info Web UI |
Master Nodes (HBase master Node and back up HBase Master node if any) |
http |
The port for the HBaseMaster web UI. Set to -1 if you do not want the info server to run. |
Yes |
hbase.master.info.port |
|
Region Server |
All Slave Nodes |
Yes (Typically admins, dev/support teams) |
hbase.regionserver.port |
|||
Region Server |
All Slave Nodes |
http |
Yes (Typically admins, dev/support teams) |
hbase.regionserver.info.port |
||
All ZooKeeper Nodes |
Port used by ZooKeeper peers to talk to each other.Seehere for more information. |
No |
hbase.zookeeper.peerport |
|||
All ZooKeeper Nodes |
Port used by ZooKeeper peers to talk to each other.Seehere for more information. |
hbase.zookeeper.leaderport |
||||
Property from ZooKeeper's config zoo.cfg. The port at which the clients will connect. |
hbase.zookeeper.property.clientPort |
Hive Ports:
Service |
Servers |
Default Ports Used |
Protocol |
Description |
Need End User Access? |
Configuration Parameters |
Hive Server2 |
Hive Server machine (Usually a utility machine) |
thrift |
Service for programatically (Thrift/JDBC) connecting to Hive |
Yes (Clients who need to connect to Hive either programatically or through UI SQL tools that use JDBC) |
ENV Variable HIVE_PORT |
|
Hive Metastore |
thrift |
Yes (Clients that run Hive, Pig and potentially M/R jobs that use HCatalog) |
hive.metastore.uris |
WebHCat Ports:
Service |
Servers |
Default Ports Used |
Protocol |
Description |
Need End User Access? |
WebHCat Server |
Any utility machine |
http |
Web API on top of HCatalog and other Hadoop services |
Yes |
Spark Ports:
Service |
Servers |
Default Ports Used |
Description |
Spark GUI |
Nodes running spark |
Spark web interface for monitoring and troubleshooting |
Impala Ports:
Service |
Servers |
Default Ports Used |
Description |
Impala Daemon |
Nodes running impala daemon |
Used by transmit commands and receive results by impala-shell |
|
Impala Daemon |
Nodes running impala daemon |
Used by applications through JDBC |
|
Impala Daemon |
Nodes running impala daemon |
Impala web interface for monitoring and troubleshooting |
|
Impala StateStore Daemon |
Nodes running impala StateStore daemon |
StateStore web interface for monitoring and troubleshooting |
|
Impala Catalog Daemon |
Nodes running impala catalog daemon |
Catalog service web interface for monitoring and troubleshooting |
Alluxio Ports:
Service |
Servers |
Default Ports Used |
Protocol |
Description |
Need End User Access? |
Alluxio Web GUI |
Any utility machine |
http |
Web GUI to check alluxio status |
Yes |
|
Alluxio API |
Any utility machine |
Tcp |
Api to access data on alluxio |
No |
Sqoop Ports:
Service |
Servers |
Default Ports Used |
Description |
Sqoop server |
Nodes running Sqoop |
Used by Sqoop client to access the sqoop server |
Hadoop Ecosystem related ports的更多相关文章
- Hadoop ecosystem notes Outline - TODO
Motivation Sometimes I fell like giving up, then I remember I have a lot of motherfuckers to prove w ...
- Hadoop ecosystem
How did it all start- huge data on the web! Nutch built to crawl this web data Huge data had to save ...
- Hadoop ecosystem 生态圈
Cascading: hadoop上面的workflow Sqoop(发音:skup)是一款开源的工具,主要用于在Hadoop(Hive)与传统的数据库(mysql.postgresql...)间进行 ...
- hadoop发行版本
Azure HDInsight Azure HDInsight is Microsoft's distribution of Hadoop. The Azure HDInsight ecosystem ...
- Hadoop HDFS 用户指南
This document is a starting point for users working with Hadoop Distributed File System (HDFS) eithe ...
- 关于hadoop
hadoop 是什么? 1. 适合海量数据的分布式存储与计算平台. 海量: 是指 1T 以上数据. 分布式: 任务分配到多态虚拟机上进行计算. 2. 多个任务是怎么被分配到多个虚拟机当中的? 分配是需 ...
- 使用Windows Azure的VM安装和配置CDH搭建Hadoop集群
本文主要内容是使用Windows Azure的VIRTUAL MACHINES和NETWORKS服务安装CDH (Cloudera Distribution Including Apache Hado ...
- Hadoop入门进阶课程10--HBase介绍、安装与应用案例
本文版权归作者和博客园共有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接,博主为石山园,博客地址为 http://www.cnblogs.com/shishanyuan ...
- [Hadoop 周边] Hadoop技术生态圈
Hadoop版本演进 当前Hadoop有两大版本:Hadoop 1.0和Hadoop 2.0. Hadoop1.0被称为第一代Hadoop,由分布式文件系统HDFS和分布式计算框架MapReduce组 ...
随机推荐
- c++基础之struct
就是让用户自己自定义一个要往里面放各种东西的抽屉 // 声明一个结构体类型 Books struct Books { ]; ]; ]; int book_id; }; int main( ) { Bo ...
- MongoDB整理笔记の管理Sharding
1.列出所有的Shard Server > db.runCommand({ listshards: 1 }) --列出所有的Shard Server { "shards" : ...
- JS和JQuery的比较
一. Jquery它是javascript的一个轻量级框架,是对javascript进行封装. 二.JQuery和JS都有加载函数,但表达方式不同. 1.JS中的加载函数: //整个文档加载完毕后执行 ...
- angular 引入第三方库
第一步 --save:自动写入package.json 第二步: 第三部: 为了让typescript识别$ 第四步:
- javascript 视频播放指定的时间段
javascript 视频播放指定的时间段 一.html5 vedio: //指定开始时间 player.currentTime=startPoint; player.play(); //使用事件来控 ...
- sublime 快捷键 1到9的顺序
p{$$}*9
- 利用keytool工具生成数字证书
一.制作数字证书 因测试微信小程序, 腾讯要求使用 https协议,所以需要使用证书.使用jdk工具制作数字证书流程如下: 1.查看JDK是否安装,使用命令java -version 2.切换目录至 ...
- Jmeter_远程启动
Jmeter 是Java 应用,对于CPU和内存的消耗比较大,因此,当需要模拟数以千计的并发用户时,使用单台机器模拟所有的并发用户就有些力不从心,甚至会引起JAVA内存溢出错误. 其实,Jmeter的 ...
- Java面向对象之多态(成员访问特点) 入门实例
一.基础概念 多态的调用方式在子父类中的特殊体现. 1.访问成员变量特点: 当子父类中出现同名成员变量时. 多态调用时,编译和运行都参考引用型变量所属的类中的成员变量. 即编译和运行看等号的左边. 2 ...
- linux系统安全及应用——账号安全(基本安全措施)
不开启桌面可以减少受攻击面 一.系统账号清理 1)非登录用户的shell改为/sbin/nologin ~] #usermod -s /sbin/nologin user1 2)锁定长期不用的账号 锁 ...