本文总结了Hadoop生态系统中各个组件使用的端口,包括了HDFS,Map Reduce,HBase,Hive,Spark,WebHCat,Impala,Alluxio,Sqoop等,后续会持续更新。

HDFS Ports

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Configuration Parameters

NameNode WebUI

Master Nodes (NameNode and any back-up NameNodes)

http

Web UI to look at current status of HDFS, explore file system

Yes (Typically admins, Dev/Support teams)

dfs.http.address

https

Secure http service

dfs.https.address

NameNode metadata service

Master Nodes (NameNode and any back-up NameNodes)

8020/9000

IPC

File system metadata operations

Yes (All clients who directly need to interact with the HDFS)

Embedded in URI specified by fs.default.name

DataNode

All Slave Nodes

http

DataNode WebUI to access the status, logs etc.

Yes (Typically admins, Dev/Support teams)

dfs.datanode.http.address

https

Secure http service

dfs.datanode.https.address

 

Data transfer

 

dfs.datanode.address

IPC

Metadata operations

No

dfs.datanode.ipc.address

Secondary NameNode

Secondary NameNode and any backup Secondary NameNode

http

Checkpoint for NameNode metadata

No

dfs.secondary.http.address

Map Reduce Ports:

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Configuration Parameters

JobTracker  WebUI

Master Nodes (JobTracker Node and any back-up Job­Tracker node )

http

Web UI for JobTracker

Yes

mapred.job.tracker.http.address

JobTracker

Master Nodes (JobTracker Node)

IPC

For job submissions

Yes (All clients who need to submit the MapReduce jobs  including Hive, Hive server, Pig)

Embedded in URI specified by mapred.job.tracker

Task­Tracker Web UI and Shuffle

All Slave Nodes

http

DataNode Web UI to access status, logs, etc.

Yes (Typically admins, Dev/Support teams)

mapred.task.tracker.http.address

History Server WebUI

 

http

Web UI for Job History

Yes

mapreduce.history.server.http.address

HBase Ports:

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Configuration Parameters

HMaster

Master Nodes (HBase Master Node and any back-up HBase Master node)

   

Yes

hbase.master.port

HMaster Info Web UI

Master Nodes (HBase master Node and back up HBase Master node if any)

http

The port for the HBase­Master web UI. Set to -1 if you do not want the info server to run.

Yes

hbase.master.info.port

Region Server

All Slave Nodes

   

Yes (Typically admins, dev/support teams)

hbase.regionserver.port

Region Server

All Slave Nodes

http

 

Yes (Typically admins, dev/support teams)

hbase.regionserver.info.port

 

All ZooKeeper Nodes

 

Port used by ZooKeeper peers to talk to each other.Seehere for more information.

No

hbase.zookeeper.peerport

 

All ZooKeeper Nodes

 

Port used by ZooKeeper peers to talk to each other.Seehere for more information.

 

hbase.zookeeper.leaderport

     

Property from ZooKeeper's config zoo.cfg. The port at which the clients will connect.

 

hbase.zookeeper.property.clientPort

Hive Ports:

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Configuration Parameters

Hive Server2

Hive Server machine (Usually a utility machine)

thrift

Service for programatically (Thrift/JDBC) connecting to Hive

Yes (Clients who need to connect to Hive either programatically or through UI SQL tools that use JDBC)

ENV Variable HIVE_PORT

Hive Metastore

 

thrift

 

Yes (Clients that run Hive, Pig and potentially M/R jobs that use HCatalog)

hive.metastore.uris

WebHCat Ports:

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

WebHCat Server

Any utility machine

http

Web API on top of HCatalog and other Hadoop services

Yes

Spark Ports:

Service

Servers

Default Ports Used

Description

Spark GUI

Nodes running spark

Spark web interface for monitoring and troubleshooting

Impala Ports:

Service

Servers

Default Ports Used

Description

Impala Daemon

Nodes running impala daemon

Used by transmit commands and receive results by impala-shell

Impala Daemon

Nodes running impala daemon

Used by applications through JDBC

Impala Daemon

Nodes running impala daemon

Impala web interface for monitoring and troubleshooting

Impala StateStore Daemon

Nodes running impala StateStore daemon

StateStore web interface for monitoring and troubleshooting

Impala Catalog Daemon

Nodes running impala catalog daemon

Catalog service web interface for monitoring and troubleshooting

Alluxio Ports:

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Alluxio Web GUI

Any utility machine

http

Web GUI to check alluxio status

Yes

Alluxio API

Any utility machine

Tcp

Api to access data on alluxio

No

Sqoop Ports:

Service

Servers

Default Ports Used

Description

Sqoop server

Nodes running Sqoop

Used by Sqoop client to access the sqoop server

Hadoop Ecosystem related ports的更多相关文章

  1. Hadoop ecosystem notes Outline - TODO

    Motivation Sometimes I fell like giving up, then I remember I have a lot of motherfuckers to prove w ...

  2. Hadoop ecosystem

    How did it all start- huge data on the web! Nutch built to crawl this web data Huge data had to save ...

  3. Hadoop ecosystem 生态圈

    Cascading: hadoop上面的workflow Sqoop(发音:skup)是一款开源的工具,主要用于在Hadoop(Hive)与传统的数据库(mysql.postgresql...)间进行 ...

  4. hadoop发行版本

    Azure HDInsight Azure HDInsight is Microsoft's distribution of Hadoop. The Azure HDInsight ecosystem ...

  5. Hadoop HDFS 用户指南

    This document is a starting point for users working with Hadoop Distributed File System (HDFS) eithe ...

  6. 关于hadoop

    hadoop 是什么? 1. 适合海量数据的分布式存储与计算平台. 海量: 是指 1T 以上数据. 分布式: 任务分配到多态虚拟机上进行计算. 2. 多个任务是怎么被分配到多个虚拟机当中的? 分配是需 ...

  7. 使用Windows Azure的VM安装和配置CDH搭建Hadoop集群

    本文主要内容是使用Windows Azure的VIRTUAL MACHINES和NETWORKS服务安装CDH (Cloudera Distribution Including Apache Hado ...

  8. Hadoop入门进阶课程10--HBase介绍、安装与应用案例

    本文版权归作者和博客园共有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接,博主为石山园,博客地址为 http://www.cnblogs.com/shishanyuan  ...

  9. [Hadoop 周边] Hadoop技术生态圈

    Hadoop版本演进 当前Hadoop有两大版本:Hadoop 1.0和Hadoop 2.0. Hadoop1.0被称为第一代Hadoop,由分布式文件系统HDFS和分布式计算框架MapReduce组 ...

随机推荐

  1. c++基础之struct

    就是让用户自己自定义一个要往里面放各种东西的抽屉 // 声明一个结构体类型 Books struct Books { ]; ]; ]; int book_id; }; int main( ) { Bo ...

  2. MongoDB整理笔记の管理Sharding

    1.列出所有的Shard Server > db.runCommand({ listshards: 1 }) --列出所有的Shard Server { "shards" : ...

  3. JS和JQuery的比较

    一. Jquery它是javascript的一个轻量级框架,是对javascript进行封装. 二.JQuery和JS都有加载函数,但表达方式不同. 1.JS中的加载函数: //整个文档加载完毕后执行 ...

  4. angular 引入第三方库

    第一步 --save:自动写入package.json 第二步: 第三部: 为了让typescript识别$ 第四步:

  5. javascript 视频播放指定的时间段

    javascript 视频播放指定的时间段 一.html5 vedio: //指定开始时间 player.currentTime=startPoint; player.play(); //使用事件来控 ...

  6. sublime 快捷键 1到9的顺序

    p{$$}*9

  7. 利用keytool工具生成数字证书

    一.制作数字证书  因测试微信小程序, 腾讯要求使用 https协议,所以需要使用证书.使用jdk工具制作数字证书流程如下: 1.查看JDK是否安装,使用命令java -version 2.切换目录至 ...

  8. Jmeter_远程启动

    Jmeter 是Java 应用,对于CPU和内存的消耗比较大,因此,当需要模拟数以千计的并发用户时,使用单台机器模拟所有的并发用户就有些力不从心,甚至会引起JAVA内存溢出错误. 其实,Jmeter的 ...

  9. Java面向对象之多态(成员访问特点) 入门实例

    一.基础概念 多态的调用方式在子父类中的特殊体现. 1.访问成员变量特点: 当子父类中出现同名成员变量时. 多态调用时,编译和运行都参考引用型变量所属的类中的成员变量. 即编译和运行看等号的左边. 2 ...

  10. linux系统安全及应用——账号安全(基本安全措施)

    不开启桌面可以减少受攻击面 一.系统账号清理 1)非登录用户的shell改为/sbin/nologin ~] #usermod -s /sbin/nologin user1 2)锁定长期不用的账号 锁 ...