What Is Apache Hadoop?

The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

The project includes these modules:

  • Hadoop Common: The common utilities that support the other Hadoop modules.
  • Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
  • Hadoop YARN: A framework for job scheduling and cluster resource management.
  • Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.

Other Hadoop-related projects at Apache include:

  • Ambari™: A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner.
  • Avro™: A data serialization system.
  • Cassandra™: A scalable multi-master database with no single points of failure.
  • Chukwa™: A data collection system for managing large distributed systems.
  • HBase™: A scalable, distributed database that supports structured data storage for large tables.
  • Hive™: A data warehouse infrastructure that provides data summarization and ad hoc querying.
  • Mahout™: A Scalable machine learning and data mining library.
  • Pig™: A high-level data-flow language and execution framework for parallel computation.
  • Spark™: A fast and general compute engine for Hadoop data. Spark provides a simple and expressive programming model that supports a wide range of applications, including ETL, machine learning, stream processing, and graph computation.
  • ZooKeeper™: A high-performance coordination service for distributed applications.

Getting Started

To get started, begin here:

  1. Learn about Hadoop by reading the documentation.
  2. Download Hadoop from the release page.
  3. Discuss Hadoop on the mailing list.

Download Hadoop

Please head to the releases page to download a release of Apache Hadoop.

Who Uses Hadoop?

A wide variety of companies and organizations use Hadoop for both research and production. Users are encouraged to add themselves to the Hadoop PoweredBy wiki page.

News

15 October, 2013: release 2.2.0 available

Apache Hadoop 2.x reaches GA milestone! Full information about this milestone release is available at Hadoop Releases.

25 August, 2013: release 2.1.0-beta available

Apache Hadoop 2.x reaches beta milestone! Full information about this milestone release is available at Hadoop Releases.

27 December, 2011: release 1.0.0 available

Hadoop reaches 1.0.0! Full information about this milestone release is available at Hadoop Releases.

March 2011 - Apache Hadoop takes top prize at Media Guardian Innovation Awards

Described by the judging panel as a "Swiss army knife of the 21st century", Apache Hadoop picked up the innovator of the year award for having the potential to change the face of media innovations.

See The Guardian web site

January 2011 - ZooKeeper Graduates

Hadoop's ZooKeeper subproject has graduated to become a top-level Apache project.

Apache ZooKeeper can now be found at http://zookeeper.apache.org/

September 2010 - Hive and Pig Graduate

Hadoop's Hive and Pig subprojects have graduated to become top-level Apache projects.

Apache Hive can now be found at http://hive.apache.org/

Pig can now be found at http://pig.apache.org/

May 2010 - Avro and HBase Graduate

Hadoop's Avro and HBase subprojects have graduated to become top-level Apache projects.

Apache Avro can now be found at http://avro.apache.org/

Apache HBase can now be found at http://hbase.apache.org/

July 2009 - New Hadoop Subprojects

Hadoop is getting bigger!

  • Hadoop Core is renamed Hadoop Common.
  • MapReduce and the Hadoop Distributed File System (HDFS) are now separate subprojects.
  • Avro and Chukwa are new Hadoop subprojects.

See the summary descriptions for all subprojects above. Visit the individual sites for more detailed information.

March 2009 - ApacheCon EU

In case you missed it.... ApacheCon Europe 2009

November 2008 - ApacheCon US

In case you missed it.... ApacheCon US 2008

July 2008 - Hadoop Wins Terabyte Sort Benchmark

Hadoop Wins Terabyte Sort Benchmark: One of Yahoo's Hadoop clusters sorted 1 terabyte of data in 209 seconds, which beat the previous record of 297 seconds in the annual general purpose (Daytona) terabyte sort benchmark. This is the first time that either a Java or an open source program has won.

Welcome to Apache™ Hadoop®!的更多相关文章

  1. Hive创建表格报【Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException】引发的血案

    在成功启动Hive之后感慨这次终于没有出现Bug了,满怀信心地打了长长的创建表格的命令,结果现实再一次给了我一棒,报了以下的错误Error, return code 1 from org.apache ...

  2. hive 使用where条件报错 java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.ppd.ExprWalkerInfo.getConvertedNode

    hadoop 版本 2.6.0 hive版本 1.1.1 错误: java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.ppd.ExprWalk ...

  3. Hive:org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota (directories and files) of directory /mydir is exceeded: quota=100000 file count=100001

    集群中遇到了文件个数超出限制的错误: 0)昨天晚上spark 任务突然抛出了异常:org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: T ...

  4. Hadoop程序运行中的Error(1)-Error: org.apache.hadoop.hdfs.BlockMissingException

    15/03/18 09:59:21 INFO mapreduce.Job: Task Id : attempt_1426641074924_0002_m_000000_2, Status : FAIL ...

  5. Ubuntu14.04用apt在线/离线安装CDH5.1.2[Apache Hadoop 2.3.0]

    目录 [TOC] 1.CDH介绍 1.1.什么是CDH和CM? CDH一个对Apache Hadoop的集成环境的封装,可以使用Cloudera Manager进行自动化安装. Cloudera-Ma ...

  6. 【解决】org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control

    [环境信息] Hadoop版本:2.4.0 客户端OS:Windows Server 2008 R2 服务器端OS:CentOS 6.4 [问题现象] 在通过Windows客户端向Linux服务器提交 ...

  7. ERROR [org.apache.hadoop.security.UserGroupInformation] - PriviledgedActionExcep

    换了个环境,出现此异常 016-10-18 23:54:01,334 WARN [org.apache.hadoop.util.NativeCodeLoader] - Unable to load n ...

  8. org.apache.hadoop.ipc.RemoteException(java.io.IOException)

    昨晚突然之间mr跑步起来了 jps查看 进程都在的,但是在reduce任务跑了85%的时候会抛异常 异常情况如下: 2016-09-21 21:32:28,538 INFO [org.apache.h ...

  9. kylin cube测试时,报错:org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

    异常: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, i ...

  10. org.apache.hadoop.security.AccessControlException: Permission denied:

    org.apache.hadoop.security.AccessControlException: Permission denied: user=xxj, access=WRITE, inode= ...

随机推荐

  1. PHP 单链表

    <?php class Hero { public $no; public $name; public $nickname; public $next=null; public function ...

  2. oracle where与having

    where与having可以过滤,一般来说尽量使用where ,但是如果过滤条件中有组函数,只能使用having SQL> select deptno,avg(sal) from emp gro ...

  3. 记录Linux下安装elasticSearch时遇到的一些错误

    记录Linux下安装elasticSearch时遇到的一些错误 http://blog.sina.com.cn/s/blog_c90ce4e001032f7w.html (2016-11-02 22: ...

  4. Android应用----如何让应用全屏

    一般Android的应用启动时都有欢迎界面,类似QQHD启动那样.比较大方绚丽.心动不如行动,有时间自己也来实现类似的效果,嘿嘿.    观察发现QQHD的欢迎界面是全屏的,这个好办.下面就Andro ...

  5. php函数整理

    php usleep() 函数延迟代码执行若干微秒. unpack() 函数从二进制字符串对数据进行解包. uniqid() 函数基于以微秒计的当前时间,生成一个唯一的 ID. time_sleep_ ...

  6. Android Init进程命令的执行和服务的启动

    这里开始分析init进程中配置文件的解析,在配置文件中的命令的执行和服务的启动. 首先init是一个可执行文件,它的对应的Makfile是init/Android.mk. Android.mk定义了i ...

  7. HttpHelper工具类

    /// <summary> /// 类说明:HttpHelper类,用来实现Http访问,Post或者Get方式的,直接访问,带Cookie的,带证书的等方式,可以设置代理 /// 重要提 ...

  8. Android 之Service

    service是运行在后台的服务,你可以启动一个服务Service来播放音乐,或者记录你地理信息位置的改变,或者启动一个服务来运行并一直监听某种动作. 接下来分析一下service 的生命周期: 1: ...

  9. Altium Design 中差分走线的设置

    1.在原理图中,将要设置的差分对的网络名称的前缀取相同的名字,在前缀后面加后缀分别为_N 和_P,并且加上差分对指示.具体操作如下:2.在原理图界面下,单击 Place>>Directiv ...

  10. 包含无数好东西的ownCloud

    ownCloud 是一个来自 KDE 社区开发的免费软件,提供私人的 Web 服务.当前主要功能包括文件管理(内建文件分享).音乐.日历.联系人等等,可在PC和服务器上运行. 简单来说就是一个基于Ph ...