在Eclipse中运行hadoop程序
1、下载hadoop-eclipse-plugin-1.2.1.jar，并将之复制到eclipse/plugins下。
2、打开map-reduce视图
在eclipse中，打开window——>open perspetive——>other，选择map/reduce。
3、选择Map/Reduce Locations标签页，新建一个Location
4、在project exploer中，可以浏览刚才定义站点的文件系统
5、准备测试数据，并上传到hdfs中。
liaoliuqingdeMacBook-Air:Downloads liaoliuqing$ hadoop fs -mkdir in
liaoliuqingdeMacBook-Air:Downloads liaoliuqing$ hadoop fs -copyFromLocal maxTemp.txt in
liaoliuqingdeMacBook-Air:Downloads liaoliuqing$ hadoop fs -ls in
Found 1 items
-rw-r--r-- 1 liaoliuqing supergroup 953 2014-12-14 09:47 /user/liaoliuqing/in/maxTemp.txt
其中maxTemp.txt的内容如下：
123456798676231190101234567986762311901012345679867623119010123456798676231190101234561+00121534567890356
123456798676231190101234567986762311901012345679867623119010123456798676231190101234562+01122934567890456
123456798676231190201234567986762311901012345679867623119010123456798676231190101234562+02120234567893456
123456798676231190401234567986762311901012345679867623119010123456798676231190101234561+00321234567803456
123456798676231190101234567986762311902012345679867623119010123456798676231190101234561+00429234567903456
123456798676231190501234567986762311902012345679867623119010123456798676231190101234561+01021134568903456
123456798676231190201234567986762311902012345679867623119010123456798676231190101234561+01124234578903456
123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+04121234678903456
123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+00821235678903456
6、准备map-reduce程序
程序请见http://blog.csdn.net/jediael_lu/article/details/37596469
7、运行程序
MaxTemperature.java——>run as——>run configuration
在arguments中填入输入及输出目录，开始run。
此处是在hdfs中运行程序，事实上也可以在本地文件系统中运行程序，此方法可以方便的用于程序调试。
如在参数中填入：
/Users/liaoliuqing/in /Users/liaoliuqing/out
即可。
8、以下是eclise console中的输出内容
14/12/14 10:52:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/12/14 10:52:05 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/12/14 10:52:05 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
14/12/14 10:52:05 INFO input.FileInputFormat: Total input paths to process : 1
14/12/14 10:52:05 WARN snappy.LoadSnappy: Snappy native library not loaded
14/12/14 10:52:06 INFO mapred.JobClient: Running job: job_local1815770300_0001
14/12/14 10:52:06 INFO mapred.LocalJobRunner: Waiting for map tasks
14/12/14 10:52:06 INFO mapred.LocalJobRunner: Starting task: attempt_local1815770300_0001_m_000000_0
14/12/14 10:52:06 INFO mapred.Task: Using ResourceCalculatorPlugin : null
14/12/14 10:52:06 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/liaoliuqing/in/maxTemp.txt:0+953
14/12/14 10:52:06 INFO mapred.MapTask: io.sort.mb = 100
14/12/14 10:52:06 INFO mapred.MapTask: data buffer = 79691776/99614720
14/12/14 10:52:06 INFO mapred.MapTask: record buffer = 262144/327680
14/12/14 10:52:06 INFO mapred.MapTask: Starting flush of map output
14/12/14 10:52:06 INFO mapred.MapTask: Finished spill 0
14/12/14 10:52:06 INFO mapred.Task: Task:attempt_local1815770300_0001_m_000000_0 is done. And is in the process of commiting
14/12/14 10:52:06 INFO mapred.LocalJobRunner:
14/12/14 10:52:06 INFO mapred.Task: Task 'attempt_local1815770300_0001_m_000000_0' done.
14/12/14 10:52:06 INFO mapred.LocalJobRunner: Finishing task: attempt_local1815770300_0001_m_000000_0
14/12/14 10:52:06 INFO mapred.LocalJobRunner: Map task executor complete.
14/12/14 10:52:06 INFO mapred.Task: Using ResourceCalculatorPlugin : null
14/12/14 10:52:06 INFO mapred.LocalJobRunner:
14/12/14 10:52:06 INFO mapred.Merger: Merging 1 sorted segments
14/12/14 10:52:06 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 90 bytes
14/12/14 10:52:06 INFO mapred.LocalJobRunner:
14/12/14 10:52:06 INFO mapred.Task: Task:attempt_local1815770300_0001_r_000000_0 is done. And is in the process of commiting
14/12/14 10:52:06 INFO mapred.LocalJobRunner:
14/12/14 10:52:06 INFO mapred.Task: Task attempt_local1815770300_0001_r_000000_0 is allowed to commit now
14/12/14 10:52:06 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1815770300_0001_r_000000_0' to hdfs://localhost:9000/user/liaoliuqing/out
14/12/14 10:52:06 INFO mapred.LocalJobRunner: reduce > reduce
14/12/14 10:52:06 INFO mapred.Task: Task 'attempt_local1815770300_0001_r_000000_0' done.
14/12/14 10:52:07 INFO mapred.JobClient: map 100% reduce 100%
14/12/14 10:52:07 INFO mapred.JobClient: Job complete: job_local1815770300_0001
14/12/14 10:52:07 INFO mapred.JobClient: Counters: 19
14/12/14 10:52:07 INFO mapred.JobClient: File Output Format Counters
14/12/14 10:52:07 INFO mapred.JobClient: Bytes Written=43
14/12/14 10:52:07 INFO mapred.JobClient: File Input Format Counters
14/12/14 10:52:07 INFO mapred.JobClient: Bytes Read=953
14/12/14 10:52:07 INFO mapred.JobClient: FileSystemCounters
14/12/14 10:52:07 INFO mapred.JobClient: FILE_BYTES_READ=450
14/12/14 10:52:07 INFO mapred.JobClient: HDFS_BYTES_READ=1906
14/12/14 10:52:07 INFO mapred.JobClient: FILE_BYTES_WRITTEN=135618
14/12/14 10:52:07 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=43
14/12/14 10:52:07 INFO mapred.JobClient: Map-Reduce Framework
14/12/14 10:52:07 INFO mapred.JobClient: Reduce input groups=5
14/12/14 10:52:07 INFO mapred.JobClient: Map output materialized bytes=94
14/12/14 10:52:07 INFO mapred.JobClient: Combine output records=0
14/12/14 10:52:07 INFO mapred.JobClient: Map input records=9
14/12/14 10:52:07 INFO mapred.JobClient: Reduce shuffle bytes=0
14/12/14 10:52:07 INFO mapred.JobClient: Reduce output records=5
14/12/14 10:52:07 INFO mapred.JobClient: Spilled Records=16
14/12/14 10:52:07 INFO mapred.JobClient: Map output bytes=72
14/12/14 10:52:07 INFO mapred.JobClient: Total committed heap usage (bytes)=329252864
14/12/14 10:52:07 INFO mapred.JobClient: SPLIT_RAW_BYTES=118
14/12/14 10:52:07 INFO mapred.JobClient: Map output records=8
14/12/14 10:52:07 INFO mapred.JobClient: Combine input records=0
14/12/14 10:52:07 INFO mapred.JobClient: Reduce input records=8
在Eclipse中运行hadoop程序的更多相关文章
- Ubuntu下Eclipse中运行Hadoop程序的参数问题
需要统一的参数: 当配置好eclipse中hadoop的程序后,几个参数需要统一一下: hadoop安装目录下/etc/core_site.xml中 fs.default.name的端口号一定要与ha ...
- 在Eclipse中运行hadoop程序 分类： A1_HADOOP 2014-12-14 11:11 624人阅读 评论(0) 收藏
1.下载hadoop-eclipse-plugin-1.2.1.jar,并将之复制到eclipse/plugins下. 2.打开map-reduce视图 在eclipse中,打开window--> ...
- 【爬坑】在 IDEA 中运行 Hadoop 程序 报 winutils.exe 不存在错误解决方案
0. 问题说明 环境为 Windows 10 在 IDEA 中运行 Hadoop 程序报 winutils.exe 不存在 错误 1. 解决方案 [1.1 解压] 解压 hadoop-2.7.3 ...
- eclipse中运行java程序
1 package ttt; public class Testttt { public static void main() { Person p =new Person(); p.name=&qu ...
- 关于在Eclipse上运行Hadoop程序的日志输出问题
在安装由Eclipse-Hadoop-Plugin的Eclipse中, 可以直接运行Hadoop的MapReduce程序, 但是如果什么都不配置的话你发现Eclipse控制台没有任何日志输出, 这个问 ...
- 关于在Eclipse中运行java程序报出：The project:XXXX which is referenced by the classpath10
1.work_space名称与project是否一样,如果是一样的可能会导致错误. 2.project所在的文件夹中的.mymetadata文件中定义的project-module名称是否与proje ...
- Ubuntu下eclipse中运行Hadoop时所需要的JRE与JDK的搭配
第一组: Eclise 版本:Indigo,Service Release 1 Build id:20110916-0149 Window-->Preferences -->Compile ...
- 使用Eclipse编译运行MapReduce程序 Hadoop2.6.0_Ubuntu/CentOS
使用Eclipse编译运行MapReduce程序 Hadoop2.6.0_Ubuntu/CentOS 2014-10-10 (updated: 2016-05-22) 64246 153 本教程介绍 ...
- Nodejs学习笔记（二）——Eclipse中运行调试Nodejs
前篇<Nodejs学习笔记(一)——初识Nodejs>主要介绍了在搭建node环境过程中遇到的小问题以及搭建Eclipse开发Node环境的前提步骤.本篇主要介绍如何在Eclipse中运行 ...
随机推荐
- C程序设计语言练习题1-10
练习1-10 编写一个将输入复制到输出的程序,并将起重的制表符替换为\t,把回退符替换成\b,把反斜杠替换为\\.这样可以将制表符和回退符以可见的方式显示出来. 代码如下: #include < ...
- angularjs学习笔记—事件指令
ngClick 适用标签:所有触发条件:单击 #html <div ng-controller="LearnCtrl"> <div ng-click=" ...
- [HDU] 2063 过山车(二分图最大匹配)
题目地址:http://acm.hdu.edu.cn/showproblem.php?pid=2063 女生为X集合,男生为Y集合,求二分图最大匹配数即可. #include<cstdio> ...
- ng-selected 与ng-options的使用
1:ng-selected用在<option>标签上面,ng-options用在<select>上面,用于动态创建options列表. <form class=" ...
- 购物车Demo,前端使用AngularJS,后端使用ASP.NET Web API(3)--Idetity,OWIN前后端验证
原文:购物车Demo,前端使用AngularJS,后端使用ASP.NET Web API(3)--Idetity,OWIN前后端验证 chsakell分享了前端使用AngularJS,后端使用ASP. ...
- Android Content Provider简介
Content Provider是Android的四大组件之一,与Activity和Service相同,使用之前需要注册: Android系统中存在大量的应用,当不同的应用程序之间需要共享数据时,可以 ...
- windows puppet manifests 文件维护
初级 puppet windows agent实现简单的msi格式安装包安装及bat文件创建;
- inconvertible types; cannot cast 'android.supoort.v4.app.Fragment' to 'com.example.sevenun.littledemo.fragment.NewsTitleFragment'
inconvertible types; cannot cast 'android.supoort.v4.app.Fragment' to 'com.example.sevenun.littledem ...
- linux高级技巧：heartbeat+lvs（一）
1.heartbeat一个简短的引论: Heartbeat 项目是 Linux-HA project的一个组成部分,它实现了一个高可用集群系统.心跳服务和集群通信是高可用集群的两个关键组 ...
- android避免service被杀
1.在service中重写下面的方法,这个方法有三个返回值, START_STICKY是service被kill掉后自动重写创建@Override public int onStartComma ...