参考官方文档:http://gora.apache.org/current/tutorial.html

项目代码见:https://code.csdn.net/jediael_lu/mygorademo

另环境准备见: http://blog.csdn.net/jediael_lu/article/details/43272521

当着数据已通过之前的示例存储在hbase中,数据如下:

\x00\x00\x00\x00\x00\x00\x00D              column=common:ip, timestamp=1422529645469, value=85.100.75.104
\x00\x00\x00\x00\x00\x00\x00D column=common:timestamp, timestamp=1422529645469, value=\x00\x00\x01\x1F\xF1\xB5\x88\xA0
\x00\x00\x00\x00\x00\x00\x00D column=common:url, timestamp=1422529645469, value=/index.php?i=2&a=1__z_nccylulyu&k=238241
\x00\x00\x00\x00\x00\x00\x00D column=http:httpMethod, timestamp=1422529645469, value=GET
\x00\x00\x00\x00\x00\x00\x00D column=http:httpStatusCode, timestamp=1422529645469, value=\x00\x00\x00\xC8
\x00\x00\x00\x00\x00\x00\x00D column=http:responseSize, timestamp=1422529645469, value=\x00\x00\x00+
\x00\x00\x00\x00\x00\x00\x00D column=misc:referrer, timestamp=1422529645469, value=http://www.buldinle.com/index.php?i=2&a=1__Z_nccYlULyU&k=238241
\x00\x00\x00\x00\x00\x00\x00D column=misc:userAgent, timestamp=1422529645469, value=Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.0.7) Gecko/2009021
910 Firefox/3.0.7
\x00\x00\x00\x00\x00\x00\x00E column=common:ip, timestamp=1422529645469, value=85.100.75.104
\x00\x00\x00\x00\x00\x00\x00E column=common:timestamp, timestamp=1422529645469, value=\x00\x00\x01\x1F\xF1\xB5\xBFP
\x00\x00\x00\x00\x00\x00\x00E column=common:url, timestamp=1422529645469, value=/index.php?i=7&a=1__yxs0vome9p8&k=4924961
\x00\x00\x00\x00\x00\x00\x00E column=http:httpMethod, timestamp=1422529645469, value=GET
\x00\x00\x00\x00\x00\x00\x00E column=http:httpStatusCode, timestamp=1422529645469, value=\x00\x00\x00\xC8
\x00\x00\x00\x00\x00\x00\x00E column=http:responseSize, timestamp=1422529645469, value=\x00\x00\x00+
\x00\x00\x00\x00\x00\x00\x00E column=misc:referrer, timestamp=1422529645469, value=http://www.buldinle.com/index.php?i=7&a=1__YxS0VoME9P8&k=4924961
\x00\x00\x00\x00\x00\x00\x00E column=misc:userAgent, timestamp=1422529645469, value=Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.0.7) Gecko/2009021
910 Firefox/3.0.7

本例将使用MR读取hbase中的数据,并进行分析,分析每个url,一天时间内有多少人在访问,输出结果保存在hbase中,表中的key为“url+时间”格式的String,value包括三列,分别是url,时间,访问次数。



0、创建java project及gora.properties,内容如下:

##gora.datastore.default is the default detastore implementation to use
##if it is not passed to the DataStoreFactory#createDataStore() method.
gora.datastore.default=org.apache.gora.hbase.store.HBaseStore ##whether to create schema automatically if not exists.
gora.datastore.autocreateschema=true

1、创建用于对应输入数据的json文件,并生成相应的类。

上个示例已经完成,见passview.json与PageView.java

{
"type": "record",
"name": "Pageview", "default":null,
"namespace": "org.apache.gora.tutorial.log.generated",
"fields" : [
{"name": "url", "type": ["null","string"], "default":null},
{"name": "timestamp", "type": "long", "default":0},
{"name": "ip", "type": ["null","string"], "default":null},
{"name": "httpMethod", "type": ["null","string"], "default":null},
{"name": "httpStatusCode", "type": "int", "default":0},
{"name": "responseSize", "type": "int", "default":0},
{"name": "referrer", "type": ["null","string"], "default":null},
{"name": "userAgent", "type": ["null","string"], "default":null}
]
}

2、创建输入数据的类与表映射文件

<?xml version="1.0" encoding="UTF-8"?>

<!--
Gora Mapping file for HBase Backend
-->
<gora-otd>
<table name="Pageview"> <!-- optional descriptors for tables -->
<family name="common"/> <!-- This can also have params like compression, bloom filters -->
<family name="http"/>
<family name="misc"/>
</table> <class name="org.apache.gora.tutorial.log.generated.Pageview" keyClass="java.lang.Long" table="AccessLog">
<field name="url" family="common" qualifier="url"/>
<field name="timestamp" family="common" qualifier="timestamp"/>
<field name="ip" family="common" qualifier="ip" />
<field name="httpMethod" family="http" qualifier="httpMethod"/>
<field name="httpStatusCode" family="http" qualifier="httpStatusCode"/>
<field name="responseSize" family="http" qualifier="responseSize"/>
<field name="referrer" family="misc" qualifier="referrer"/>
<field name="userAgent" family="misc" qualifier="userAgent"/>
</class> </gora-otd>

3、创建用于对于输出数据的json文件,并生成相应的类。

{
"type": "record",
"name": "MetricDatum",
"namespace": "org.apache.gora.tutorial.log.generated",
"fields" : [
{"name": "metricDimension", "type": "string"},
{"name": "timestamp", "type": "long"},
{"name": "metric", "type" : "long"}
]
}

liaoliuqingdeMacBook-Air:MyGoraDemo liaoliuqing$ gora goracompiler avro/metricdatum.json src/

Compiling: /Users/liaoliuqing/99_Project/git/MyGoraDemo/avro/metricdatum.json

Compiled into: /Users/liaoliuqing/99_Project/git/MyGoraDemo/src

Compiler executed SUCCESSFULL.





4、创建输出数据的类与表映射内容,并将之加入第2步创建的文件中。

  <class name="org.apache.gora.tutorial.log.generated.MetricDatum" keyClass="java.lang.String" table="Metrics">
<field name="metricDimension" family="common" qualifier="metricDimension"/>
<field name="timestamp" family="common" qualifier="ts"/>
<field name="metric" family="common" qualifier="metric"/>
</class>

5、写主类文件

程序处理的关键步骤:

(1)获取输入、输出DataStore

    if(args.length > 0) {
String dataStoreClass = args[0];
inStore = DataStoreFactory.
getDataStore(dataStoreClass, Long.class, Pageview.class, conf);
if(args.length > 1) {
dataStoreClass = args[1];
}
outStore = DataStoreFactory.
getDataStore(dataStoreClass, String.class, MetricDatum.class, conf);
} else {
inStore = DataStoreFactory.getDataStore(Long.class, Pageview.class, conf);
outStore = DataStoreFactory.getDataStore(String.class, MetricDatum.class, conf);
}

(2)设置job的一些基本属性

    Job job = new Job(getConf());
job.setJobName("Log Analytics");
log.info("Creating Hadoop Job: " + job.getJobName());
job.setNumReduceTasks(numReducer);
job.setJarByClass(getClass());

(3)定义job相关的Map类及mapr的输入输出信息。

GoraMapper.initMapperJob(job, inStore, TextLong.class, LongWritable.class,
LogAnalyticsMapper.class, true);

(4)定义job相关的Reduce类及reduce的输入输出信息。

    GoraReducer.initReducerJob(job, outStore, LogAnalyticsReducer.class);

(5)定义map类

public static class LogAnalyticsMapper extends GoraMapper<Long, Pageview, TextLong,
LongWritable> { private LongWritable one = new LongWritable(1L); private TextLong tuple; @Override
protected void setup(Context context) throws IOException ,InterruptedException {
tuple = new TextLong();
tuple.setKey(new Text());
tuple.setValue(new LongWritable());
}; @Override
protected void map(Long key, Pageview pageview, Context context)
throws IOException ,InterruptedException { CharSequence url = pageview.getUrl();
long day = getDay(pageview.getTimestamp()); tuple.getKey().set(url.toString());
tuple.getValue().set(day); context.write(tuple, one);
}; /** Rolls up the given timestamp to the day cardinality, so that
* data can be aggregated daily */
private long getDay(long timeStamp) {
return (timeStamp / DAY_MILIS) * DAY_MILIS;
}
}

(6)定义reduce类

public static class LogAnalyticsReducer extends GoraReducer<TextLong, LongWritable,
String, MetricDatum> { private MetricDatum metricDatum = new MetricDatum(); @Override
protected void reduce(TextLong tuple, Iterable<LongWritable> values, Context context)
throws IOException ,InterruptedException { long sum = 0L; //sum up the values
for(LongWritable value: values) {
sum+= value.get();
} String dimension = tuple.getKey().toString();
long timestamp = tuple.getValue().get(); metricDatum.setMetricDimension(new Utf8(dimension));
metricDatum.setTimestamp(timestamp); String key = metricDatum.getMetricDimension().toString();
key += "_" + Long.toString(timestamp);
metricDatum.setMetric(sum); context.write(key, metricDatum);
};
}

(8)使用输入输出DataStore来创建一个job,并执行

    Job job = createJob(inStore, outStore, 3);
boolean success = job.waitForCompletion(true);

其实使用Gora与一般的MR程序的主要区别在于:

(1)继承于GoraMapper/GoraReducer,而不是Mapper/Reducer。

(2)使用GoraMapper.initMapperJob(), GoraReducer.initReducerJob()设置输入输出类型,而且可以使用一个DataSource类对象表示输入/输出的KEY-VALUE。

如本例中的mapper,使用instroe来代替指定了输入KV类型为Long,Pageview,本例中的reducer,使用outstore来代替指定了输出类型为String, MetricDatum。

对比http://blog.csdn.net/jediael_lu/article/details/43416751中所描述的运行一个job所需的基本属性:

GoraMapper.initMapperJob(job, inStore, TextLong.class, LongWritable.class,  LogAnalyticsMapper.class, true);
GoraReducer.initReducerJob(job, outStore, LogAnalyticsReducer.class);

以上语句同时完成了2、3、4、5步,即

指定了2、Map/Reduce的类:LogAnalyticsMapper.class与LogAnalyticsReducer.class

指定了3、4、输入格式及内容及5、reduce的输出类型:即输入输出均为DataSource格式,内容为inStore与outStore中的内容。

指定了5、指定了map的输出类型,这也是reduce的输入类型。

附详细代码:

(1)KeyValueWritable.java

package org.apache.gora.tutorial.log;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException; import org.apache.hadoop.io.WritableComparable; /**
* A WritableComparable containing a key-value WritableComparable pair.
* @param <K> the class of key
* @param <V> the class of value
*/
public class KeyValueWritable<K extends WritableComparable, V extends WritableComparable>
implements WritableComparable<KeyValueWritable<K,V>> { protected K key = null;
protected V value = null; public KeyValueWritable() {
} public KeyValueWritable(K key, V value) {
this.key = key;
this.value = value;
} public K getKey() {
return key;
} public void setKey(K key) {
this.key = key;
} public V getValue() {
return value;
} public void setValue(V value) {
this.value = value;
} @Override
public void readFields(DataInput in) throws IOException {
if(key == null) { }
key.readFields(in);
value.readFields(in);
} @Override
public void write(DataOutput out) throws IOException {
key.write(out);
value.write(out);
} @Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((key == null) ? 0 : key.hashCode());
result = prime * result + ((value == null) ? 0 : value.hashCode());
return result;
} @Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (obj == null)
return false;
if (getClass() != obj.getClass())
return false;
KeyValueWritable other = (KeyValueWritable) obj;
if (key == null) {
if (other.key != null)
return false;
} else if (!key.equals(other.key))
return false;
if (value == null) {
if (other.value != null)
return false;
} else if (!value.equals(other.value))
return false;
return true;
} @Override
public int compareTo(KeyValueWritable<K, V> o) {
int cmp = key.compareTo(o.key);
if(cmp != 0)
return cmp; return value.compareTo(o.value);
}
}

(2) TextLong.java

package org.apache.gora.tutorial.log;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text; /**
* A {@link KeyValueWritable} of {@link Text} keys and
* {@link LongWritable} values.
*/
public class TextLong extends KeyValueWritable<Text, LongWritable> { public TextLong() {
key = new Text();
value = new LongWritable();
} }

(3) LogAnalytics.java

package org.apache.gora.tutorial.log;

import java.io.IOException;

import org.apache.avro.util.Utf8;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.gora.mapreduce.GoraMapper;
import org.apache.gora.mapreduce.GoraReducer;
import org.apache.gora.store.DataStore;
import org.apache.gora.store.DataStoreFactory;
import org.apache.gora.tutorial.log.generated.MetricDatum;
import org.apache.gora.tutorial.log.generated.Pageview;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner; /**
* LogAnalytics is the tutorial class to illustrate Gora MapReduce API.
* The analytics mapreduce job reads the web access data stored earlier by the
* {@link LogManager}, and calculates the aggregate daily pageviews. The
* output of the job is stored in a Gora compatible data store.
*
* <p>See the tutorial.html file in docs or go to the
* <a href="http://incubator.apache.org/gora/docs/current/tutorial.html">
* web site</a>for more information.</p>
*/
public class LogAnalytics extends Configured implements Tool { private static final Logger log = LoggerFactory.getLogger(LogAnalytics.class); /** The number of miliseconds in a day */
private static final long DAY_MILIS = 1000 * 60 * 60 * 24; /**
* The Mapper takes Long keys and Pageview objects, and emits
* tuples of <url, day> as keys and 1 as values. Input values are
* read from the input data store.
* Note that all Hadoop serializable classes can be used as map output key and value.
*
*/
//6、定义map类
public static class LogAnalyticsMapper extends GoraMapper<Long, Pageview, TextLong,
LongWritable> { private LongWritable one = new LongWritable(1L); private TextLong tuple; @Override
protected void setup(Context context) throws IOException ,InterruptedException {
tuple = new TextLong();
tuple.setKey(new Text());
tuple.setValue(new LongWritable());
}; @Override
protected void map(Long key, Pageview pageview, Context context)
throws IOException ,InterruptedException { CharSequence url = pageview.getUrl();
long day = getDay(pageview.getTimestamp()); tuple.getKey().set(url.toString());
tuple.getValue().set(day); context.write(tuple, one);
}; /** Rolls up the given timestamp to the day cardinality, so that
* data can be aggregated daily */
private long getDay(long timeStamp) {
return (timeStamp / DAY_MILIS) * DAY_MILIS;
}
} /**
* The Reducer receives tuples of <url, day> as keys and a list of
* values corresponding to the keys, and emits a combined keys and
* {@link MetricDatum} objects. The metric datum objects are stored
* as job outputs in the output data store.
*/
//7、定义reduce类
public static class LogAnalyticsReducer extends GoraReducer<TextLong, LongWritable,
String, MetricDatum> { private MetricDatum metricDatum = new MetricDatum(); @Override
protected void reduce(TextLong tuple, Iterable<LongWritable> values, Context context)
throws IOException ,InterruptedException { long sum = 0L; //sum up the values
for(LongWritable value: values) {
sum+= value.get();
} String dimension = tuple.getKey().toString();
long timestamp = tuple.getValue().get(); metricDatum.setMetricDimension(new Utf8(dimension));
metricDatum.setTimestamp(timestamp); String key = metricDatum.getMetricDimension().toString();
key += "_" + Long.toString(timestamp);
metricDatum.setMetric(sum); context.write(key, metricDatum);
};
} /**
* Creates and returns the {@link Job} for submitting to Hadoop mapreduce.
* @param inStore
* @param outStore
* @param numReducer
* @return
* @throws IOException
*/
public Job createJob(DataStore<Long, Pageview> inStore,
DataStore<String, MetricDatum> outStore, int numReducer) throws IOException {
//3、设置job的一些基本属性
Job job = new Job(getConf());
job.setJobName("Log Analytics");
log.info("Creating Hadoop Job: " + job.getJobName());
job.setNumReduceTasks(numReducer);
job.setJarByClass(getClass()); /* Mappers are initialized with GoraMapper.initMapper() or
* GoraInputFormat.setInput()*/
//4、定义job相关的Map类及mapr的输入输出信息。
GoraMapper.initMapperJob(job, inStore, TextLong.class, LongWritable.class,
LogAnalyticsMapper.class, true); //4、定义job相关的Reduce类及reduce的输入输出信息。
/* Reducers are initialized with GoraReducer#initReducer().
* If the output is not to be persisted via Gora, any reducer
* can be used instead. */
GoraReducer.initReducerJob(job, outStore, LogAnalyticsReducer.class); return job;
} @Override
public int run(String[] args) throws Exception { DataStore<Long, Pageview> inStore;
DataStore<String, MetricDatum> outStore;
Configuration conf = new Configuration(); //1、获取输入、输出DataStore。
if(args.length > 0) {
String dataStoreClass = args[0];
inStore = DataStoreFactory.
getDataStore(dataStoreClass, Long.class, Pageview.class, conf);
if(args.length > 1) {
dataStoreClass = args[1];
}
outStore = DataStoreFactory.
getDataStore(dataStoreClass, String.class, MetricDatum.class, conf);
} else {
inStore = DataStoreFactory.getDataStore(Long.class, Pageview.class, conf);
outStore = DataStoreFactory.getDataStore(String.class, MetricDatum.class, conf);
} //2、使用输入输出DataStore来创建一个job
Job job = createJob(inStore, outStore, 3);
boolean success = job.waitForCompletion(true); inStore.close();
outStore.close(); log.info("Log completed with " + (success ? "success" : "failure")); return success ? 0 : 1;
} private static final String USAGE = "LogAnalytics <input_data_store> <output_data_store>"; public static void main(String[] args) throws Exception {
if(args.length < 2) {
System.err.println(USAGE);
System.exit(1);
}
//run as any other MR job
int ret = ToolRunner.run(new LogAnalytics(), args);
System.exit(ret);
} }

6、运行程序

(1)导出程序—>runnable jar file,并将其上传到服务器

(2)运行程序

$ java -jar MyGoraDemo.jar org.apache.gora.hbase.store.HBaseStore org.apache.gora.hbase.store.HBaseStore



(3)查看hbase中的结果

hbase(main):001:0> list

TABLE                                                                                                                                                                   

AccessLog                                                                                                                                                               

Jan2814_webpage                                                                                                                                                         

Jan2819_webpage                                                                                                                                                         

Jan2910_webpage                                                                                                                                                         

Jan2920_webpage                                                                                                                                                         

Metrics                                                                                                                                                                 

Passwd                                                                                                                                                                  

member                                                                                                                                                                  

8 row(s) in 2.6450 seconds



hbase(main):002:0> scan 'Metrics'

版权声明:本文为博主原创文章,未经博主允许不得转载。

Gora官方文档之二:Gora对Map-Reduce的支持 分类: C_OHTERS 2015-01-31 11:27 232人阅读 评论(0) 收藏的更多相关文章

  1. 【Lucene4.8教程之二】索引 2014-06-16 11:30 3845人阅读 评论(0) 收藏

    一.基础内容 0.官方文档说明 (1)org.apache.lucene.index provides two primary classes: IndexWriter, which creates ...

  2. JSON入门之二:org.json的基本用法 分类: C_OHTERS 2014-05-14 11:25 6001人阅读 评论(0) 收藏

    java中用于解释json的主流工具有org.json.json-lib与gson,本文介绍org.json的应用. 官方文档: http://www.json.org/java/ http://de ...

  3. 【solr专题之二】配置文件:solr.xml solrConfig.xml schema.xml 分类: H4_SOLR/LUCENCE 2014-07-23 21:30 1959人阅读 评论(0) 收藏

    1.关于默认搜索域 If you are using the Lucene query parser, queries that don't specify a field name will use ...

  4. 【solr基础教程之二】索引 分类: H4_SOLR/LUCENCE 2014-07-18 21:06 3331人阅读 评论(0) 收藏

    一.向Solr提交索引的方式 1.使用post.jar进行索引 (1)创建文档xml文件 <add> <doc> <field name="id"&g ...

  5. Gora官方文档之二:Gora对Map-Reduce的支持

    参考官方文档:http://gora.apache.org/current/tutorial.html 项目代码见:https://code.csdn.net/jediael_lu/mygoradem ...

  6. Gora官方范例 分类: C_OHTERS 2015-01-29 16:14 632人阅读 评论(0) 收藏

    参考官方文档:http://gora.apache.org/current/tutorial.html 项目代码见:https://code.csdn.net/jediael_lu/mygoradem ...

  7. 单机/伪分布式Hadoop2.4.1安装文档 2014-07-08 21:16 2275人阅读 评论(0) 收藏

    转载自官方文档,最新版请见:http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/SingleCluster.h ...

  8. Loader之二:CursorLoader基本实例 分类: H1_ANDROID 2013-11-16 10:50 5447人阅读 评论(0) 收藏

    参考APIDEMO:sdk\samples\android-19\content\LoaderCursor 1.创建主布局文件,里面只包含一个Fragment. <FrameLayout xml ...

  9. Gora快速入门 分类: C_OHTERS 2015-01-30 09:55 465人阅读 评论(0) 收藏

    概述 Gora是apache的一个开源项目. The Apache Gora open source framework provides an in-memory data model and pe ...

随机推荐

  1. WPF转换器

    1. 前文 在普遍的也业务系统中, 数据要驱动到操作的用户界面, 它实际储存的方式和表达方式会多种多样, 数据库存储的数字 0或1, 在界面用户看到显示只是 成功或失败, 或者存储的字符.或更多的格式 ...

  2. oracle多实例的启动与关闭

    Oracle/oracle登录 1.启监听器 lsnrctl start 监听一般不需要动,如果机器重新启动的话需要将监听启动. 查看当前SID:echo $ORACLE_SID 2.启动数据库实例: ...

  3. 通过wireshark,以及python代码收发邮件,了解smtp协议,pop协议工作过程

    40返回连接server成功 41.43发送ehlo命令查询server支持命令 返回250 44.46请求认证  server响应235认证成功 47.49发送mail命令发送者邮箱  返回250 ...

  4. 使用框架的php假设使用定时服务Cronjob

    工作须要用php开发了个监控的小程序,既然是监控就须要定时运行. 之前我用的是chrome加个定时刷新的小插件,放在server上执行.也能实现,就是别扭. 通用正规的做法应该是:linux上的Cro ...

  5. less中混合

    @charset "UTF-8"; //1 普通混合 //2 不带输出的混合:加() .font_hx(){ font-size: 28px; color: red; } h1{ ...

  6. JAVA Mail邮件实现发送

    package com.test;import java.util.Date;import java.util.Properties;import javax.mail.Message;import ...

  7. 基于Lwip协议栈中独立模式下回调函数的使用

    一.使用Lwip协议独立模式开发 最近在STM32F4上边移植了Lwip,Lwip是一个小型开源的TCP/IP协议栈,有无操作系统的支持都可以运行.我当前只测试了TCP Server功能,然后对TCP ...

  8. ToggleButton控件

    ToggleButton 两种状态 ·状态button     -继承自CompoundButton ·主要属性:-Android:textOn    -Android:textOff ·主要方法: ...

  9. 【Java】Java Socket 通信演示样例

    用socket(套接字)实现client与服务端的通信. 这里举两个样例: 第一种是每次client发送一个数据,服务端就做一个应答. (也就是要轮流发) 另外一种是client能够连续的向服务端发数 ...

  10. 洛谷 P2646 数数zzy

    P2646 数数zzy 题目描述 zzy自从数学考试连续跪掉之后,上数学课就从来不认真听了(事实上他以前也不认真听).于是他开始在草稿纸上写写画画,比如写一串奇怪的字符串.然后他决定理♂性♂愉♂悦♂一 ...