要求:计算hasgj表,计算每天新增mac数量。

因为spark直接扫描hbase表,对hbase集群访问量太大,给集群造成压力,这里考虑用spark读取HFile进行数据分析。

1、建立hasgj表的快照表:hasgjSnapshot

语句为:snapshot 'hasgj','hasgjSnapshot'

2、计算每天mac增量的代码如下:

package com.ba.sparkReadHbase.operatorHfile.hfileinputformat;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.NavigableMap;
import java.util.Set;
import java.util.Map.Entry;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableInputFormat;
import org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat;
import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
import org.apache.hadoop.hbase.protobuf.generated.ClientProtos;
import org.apache.hadoop.hbase.util.Base64;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.mapreduce.Job;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import scala.Tuple2; public class SparkReadHFile {
private static String convertScanToString(Scan scan) throws IOException {
ClientProtos.Scan proto = ProtobufUtil.toScan(scan);
return Base64.encodeBytes(proto.toByteArray());
} public static void main(String[] args) throws IOException {
final String date=args[0];
int max_versions = 3;
SparkConf sparkConf = new SparkConf().setAppName("sparkReadHfile");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
Configuration hconf = HBaseConfiguration.create();
hconf.set("hbase.rootdir", "/hbase");
hconf.set("hbase.zookeeper.quorum", "master,slave1,slave2");
Scan scan = new Scan();
scan.addFamily(Bytes.toBytes("ba"));
scan.setMaxVersions(max_versions);
hconf.set(TableInputFormat.SCAN, convertScanToString(scan));
Job job = Job.getInstance(hconf);
Path path = new Path("/snapshot");
String snapName ="hasgjSnapshot";
TableSnapshotInputFormat.setInput(job, snapName, path);
JavaPairRDD<ImmutableBytesWritable, Result> newAPIHadoopRDD = sc.newAPIHadoopRDD(job.getConfiguration(), TableSnapshotInputFormat.class, ImmutableBytesWritable.class,Result.class);
List<String> collect = newAPIHadoopRDD.map(new Function<Tuple2<ImmutableBytesWritable, Result>, String>(){
private static final long serialVersionUID = 1L;
public String call(Tuple2<ImmutableBytesWritable, Result> v1)
throws Exception {
// TODO Auto-generated method stub
String newMac =null;
Result result = v1._2();
if (result.isEmpty()) {
return null;
}
String rowKey = Bytes.toString(result.getRow());
//System.out.println("行健为:"+rowKey);
NavigableMap<byte[], byte[]> familyMap = result.getFamilyMap(Bytes.toBytes("ba"));
Set<Entry<byte[], byte[]>> entrySet = familyMap.entrySet();
java.util.Iterator<Entry<byte[], byte[]>> it = entrySet.iterator();
String colunNmae =null;
String minDate="34561213";
while(it.hasNext()){
colunNmae = new String(it.next().getKey());//列
if(colunNmae.compareTo(minDate)<0){
minDate=colunNmae;
}
} if (date.equals(minDate)) {
// row=rowKey.substring(4);
newMac=rowKey;
//ls.add(rowKey.substring(4));
//bf.append(rowKey+"----");
}
return newMac;
}
}).collect();
ArrayList<String> arrayList = new ArrayList<String>();
for (int i = 0; i < collect.size(); i++) {
if (collect.get(i) !=null) {
arrayList.add(collect.get(i));
}
}
System.out.println("新增mac数"+(arrayList.size())); }
}

3、特别说明:

hasgj表的表结构:

0000F470ABF3A587                          column=ba:20170802, timestamp=1517558687930, value=                                                                         
 0000F470ABF3A587                          column=ba:20170804, timestamp=1517593923254, value=                                                                         
 0000F470ABF3A587                          column=ba:20170806, timestamp=1517620990589, value=                                                                         
 0000F470ABF3A587                          column=ba:20170809, timestamp=1517706294758, value=                                                                         
 0000F470ABF3A587                          column=ba:20170810, timestamp=1517722369020, value=                                                                         
 0000F470ABF3A587                          column=ba:20170811, timestamp=1517796060984, value=                                                                         
 0000F470ABF3A587                          column=ba:20170816, timestamp=1517882948856, value=                                                                         
 0000F470ABF3A587                          column=ba:20170818, timestamp=1517912603602, value=                                                                         
 0000F470ABF3A587                          column=ba:20170819, timestamp=1517938488763, value=                                                                         
 0000F470ABF3A587                          column=ba:20170821, timestamp=1517989742180, value=                                                                         
 0000F470ABF3A587                          column=ba:20170827, timestamp=1518383470292, value=                                                                         
 0000F470ABF3A587                          column=ba:20170828, timestamp=1520305841272, value=                                                                         
 0000F470ABF3A587                          column=ba:20170831, timestamp=1522115116459, value=                                                                         
 0000F4730088A5D3                          column=ba:20170805, timestamp=1517598564121, value=                                                                         
 0000F47679E83F7D                          column=ba:20170817, timestamp=1517890046587, value=                                                                         
 0000F47FBA753FC7                          column=ba:20170827, timestamp=1518365792130, value=                                                                         
 0000F48C02F8EB83                          column=ba:20170810, timestamp=1517729864592, value=                                                                         
 0000F49578E63F55                          column=ba:20170828, timestamp=1520302223714, value=                                                                         
 0000F4AC4A93F7A5                          column=ba:20170810, timestamp=1517724545955, value=                                                                         
 0000F4B4807679AA                          column=ba:20170801, timestamp=1517543775374, value=                                                                         
 0000F4B7E374C0FF                          column=ba:20170804, timestamp=1517578239073, value=                                                                         
 0000F4BDBF6EBF37                          column=ba:20170829, timestamp=1520558747936, value=                                                                         
 0000F4CB52FDDA58                          column=ba:20170806, timestamp=1517638015583, value=                                                                         
 0000F4CB52FDDA58                          column=ba:20170807, timestamp=1517677405900, value=

4、提交作业命令:

./spark-submit --master yarn-client  --num-executors 7 --executor-cores 2 --driver-memory 2g  --executor-memory 30g --class com.ba.sparkReadHbase.operatorHfile.hfileinputformat.SparkReadHFile  /home/xxx0108/ftttttttt/testJar/sparkTest9.jar 20170806

spark读HFile对hbase表数据进行分析的更多相关文章

  1. 数据分页处理系列之二:HBase表数据分页处理

      HBase是Hadoop大数据生态技术圈中的一项关键技术,是一种用于分布式存储大数据的列式数据库,关于HBase更加详细的介绍和技术细节,朋友们可以在网络上进行搜寻,笔者本人在接下来的日子里也会写 ...

  2. HBase(三): Azure HDInsigt HBase表数据导入本地HBase

    目录: hdfs 命令操作本地 hbase Azure HDInsight HBase表数据导入本地 hbase hdfs命令操作本地hbase: 参见  HDP2.4安装(五):集群及组件安装 , ...

  3. 一种HBase表数据迁移方法的优化

    1.背景调研: 目前存在的hbase数据迁移主要分如下几类: 根据上图,可以看出: 其实主要分为两种方式:(1)hadoop层:因为hbase底层是基于hdfs存储的,所以可以通过把hdfs上的数据拷 ...

  4. HBase表数据分页处理

    HBase表数据分页处理 HBase是Hadoop大数据生态技术圈中的一项关键技术,是一种用于分布式存储大数据的列式数据库,关于HBase更加详细的介绍和技术细节,朋友们可以在网络上进行搜寻,笔者本人 ...

  5. HBase表数据的转移之使用自定义MapReduce

    目标:将fruit表中的一部分数据,通过MR迁入到fruit_mr表中 Step1.构建ReadFruitMapper类,用于读取fruit表中的数据 package com.z.hbase_mr; ...

  6. Spark读HBase写MySQL

    1 Spark读HBase Spark读HBase黑名单数据,过滤出当日新增userid,并与mysql黑名单表内userid去重后,写入mysql. def main(args: Array[Str ...

  7. hbase操作(shell 命令,如建表,清空表,增删改查)以及 hbase表存储结构和原理

    两篇讲的不错文章 http://www.cnblogs.com/nexiyi/p/hbase_shell.html http://blog.csdn.net/u010967382/article/de ...

  8. Hive如何加载和导入HBase的数据

    当我们用HBase 存储实时数据的时候, 如果要做一些数据分析方面的操作, 就比较困难了, 要写MapReduce Job. Hive 主要是用来做数据分析的数据仓库,支持标准SQL 查询, 做数据分 ...

  9. 用Spark查询HBase中的表数据

    java代码如下: package db.query; import org.apache.commons.logging.Log; import org.apache.commons.logging ...

随机推荐

  1. springboot:bootstrap和application有什么区别?

    bootstrap和application区别: Spring Cloud 构建于 Spring Boot 之上,在 Spring Boot 中有两种上下文,一种是 bootstrap,另外一种是 a ...

  2. Word表格和文本自由互换

    未完 ...... 点击访问原文(进入后根据右侧标签,快速定位到本文)

  3. FireDAC 如何按整型(Byte)读取 MySQL TinyInt(1) 类型字段?

    最近使用 MySQL 发现 FireDAC 中查询 TinyInt(1) 字段结果是 Boolean 类型,这并不是我想要的结果,而TinyInt(1)的范围是-128-127之间,如何按整型读取呢? ...

  4. Python重要配置大全

    PYTHON 环境安装 安装虚拟环境 pip install virtualenv 卸载包是用:pip uninstall virtualenv 快捷下载安装可用豆瓣源,方法为: pip instal ...

  5. python读取文件行数和某行内容

    学习记录: python计算文件的行数和读取某一行内容的实现方法 - nkwy2012 - 博客园https://www.cnblogs.com/nkwy2012/p/6023710.html 文本文 ...

  6. C#7 进入快速迭代道路

    out变量 有一定C#编程经历的园友一定没少写如下这样的代码: int speed; if (int.TryParse(speedStr, out speed)) speed*=; 注释:int.Tr ...

  7. lock的等效代码

    1.lock的等效代码 在.NET的多线程程序中,经常会遇到lock关键字来控制同步,比如下列代码: private object o = new object(); public void Work ...

  8. ivew数控件Tree自定义节点内容示例分析

    ivew数控件Tree自定义节点内容示例分析 demo地址:https://run.iviewui.com/plcWlM4H <template> <Tree :data=" ...

  9. J.U.C之重入锁:ReentrantLock

    此篇博客所有源码均来自JDK 1.8 ReentrantLock,可重入锁,是一种递归无阻塞的同步机制.它可以等同于synchronized的使用,但是ReentrantLock提供了比synchro ...

  10. 五、eureka客户端自动配置

    所有文章 https://www.cnblogs.com/lay2017/p/11908715.html 正文 前面的几篇文章中,我们从eureka Server端的角度看了看eureka的几个核心要 ...