As we all know , up to Spark 1.6.2, JavaSparkContext only provides two kinds of accumulators: Integer and Double.

However, unfortunately I've met with problems of Integer overflow and the program returned me a negative number.

So I have to use original sparkcontext to implement the Long accumulator.

public static class LongAccumulatorParam implements AccumulatorParam<Long>,Serializable {
@Override
public Long addAccumulator(final Long r, final Long t) {
return r + t;
}
@Override
public Long addInPlace(final Long r1, final Long r2) {
return r1 + r2;
}
@Override
public Long zero(final Long initialValue) {
return 0L;
}
}
final Accumulator<Long> acc = jsc.sc().accumulator(new Long(0), new LongAccumulatorParam());

Actually it is pretty simple. I haven't looked into Spark 2 yet, hope the developers have fixed this issue.

Accumulator<Long> implements of JavaSparkContext in Spark1.x的更多相关文章

  1. java使用spark/spark-sql处理schema数据(spark1.6)

    1.spark是什么? Spark是基于内存计算的大数据并行计算框架. 1.1 Spark基于内存计算 相比于MapReduce基于IO计算,提高了在大数据环境下数据处理的实时性. 1.2 高容错性和 ...

  2. 【Spark Java API】broadcast、accumulator

    转载自:http://www.jianshu.com/p/082ef79c63c1 broadcast 官方文档描述: Broadcast a read-only variable to the cl ...

  3. spark-2.2.0-bin-hadoop2.6和spark-1.6.1-bin-hadoop2.6发行包自带案例全面详解(java、python、r和scala)之Basic包下的JavaPageRank.java(图文详解)

    不多说,直接上干货! spark-1.6.1-bin-hadoop2.6里Basic包下的JavaPageRank.java /* * Licensed to the Apache Software ...

  4. spark 变量使用 broadcast、accumulator

    broadcast 官方文档描述: Broadcast a read-only variable to the cluster, returning a [[org.apache.spark.broa ...

  5. Spark1.6.2 java实现读取json数据文件插入MySql数据库

    public class Main implements Serializable { /** * */ private static final long serialVersionUID = -8 ...

  6. Spark1.6.2 java实现读取txt文件插入MySql数据库代码

    package com.gosun.spark1; import java.util.ArrayList;import java.util.List;import java.util.Properti ...

  7. flink - accumulator

      读accumlator JobManager 在job finish的时候会汇总accumulator的值, newJobStatus match { case JobStatus.FINISHE ...

  8. spark1.4的本地模式编程练习(1)

    spark编程练习 申明:以下代码仅作学习参考使用,勿使用在商业用途. Wordcount UserMining TweetMining HashtagMining InvertedIndex Tes ...

  9. Spark1.0.x入门指南

    1 节点说明   IP Role 192.168.1.111 ActiveNameNode 192.168.1.112 StandbyNameNode,Master,Worker 192.168.1. ...

随机推荐

  1. [转]Spring Boot修改最大上传文件限制:The field file exceeds its maximum permitted size of 1048576 bytes.

    来源:http://blog.csdn.net/awmw74520/article/details/70230591 SpringBoot做文件上传时出现了The field file exceeds ...

  2. python基础——list和tuple(列表和元组)

    1.list的定义,插入insert,append,按位置索引. >>> name = ['Macal','lily','lucy','bob']  --初始化>>> ...

  3. Spring Boot之 Controller 接收参数和返回数据总结(包括上传、下载文件)

            一.接收参数(postman发送) 1.form表单 @RequestParam("name") String name 会把传递过来的Form表单中的name对应 ...

  4. php 操作数据库

    $datetoday = date('Y-m-d'); $datetime = $thedate; $data_info = $data; $db = array( 'dsn' => 'mysq ...

  5. 最短路径(给定一个包含非负整数的 m x n 网格,请找出一条从左上角到右下角的路径,使得路径上的数字总和为最小。 说明:每次只能向下或者向右移动一步。)

    给定一个包含非负整数的 m x n 网格,请找出一条从左上角到右下角的路径,使得路径上的数字总和为最小. 说明:每次只能向下或者向右移动一步. 例: 输入: [ [1,3,1], [1,5,1], [ ...

  6. redics在windows平台下的使用

    https://www.cnblogs.com/ding2011/p/4745732.html 一::下载    windows 下的redis: https://github.com/Service ...

  7. Python编译安装遇到的问题

    1.python在make时候报错 Python build finished, but the necessary bits to build these modules were not foun ...

  8. TF之RNN:TF的RNN中的常用的两种定义scope的方式get_variable和Variable—Jason niu

    # tensorflow中的两种定义scope(命名变量)的方式tf.get_variable和tf.Variable.Tensorflow当中有两种途径生成变量 variable import te ...

  9. Full Tank? POJ - 3635 (bfs | 最短路)

    After going through the receipts from your car trip through Europe this summer, you realised that th ...

  10. 004.Ansible Ad-Hoc命令集

    一 Ad-Hoc使用场景 Ad-Hoc更倾向于解决简单.临时性任务. 1.1 Ad-Hoc基础命令 基本语法: 1 ansible <host-pattern> [options] < ...