/**
* Returns a new Dataset that has exactly `numPartitions` partitions, when the fewer partitions
* are requested. If a larger number of partitions is requested, it will stay at the current
* number of partitions. Similar to coalesce defined on an `RDD`, this operation results in
* a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not
* be a shuffle, instead each of the 100 new partitions will claim 10 of the current partitions.
*
* However, if you're doing a drastic coalesce, e.g. to numPartitions = 1,
* this may result in your computation taking place on fewer nodes than
* you like (e.g. one node in the case of numPartitions = 1). To avoid this,
* you can call repartition. This will add a shuffle step, but means the
* current upstream partitions will be executed in parallel (per whatever
* the current partitioning is).
*
* @group typedrel
* @since 1.6.0
*/
def coalesce(numPartitions: Int): Dataset[T] = withTypedPlan {
Repartition(numPartitions, shuffle = false, planWithBarrier)
}

关于coalsece:

1、用于减少分区数量,如果设置的numPartitions超过目前实际有的分区数,则分区数保持不变。

2、窄依赖,不会发生shuffle

3、极端的coalsece可能会影响性能,比如coalsece(1),则只会在一个节点上运行单个任务。这种情况下建议使用repartition,

虽然repartition会发生shuffle,但是repartition对上游的计算,还是多分区并行执行的。

4、应用场景:多用于对一个大数据集filter以后,执行coalsece

  /**
* Returns a new Dataset that has exactly `numPartitions` partitions.
*
* @group typedrel
* @since 1.6.0
*/
def repartition(numPartitions: Int): Dataset[T] = withTypedPlan {
Repartition(numPartitions, shuffle = true, planWithBarrier)
}

关于repartition:

跟coalsece一样,都是用于明确设置多少个分区,但是repartition是个宽依赖,会发生shuffle。主要注意的地方就是repartition不影响上游计算的分区

如果想极端的控制生成的文件数量来避免太多的小文件,建议repartition

测试:对一个大表查询,将查询的结果写到一张表

coalsece测试,coalsece(100),可以看到只有一个stage,并且并行度是100

repartition测试,repartition(100),发生了shuffle。两个stage,stage0对大表指定条件查询,对应的并行度默认是大表的数据量/128M,在repartition将结果输出到表的时候并行度为我们设置的repartition(100),然后shuffle数据,最后输出


  /**
* Return a new RDD that has exactly numPartitions partitions.
*
* Can increase or decrease the level of parallelism in this RDD. Internally, this uses
* a shuffle to redistribute data.
*
* If you are decreasing the number of partitions in this RDD, consider using `coalesce`,
* which can avoid performing a shuffle.
*
* TODO Fix the Shuffle+Repartition data loss issue described in SPARK-23207.
*/
def repartition(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = withScope {
coalesce(numPartitions, shuffle = true)
} /**
* Return a new RDD that is reduced into `numPartitions` partitions.
*
* This results in a narrow dependency, e.g. if you go from 1000 partitions
* to 100 partitions, there will not be a shuffle, instead each of the 100
* new partitions will claim 10 of the current partitions. If a larger number
* of partitions is requested, it will stay at the current number of partitions.
*
* However, if you're doing a drastic coalesce, e.g. to numPartitions = 1,
* this may result in your computation taking place on fewer nodes than
* you like (e.g. one node in the case of numPartitions = 1). To avoid this,
* you can pass shuffle = true. This will add a shuffle step, but means the
* current upstream partitions will be executed in parallel (per whatever
* the current partitioning is).
*
* @note With shuffle = true, you can actually coalesce to a larger number
* of partitions. This is useful if you have a small number of partitions,
* say 100, potentially with a few partitions being abnormally large. Calling
* coalesce(1000, shuffle = true) will result in 1000 partitions with the
* data distributed using a hash partitioner. The optional partition coalescer
* passed in must be serializable.
*/
def coalesce(numPartitions: Int, shuffle: Boolean = false,
partitionCoalescer: Option[PartitionCoalescer] = Option.empty)
(implicit ord: Ordering[T] = null)
: RDD[T] = withScope {
require(numPartitions > 0, s"Number of partitions ($numPartitions) must be positive.")
if (shuffle) {
/** Distributes elements evenly across output partitions, starting from a random partition. */
val distributePartition = (index: Int, items: Iterator[T]) => {
var position = new Random(hashing.byteswap32(index)).nextInt(numPartitions)
items.map { t =>
// Note that the hash code of the key will just be the key itself. The HashPartitioner
// will mod it with the number of total partitions.
position = position + 1
(position, t)
}
} : Iterator[(Int, T)] // include a shuffle step so that our upstream tasks are still distributed
new CoalescedRDD(
new ShuffledRDD[Int, T, T](mapPartitionsWithIndex(distributePartition),
new HashPartitioner(numPartitions)),
numPartitions,
partitionCoalescer).values
} else {
new CoalescedRDD(this, numPartitions, partitionCoalescer)
}
}

dataset的reparation和coalesce的更多相关文章

  1. Spark编程指南分享

    转载自:https://www.2cto.com/kf/201604/497083.html 1.概述 在高层的角度上看,每一个Spark应用都有一个驱动程序(driver program).驱动程序 ...

  2. Spark学习笔记之SparkRDD

    Spark学习笔记之SparkRDD 一.   基本概念 RDD(resilient distributed datasets)弹性分布式数据集. 来自于两方面 ①   内存集合和外部存储系统 ②   ...

  3. Spark-RDD之Partition源码分析

    概要 Spark RDD主要由Dependency.Partition.Partitioner组成,Partition是其中之一.一份待处理的原始数据会被按照相应的逻辑(例如jdbc和hdfs的spl ...

  4. spark(二)

    一.spark的提交模式 --master(standalone\YRAN\mesos) standalone:-client -cluster  如果我们用client模式去提交程序,我们在哪个地方 ...

  5. Spark:JavaRDD 转化为 Dataset<Row>的两种方案

    JavaRDD 转化为 Dataset<Row>方案一: 实体类作为schema定义规范,使用反射,实现JavaRDD转化为Dataset<Row> Student.java实 ...

  6. Spark入门之DataFrame/DataSet

    目录 Part I. Gentle Overview of Big Data and Spark Overview 1.基本架构 2.基本概念 3.例子(可跳过) Spark工具箱 1.Dataset ...

  7. Update(Stage4):sparksql:第3节 Dataset (DataFrame) 的基础操作 & 第4节 SparkSQL_聚合操作_连接操作

    8. Dataset (DataFrame) 的基础操作 8.1. 有类型操作 8.2. 无类型转换 8.5. Column 对象 9. 缺失值处理 10. 聚合 11. 连接 8. Dataset ...

  8. SQL Server-分页方式、ISNULL与COALESCE性能分析(八)

    前言 上一节我们讲解了数据类型以及字符串中几个需要注意的地方,这节我们继续讲讲字符串行数同时也讲其他内容和穿插的内容,简短的内容,深入的讲解,Always to review the basics. ...

  9. HTML5 数据集属性dataset

    有时候在HTML元素上绑定一些额外信息,特别是JS选取操作这些元素时特别有帮助.通常我们会使用getAttribute()和setAttribute()来读和写非标题属性的值.但为此付出的代价是文档将 ...

随机推荐

  1. NAT-T和PAT(IPSec)

    ¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥NAT-T技术介绍¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥ 为什么TCP和UDP不能穿越:TCP和UDP有一个IP头的尾部校验(校验头部和负载 ...

  2. 页面阻止某个按键的键盘事件(event.stopPropagation与event.preventDefault的区别及如何兼容IE8)

    http://blog.csdn.net/rocklee/article/details/53160656

  3. [2/100] MySQL在Windows下安装及一些问题

    mysql 是RDBMS(关系型数据库) 其他: redis 一般做缓存用 mangoDB 一般做爬虫用 国内镜像下载地址: http://mirrors.sohu.com/mysql/MySQL-8 ...

  4. Android 调用系统Email发送带多附件的邮件

    转自:http://www.open-open.com/lib/view/open1347005126912.html 众所周知,在Android中调用其他程序进行相关处理,都是使用的Intent.当 ...

  5. Python3.5学习之旅——day2

    本节内容: 1.模块初识 2..pyc是什么? 3.Python的数据类型 4.三元运算 5.进制 6.byte类型 7.数据运算 8.列表 9.元组 10.课后练习 一.模块初识 由day1的学习我 ...

  6. C++代码如何附加到C#写的主程序中?

    背景是这样:C#的exe程序,C#中调用C++的dll,也就是所谓的托管吧. C#的exe出了点问题,想在C++的dll的源码中调试,附加到进程时,加了断点,怎么也跟不进去.断点不变红啊,急死了. 最 ...

  7. linux关闭进程

    查看端口对应的pid lsof -i:8888 若要关闭使用这个端口的程序,使用kill + 对应的pid kill -9 PID号

  8. 去重sort|uniq -d

    #!/bin/bash ############################################################### #Author :Bing # #Create ...

  9. JQuery 实现PPT效果,点跳目标页及翻页(待改进)

    实现PPT效果 <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> < ...

  10. 【PAT甲级】1036 Boys vs Girls (25 分)

    题意: 输入一个正整数N(题干没指出范围,默认1e5可以AC),接下来输入N行数据,每行包括一名学生的姓名,性别,学号和分数.输出三行,分别为最高分女性学生的姓名和学号,最低分男性学生的姓名和学号,前 ...