IDEA 开发环境中 调试Spark SQL及遇到问题解决办法

1.问题

java.lang.OutOfMemoryError: PermGen space

java.lang.OutOfMemoryError: Java heap space

// :: WARN NettyRpcEndpointRef: Error sending message [message = Heartbeat(driver,[Lscala.Tuple2;@631e6c90,BlockManagerId(driver, localhost, ))] in  attempts
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [ seconds]. This timeout is controlled by spark.executor.heartbeatInterval
    at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:)
    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$.applyOrElse(RpcTimeout.scala:)
    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$.applyOrElse(RpcTimeout.scala:)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:)
    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:)
    at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:)
    at org.apache.spark.executor.Executor$$anon$$$anonfun$run$.apply$mcV$sp(Executor.scala:)
    at org.apache.spark.executor.Executor$$anon$$$anonfun$run$.apply(Executor.scala:)
    at org.apache.spark.executor.Executor$$anon$$$anonfun$run$.apply(Executor.scala:)
    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:)
    at org.apache.spark.executor.Executor$$anon$.run(Executor.scala:)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$(ScheduledThreadPoolExecutor.java:)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
    at java.lang.Thread.run(Thread.java:)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [ seconds]
    at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:)
    at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:)
    at scala.concurrent.Await$$anonfun$result$.apply(package.scala:)
    at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:)
    at scala.concurrent.Await$.result(package.scala:)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:)
    ...  more

17/04/17 17:46:36 ERROR TaskSetManager: Task 1 in stage 3.0 failed 1 times; aborting job
Exception in thread "qtp502891368-59" java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 ERROR Utils: uncaught error in thread Spark Context Cleaner, stopping SparkContext
java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 413, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 182499 ms
17/04/17 17:57:36 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
Exception in thread "qtp502891368-62" java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 WARN SingleThreadEventExecutor: Unexpected exception from an event executor:
java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 413)
java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 WARN NettyRpcEnv: Ignored message: true

猜测原因：

Spark对内存的消耗主要分为三部分（即取决于你的应用程序的需求）：

1. 数据集中对象的大小
2. 访问这些对象的内存消耗
3. 垃圾回收GC的消耗

---

由网络或者gc引起,worker或executor没有接收到executor或task的心跳反馈，导致 Executor&Task Lost，这时要提高 spark.network.timeout 的值，根据情况改成300(5min)或更高。

解决办法：

这个问题，需要设置IEDA的JVM参数:  -Xms256m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=256M

若在Linux上命令方式的话：

参考： Hadoop与Spark常用配置参数总结 http://www.tuicool.com/articles/naaAzq2