Spark代码Eclipse远程调试

我们在编写Spark Application或者是阅读源码的时候，我们很想知道代码的运行情况，比如参数设置的是否正确等等。用Logging方式来调试是一个可以选择的方式，但是，logging方式调试代码有很多的局限和不便。今天我就来介绍如何通过IDE来远程调试Spark的Application或者是Spark的源码。
　　本文以调试Spark Application为例进行说明，本文用到的IDE是Eclipse。步骤如下：

一、JVM里面设置以下参数

-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=6666

这里对上面的几个参数进行说明：

-Xdebug 启用调试特性
-Xrunjdwp 启用JDWP实现，包含若干子选项：
transport=dt_socket JPDA front-end和back-end之间的传输方法。dt_socket表示使用套接字传输。
address=6666 JVM在6666端口上监听请求，这个设定为一个不冲突的端口即可。
server=y y表示启动的JVM是被调试者。如果为n，则表示启动的JVM是调试器。
suspend=y y表示启动的JVM会暂停等待，直到调试器连接上才继续执行。suspend=n，则JVM不会暂停等待。

二、启动Spark Application

spark-submit提交程序的时候带上

--driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=6666" \

　　启动Spark Application的时候我们可以在终端看到如下输出：

[root@master Spark-Submit]# spark-submit \
> --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=6666" \
> --driver-class-path /usr/local/TrueTimeControlDeploy/myApp/ojdbc5.jar \
> --class 'com.spark.main.CompareAndDistinctResult' /usr/local/TrueTimeControlDeploy/myApp/TrueTimeControlOnSparkByJava-0.0.1-jar-with-dependencies.jar
Listening for transport dt_socket at address: 6666

如果你看到第5行的输出（Listening for transport dt_socket at address: 6666），那恭喜你了，启动了远程调试。而且Spark Application正在等待我们的IDE连接它。

三、在Eclipse设置远程调试的IP和port

双击【Remote Java Application】，如下图：

点击【Select one...】，如下图：

点击Debug，可以Linux客户端Spark-submit程序已经开始运行：

[root@master Spark-Submit]# spark-submit --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=6666" --driver-class-path /usr/local/TrueTimeControlDeploy/myApp/ojdbc5.jar --class 'com.spark.main.CompareAndDistinctResult' /usr/local/TrueTimeControlDeploy/myApp/TrueTimeControlOnSparkByJava-0.0.1-jar-with-dependencies.jar
Listening for transport dt_socket at address: 6666
17/09/22 21:38:40 INFO spark.SparkContext: Running Spark version 2.0.2
17/09/22 21:38:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/09/22 21:38:42 INFO spark.SecurityManager: Changing view acls to: root
17/09/22 21:38:42 INFO spark.SecurityManager: Changing modify acls to: root
17/09/22 21:38:42 INFO spark.SecurityManager: Changing view acls groups to:
17/09/22 21:38:42 INFO spark.SecurityManager: Changing modify acls groups to:
17/09/22 21:38:42 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
17/09/22 21:38:43 INFO util.Utils: Successfully started service 'sparkDriver' on port 51652.
17/09/22 21:38:43 INFO spark.SparkEnv: Registering MapOutputTracker
17/09/22 21:38:43 INFO spark.SparkEnv: Registering BlockManagerMaster
17/09/22 21:38:43 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-e04a2402-1821-49f2-92a6-21e97e1ebb15
17/09/22 21:38:43 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
17/09/22 21:38:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/09/22 21:38:44 INFO util.log: Logging initialized @20047ms
17/09/22 21:38:44 INFO server.Server: jetty-9.2.z-SNAPSHOT
17/09/22 21:38:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1be6d5ac{/jobs,null,AVAILABLE}
17/09/22 21:38:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4fd61d83{/jobs/json,null,AVAILABLE}
17/09/22 21:38:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@722bac67{/jobs/job,null,AVAILABLE}
17/09/22 21:38:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1143d8c0{/jobs/job/json,null,AVAILABLE}
17/09/22 21:38:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7d4b2e1a{/stages,null,AVAILABLE}

然后你可以开始远程调试你的代码了！

注意：远程断点调试只能调试运行在Driver端的代码，运行在Worker端的代码是无法调试，除非使用local模式。

lines.foreachRDD(new VoidFunction<JavaRDD<String>>() {
private static final long serialVersionUID = 340756330712789698L;
@Override
public void call(JavaRDD<String> rdd) throws Exception {
//此时代码在Driver端执行
String addr = InetAddress.getLocalHost().getHostAddress();//获得本机IP
System.out.println(addr);
System.out.println("断点一测试成功！！！");
rdd.foreach(new VoidFunction<String>() {
private static final long serialVersionUID = -1096797241091398607L;
@Override
public void call(String s) throws Exception {
// 这个时候执行的位置已经是在Worker执行了
String addr = InetAddress.getLocalHost().getHostAddress();//获得本机IP
System.out.println(addr);
System.out.println("断点二测试成功！！！");
}
});
}
});