Ignite与Spark集成时，ClassNotFoundException问题解决

参考文章：https://apacheignite-fs.readme.io/docs/installation-deployment

Spark application deployment model allows dynamic jar distribution during application start. This model, however, has some drawbacks:

• Spark dynamic class loader does not implement getResource methods, so you will not be able to access resources located in jar files.
• Java logger uses application class loader (not the context class loader) to load log handlers which results in ClassNotFoundException when using Java logging in Ignite.

There is a way to alter default Spark classpath for each launched application (this should be done on each machine of the Spark cluster, including master, worker and driver nodes).

1. Locate the $SPARK_HOME/conf/spark-env.sh file. If this file does not exist, create it from template using $SPARK_HOME/conf/spark-env.sh.template
2. Add the following lines to the end of the spark-env.sh file (uncomment the line settingIGNITE_HOME in case if you do not have it globally set):

# Optionally set IGNITE_HOME here.
# IGNITE_HOME=/path/to/ignite

IGNITE_LIBS="${IGNITE_HOME}/libs/*"

for file in ${IGNITE_HOME}/libs/*
do
    if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ]; then
        IGNITE_LIBS=${IGNITE_LIBS}:${file}/*
    fi
done

export SPARK_CLASSPATH=$IGNITE_LIBS

You can verify that the Spark classpath is changed by running bin/spark-shell and typing a simple import statement:

scala> import org.apache.ignite.configuration._
import org.apache.ignite.configuration._

官方给出的以上解决方案，在spark2.1上是可以的，但是在spark2.2中不起作用，对比了一下spark2.1和spark2.2中的spark-env.sh脚本，发现其中一些变量，在spark2.2中不存在了：