Spark：相关错误总结

http://blog.csdn.net/pipisorry/article/details/52916307

路径错误

spark FileNotFoundError: [Errno 2] No such file or directory: '~/ENV/spark/./bin/spark-submit'

居然是因为这个引发的错误：SPARK_HOME = ~/ENV/spark应该改成SPARK_HOME = /home/pipi/ENV/spark

之前就不会有这个错误，不知道为啥？

spark.textFile读取用户主目录下的文件出错

org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/home/pipi/files/...SparkLDA/～/files

文件名改成绝对路径就不会出错了，而不能使用主目录标志~。

importerror no module named 'pyspark'

可能出错原因：

1 设置的路径应该在import语句之前

2 pyspark_python路径出错，且如果在pycharm中修改后要重启pycharm才能生效，否则还是原来的路径（可能是因为之前的路径也保存在pycharm解释器中了？）

PYSPARK_PYTHON = /home/pipi/opt/anaconda3/bin/python

Note: 并不用设置py4j的路径。

皮皮blog

系统环境错误

1 在~/.bashrc中设置的系统环境只能在terminal shell下运行spark程序才有效，因为.bashrc is only read for interactive shells.

如果要在当前用户整个系统中都有效（包括pycharm等等IDE），就应该将系统环境变量设置在~/.profile文件中。如果是设置所有用户整个系统，修改/etc/profile或者/etc/environment吧。

2 当然也可以在代码中设置，这样不管环境有没有问题了

# spark environment settings
import sys, os
os.environ['SPARK_HOME'] = conf.get(SECTION, 'SPARK_HOME')
sys.path.append(os.path.join(conf.get(SECTION, 'SPARK_HOME'), 'python'))
os.environ["PYSPARK_PYTHON"] = conf.get(SECTION, 'PYSPARK_PYTHON')
os.environ['SPARK_LOCAL_IP'] = conf.get(SECTION, 'SPARK_LOCAL_IP')
os.environ['JAVA_HOME'] = conf.get(SECTION, 'JAVA_HOME')
os.environ[], x[]]).cache()

lambda函数中的x下标在pyspark不能这样写？
解决：改成corpus ], x[]]).cache()就不报这个错误了，也是醉了。
py4j.protocol.Py4JError
LDA引入自from pyspark.mllib.clustering import LDA
py4j.protocol.Py4JError: An error occurred while calling o39.trainLDAModel. Trace:
py4j.Py4JException: Method trainLDAModel([class org.apache.spark.api.java.JavaRDD, class java.lang.Integer, class java.lang.Integer, class java.util.ArrayList, class java.lang.Double, class java.lang.Integer, class java.lang.Integer, class java.lang.String]) does not exist
原因：lda_model = LDA.train(corpus, maxIterations=max_iter, seed=seed, checkpointInterval=checkin_point_interval, k=K, optimizer=optimizer, docConcentration=[alpha], topicConcentration=beta)
解决：lda_model = LDA.train(corpus, maxIterations=max_iter, seed=seed, checkpointInterval=checkin_point_interval, k=K, optimizer=optimizer, docConcentration=alpha, topicConcentration=beta)
TypeError: Invalid param value given for param "docConcentration".
这个错误与上一个错误相关啊，呵呵了。。。
LDA引入自from pyspark.ml.clustering import LDA
TypeError: Invalid param value given for param "docConcentration".Could not convert 5.0 to list of floats.
原因：lda_model = LDA.train(corpus, maxIterations=max_iter, seed=seed, checkpointInterval=checkin_point_interval, k=K, optimizer=optimizer, docConcentration=alpha, topicConcentration=beta)
解决：lz设置docConcentration和TopicConcentration时，只能使用docConcentration=[value]和TopicConcentration=value两种不同的初始化方式，不知道为啥。。。否则出错。
lda_model = LDA.train(corpus, maxIterations=max_iter, seed=seed, checkpointInterval=checkin_point_interval, k=K, optimizer=optimizer, docConcentration=[alpha], topicConcentration=beta)
spark输出太多warning messages
WARN Executor: 2 block locks were not released by TID =
Lock release errors occur frequently in executor logs
原因：If there are any releasedLocks (after calling BlockManager.releaseAllLocksForTask earlier) and spark.storage.exceptionOnPinLeak is enabled (it is not by default) with no exception having been thrown while the task was running, a SparkException is thrown:
[releasedLocks] block locks were not released by TID = [taskId]:
[releasedLocks separated by comma]
Otherwise, if spark.storage.exceptionOnPinLeak is disabled or an exception was thrown by the task, the following WARN message is displayed in the logs instead:
WARN Executor: [releasedLocks] block locks were not released by TID = [taskId]:
[releasedLocks separated by comma]
Note    If there are any releaseLocks, they lead to a SparkException or WARN message in the logs. 
[jaceklaskowski/mastering-apache-spark-book/spark-executor-taskrunner.adoc]
mapWithState causes block lock warning?
The warning was added by: SPARK-12757 Add block-level read/write locks to BlockManager?
[connectedComponents() raises lots of warnings that say "block locks were not released by TID = ..."]
[Lock release errors occur frequently in executor logs]
解决：终于在调试log时候发现问题解决了
在简略Spark输出设置时[Spark安装和配置]修改过$SPARK_HOME/conf/log4j.properties.template文件只输出WARN信息，就算改成了ERROR，信息也还是会自动修改成WARN输出出来，不过多了一条提示：
Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel).
就在这时发现了一个解决方案：
根据提示在代码中加入sc.setLogLevel('ERROR')就可以解决了！
from: http://blog.csdn.net/pipisorry/article/details/52916307
ref: