AAS代码运行-第11章-2

hdfs dfs -ls /user/littlesuccess/AdvancedAnalysisWithSpark
hdfs dfs -mkdir /user/littlesuccess/AdvancedAnalysisWithSpark/ch11
hdfs dfs -put fish.py /user/littlesuccess/AdvancedAnalysisWithSpark/ch11

做好上述准备工作之后，就可以运行pyspark代码了:

raw_data = sc.textFile('hdfs://172.31.25.243:8020/user/littlesuccess/AdvancedAnalysisWithSpark/ch11/fish.py')
data = (raw_data.filter(lambda x: x.startswith("#")).map(lambda x: map(float, x.split(',')))) 
data.take(5)

运行过程中发现了一个错误：

>>> data.take()
Traceback (most recent call last):
  File "<stdin>", line , in <module>
  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/spark/python/pyspark/rdd.py", line , in take
    totalParts = self._jrdd.partitions().size()
  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line , in __call__
  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line , in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o31.partitions.
: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
    at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:)
    at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getFileInfo(AuthorizationProviderProxyClientProtocol.java:)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
    

发现原因在于我的集群设置了NameNode HA,而我的脚本中的hdfs用的是StandBy NameNode的地址，这个问题就解决了。

重新运行命令，又发现如下错误:

// :: WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID , ip----.us-west-.compute.internal): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/jars/spark-assembly-1.2.0-cdh5.3.3-hadoop2.5.0-cdh5.3.3.jar/pyspark/worker.py", line , in main
    process()
  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/jars/spark-assembly-1.2.0-cdh5.3.3-hadoop2.5.0-cdh5.3.3.jar/pyspark/worker.py", line , in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/jars/spark-assembly-1.2.0-cdh5.3.3-hadoop2.5.0-cdh5.3.3.jar/pyspark/serializers.py", line , in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/spark/python/pyspark/rdd.py", line , in takeUpToNumLeft
    while taken < left:
ImportError: No module named iter

    at org.apache.spark.api.python.PythonRDD$$anon$.read(PythonRDD.scala:)
    at org.apache.spark.api.python.PythonRDD$$anon$.<init>(PythonRDD.scala:)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)
    at org.apache.spark.scheduler.Task.run(Task.scala:)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
    at java.lang.Thread.run(Thread.java:)