AAS代码运行-第11章-1

启动PySpark

export IPYTHON= # PySpark也可使用IPython shell
pyspark --master yarn --num-executors 

发生如下错误:

/opt/cloudera/parcels/CDH-5.3.-.cdh5.3.3.p0./bin/../lib/spark/bin/pyspark: line : exec: ipython: not found

原因是没有按照ipython，在google所有一下安装的方法，到网站http://continuum.io/downloads#all上下载Linux-64位的Anaconda版本，

wget https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda-2.3.0-Linux-x86_64.sh

下载完成之后，运行如下命令进行安装：

bash Anaconda-2.3.-Linux-x86_64.sh

上述命令会进入一个类似shell的交互式环境，安装过程中都选择yes即可。安装完成之后，在安装ipython,Anaconda在交换式环境中输入:

pip install ipython

这时候会安装一大堆的包。安装完成之后会提示是否修改环境变量，输入yes，安装脚本自动将如下脚本写入~/.bashrc中的尾部

export PATH=/root/anaconda/bin:$PATH

运行source ~/.bashrc使新环境变量生效。

source ~/.bashrc

再次运行pyspark命令即可，如下：

pyspark --master yarn --num-executors 

得到如下输出:

Type "copyright", "credits" or "license" for more information.

IPython 3.2. -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
// :: INFO SecurityManager: Changing view acls to: root
// :: INFO SecurityManager: Changing modify acls to: root
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
// :: INFO Slf4jLogger: Slf4jLogger started
// :: INFO Remoting: Starting remoting
// :: INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@ip-172-31-25-243.us-west-2.compute.internal:50324]
// :: INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@ip-172-31-25-243.us-west-2.compute.internal:50324]
// :: INFO Utils: Successfully started service 'sparkDriver' on port .
// :: INFO SparkEnv: Registering MapOutputTracker
// :: INFO SparkEnv: Registering BlockManagerMaster
// :: INFO DiskBlockManager: Created local directory at /tmp/spark-local--7afc
// :: INFO MemoryStore: MemoryStore started with capacity 265.4 MB
// :: INFO HttpFileServer: HTTP File server directory is /tmp/spark-e2f4c4e3-dc9b-4db0-8fd4-1dcbb8819b05
// :: INFO HttpServer: Starting HTTP Server
// :: INFO Utils: Successfully started service 'HTTP file server' on port .
// :: INFO Utils: Successfully started service 'SparkUI' on port .
// :: INFO SparkUI: Started SparkUI at http://ip-172-31-25-243.us-west-2.compute.internal:4040
// :: INFO RMProxy: Connecting to ResourceManager at ip----.us-west-.compute.internal/172.31.25.243:
// :: INFO Client: Requesting a new application from cluster with  NodeManagers
// :: INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster ( MB per container)
// :: INFO Client: Will allocate AM container, with  MB memory including  MB overhead
// :: INFO Client: Setting up container launch context for our AM
// :: INFO Client: Preparing resources for our AM container
// :: INFO Client: Setting up the launch environment for our AM container
// :: INFO SecurityManager: Changing view acls to: root
// :: INFO SecurityManager: Changing modify acls to: root
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
// :: INFO Client: Submitting application  to ResourceManager
// :: INFO YarnClientImpl: Submitted application application_1436008024626_0001
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client:
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -
     queue: root.root
     start time:
     final status: UNDEFINED
     tracking URL: http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001/
     user: root
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka.tcp://sparkYarnAM@ip-172-31-25-246.us-west-2.compute.internal:45245/user/YarnAM#-1546613765]
// :: INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> ip----.us-west-.compute.internal, PROXY_URI_BASES -> http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001), /proxy/application_1436008024626_0001
// :: INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
// :: INFO Client: Application report for application_1436008024626_0001 (state: RUNNING)
// :: INFO Client:
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: ip----.us-west-.compute.internal
     ApplicationMaster RPC port:
     queue: root.root
     start time:
     final status: UNDEFINED
     tracking URL: http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001/
     user: root
// :: INFO YarnClientSchedulerBackend: Application application_1436008024626_0001 has started running.
// :: INFO NettyBlockTransferService: Server created on
// :: INFO BlockManagerMaster: Trying to register BlockManager
// :: INFO BlockManagerMasterActor: Registering block manager ip----.us-west-.compute.internal: with 265.4 MB RAM, BlockManagerId(<driver>, ip----.us-west-.compute.internal, )
// :: INFO BlockManagerMaster: Registered BlockManager
// :: INFO EventLoggingListener: Logging events to hdfs://ns-ha/user/spark/applicationHistory/application_1436008024626_0001
// :: INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-31-25-245.us-west-2.compute.internal:43360/user/Executor#1565208330] with ID 1
// :: INFO RackResolver: Resolved ip----.us-west-.compute.internal to /default
// :: INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-31-25-244.us-west-2.compute.internal:39613/user/Executor#-786460899] with ID 2
// :: INFO RackResolver: Resolved ip----.us-west-.compute.internal to /default
// :: INFO BlockManagerMasterActor: Registering block manager ip----.us-west-.compute.internal: with 530.3 MB RAM, BlockManagerId(, ip----.us-west-.compute.internal, )
// :: INFO BlockManagerMasterActor: Registering block manager ip----.us-west-.compute.internal: with 530.3 MB RAM, BlockManagerId(, ip----.us-west-.compute.internal, )
// :: INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: (ms)
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.2.-SNAPSHOT
      /_/

Using Python version 2.7. (default, May   ::)
SparkContext available as sc.

In []: