启动PySpark

export IPYTHON= # PySpark也可使用IPython shell
pyspark --master yarn --num-executors

发生如下错误:

/opt/cloudera/parcels/CDH-5.3.-.cdh5.3.3.p0./bin/../lib/spark/bin/pyspark: line : exec: ipython: not found

原因是没有按照ipython,在google所有一下安装的方法,到网站http://continuum.io/downloads#all上下载Linux-64位的Anaconda版本,

wget https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda-2.3.0-Linux-x86_64.sh

下载完成之后,运行如下命令进行安装:

bash Anaconda-2.3.-Linux-x86_64.sh

上述命令会进入一个类似shell的交互式环境,安装过程中都选择yes即可。安装完成之后,在安装ipython,Anaconda在交换式环境中输入:

pip install ipython

这时候会安装一大堆的包。安装完成之后会提示是否修改环境变量,输入yes,安装脚本自动将如下脚本写入~/.bashrc中的尾部

export PATH=/root/anaconda/bin:$PATH

运行source ~/.bashrc使新环境变量生效。

source ~/.bashrc

再次运行pyspark命令即可,如下:

pyspark --master yarn --num-executors 

得到如下输出:

Type "copyright", "credits" or "license" for more information.

IPython 3.2. -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
// :: INFO SecurityManager: Changing view acls to: root
// :: INFO SecurityManager: Changing modify acls to: root
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
// :: INFO Slf4jLogger: Slf4jLogger started
// :: INFO Remoting: Starting remoting
// :: INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@ip-172-31-25-243.us-west-2.compute.internal:50324]
// :: INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@ip-172-31-25-243.us-west-2.compute.internal:50324]
// :: INFO Utils: Successfully started service 'sparkDriver' on port .
// :: INFO SparkEnv: Registering MapOutputTracker
// :: INFO SparkEnv: Registering BlockManagerMaster
// :: INFO DiskBlockManager: Created local directory at /tmp/spark-local--7afc
// :: INFO MemoryStore: MemoryStore started with capacity 265.4 MB
// :: INFO HttpFileServer: HTTP File server directory is /tmp/spark-e2f4c4e3-dc9b-4db0-8fd4-1dcbb8819b05
// :: INFO HttpServer: Starting HTTP Server
// :: INFO Utils: Successfully started service 'HTTP file server' on port .
// :: INFO Utils: Successfully started service 'SparkUI' on port .
// :: INFO SparkUI: Started SparkUI at http://ip-172-31-25-243.us-west-2.compute.internal:4040
// :: INFO RMProxy: Connecting to ResourceManager at ip----.us-west-.compute.internal/172.31.25.243:
// :: INFO Client: Requesting a new application from cluster with NodeManagers
// :: INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster ( MB per container)
// :: INFO Client: Will allocate AM container, with MB memory including MB overhead
// :: INFO Client: Setting up container launch context for our AM
// :: INFO Client: Preparing resources for our AM container
// :: INFO Client: Setting up the launch environment for our AM container
// :: INFO SecurityManager: Changing view acls to: root
// :: INFO SecurityManager: Changing modify acls to: root
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
// :: INFO Client: Submitting application to ResourceManager
// :: INFO YarnClientImpl: Submitted application application_1436008024626_0001
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -
queue: root.root
start time:
final status: UNDEFINED
tracking URL: http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001/
user: root
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka.tcp://sparkYarnAM@ip-172-31-25-246.us-west-2.compute.internal:45245/user/YarnAM#-1546613765]
// :: INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> ip----.us-west-.compute.internal, PROXY_URI_BASES -> http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001), /proxy/application_1436008024626_0001
// :: INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
// :: INFO Client: Application report for application_1436008024626_0001 (state: RUNNING)
// :: INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: ip----.us-west-.compute.internal
ApplicationMaster RPC port:
queue: root.root
start time:
final status: UNDEFINED
tracking URL: http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001/
user: root
// :: INFO YarnClientSchedulerBackend: Application application_1436008024626_0001 has started running.
// :: INFO NettyBlockTransferService: Server created on
// :: INFO BlockManagerMaster: Trying to register BlockManager
// :: INFO BlockManagerMasterActor: Registering block manager ip----.us-west-.compute.internal: with 265.4 MB RAM, BlockManagerId(<driver>, ip----.us-west-.compute.internal, )
// :: INFO BlockManagerMaster: Registered BlockManager
// :: INFO EventLoggingListener: Logging events to hdfs://ns-ha/user/spark/applicationHistory/application_1436008024626_0001
// :: INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-31-25-245.us-west-2.compute.internal:43360/user/Executor#1565208330] with ID 1
// :: INFO RackResolver: Resolved ip----.us-west-.compute.internal to /default
// :: INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-31-25-244.us-west-2.compute.internal:39613/user/Executor#-786460899] with ID 2
// :: INFO RackResolver: Resolved ip----.us-west-.compute.internal to /default
// :: INFO BlockManagerMasterActor: Registering block manager ip----.us-west-.compute.internal: with 530.3 MB RAM, BlockManagerId(, ip----.us-west-.compute.internal, )
// :: INFO BlockManagerMasterActor: Registering block manager ip----.us-west-.compute.internal: with 530.3 MB RAM, BlockManagerId(, ip----.us-west-.compute.internal, )
// :: INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: (ms)
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 1.2.-SNAPSHOT
/_/ Using Python version 2.7. (default, May ::)
SparkContext available as sc. In []:

AAS代码运行-第11章-1的更多相关文章

  1. AAS代码运行-第11章-2

    hdfs dfs -ls /user/littlesuccess/AdvancedAnalysisWithSparkhdfs dfs -mkdir /user/littlesuccess/Advanc ...

  2. AAS代码运行-第4章

    [root@node1 aas]# ls ch02 ch03 spark--bin-hadoop2. spark--bin-hadoop2..tgz [root@node1 aas]# cd spar ...

  3. 第11章 Windows线程池(1)_传统的Windows线程池

    第11章 Windows线程池 11.1 传统的Windows线程池及API (1)线程池中的几种底层线程 ①可变数量的长任务线程:WT_EXECUTELONGFUNCTION ②Timer线程:调用 ...

  4. 高性能Linux服务器 第11章 构建高可用的LVS负载均衡集群

    高性能Linux服务器 第11章 构建高可用的LVS负载均衡集群 libnet软件包<-依赖-heartbeat(包含ldirectord插件(需要perl-MailTools的rpm包)) l ...

  5. Linux就这个范儿 第11章 独霸网络的蜘蛛神功

    Linux就这个范儿 第11章  独霸网络的蜘蛛神功  第11章 应用层 (Application):网络服务与最终用户的一个接口.协议有:HTTP FTP TFTP SMTP SNMP DNS表示层 ...

  6. 第11章 享元模式(Flyweight Pattern)

    原文 第11章 享元模式(Flyweight Pattern) 概述:   面向对象的思想很好地解决了抽象性的问题,一般也不会出现性能上的问题.但是在某些情况下,对象的数量可能会太多,从而导致了运行时 ...

  7. 翻译连载 | 第 11 章:融会贯通 -《JavaScript轻量级函数式编程》 |《你不知道的JS》姊妹篇

    原文地址:Functional-Light-JS 原文作者:Kyle Simpson-<You-Dont-Know-JS>作者 关于译者:这是一个流淌着沪江血液的纯粹工程:认真,是 HTM ...

  8. 【STM32H7教程】第11章 STM32H7移植SEGGER的硬件异常分析

    完整教程下载地址:http://forum.armfly.com/forum.php?mod=viewthread&tid=86980 第11章       STM32H7移植SEGGER的硬 ...

  9. Java核心技术卷一基础知识-第11章-异常、断言、日志和调试-读书笔记

    第11章 异常.断言.日志和调试 本章内容: * 处理错误 * 捕获异常 * 使用异常机制的技巧 * 使用断言 * 日志 * 调试技巧 * GUI程序排错技巧 * 使用调试器 11.1 处理错误 如果 ...

随机推荐

  1. SQL 个版本下载地址

    备用: SQL Server 2016简体中文企业版 文件名:cn_sql_server_2016_enterprise_x64_dvd_8699450.iso 64位下载地址:ed2k://|fil ...

  2. asp.net Forms身份验证

    Web.config中的配置<system.web><authentication mode="Forms"> <forms name="K ...

  3. LVS

    1.LVS-NAT, DNAT(多目标) 2.LVS-DR(Direct Routing) 返回报文不经过Direct real server  不能跨越路由 调度算法:Scheduling 静态方法 ...

  4. 无法启动调试。未安装Silverlight Developer运行时。最新运行时可以从以下地址下载: http://go.microsoft.com/fwlink/?LinkId=146060.

    无法启动调试.未安装Silverlight Developer运行时.最新运行时可以从以下地址下载: http://go.microsoft.com/fwlink/?LinkId=146060. 解决 ...

  5. 如何在自己的窗体(控件)中显示XAF的视图

    Form form = new Form(); DevExpress.ExpressApp.View listView = Application.CreateListView(Application ...

  6. iOS自动化编译方案

    本文主要来源以下Bryce Zhang博主的文章,感谢博主的无私分享,转载请注明出处,尊重原创 然,根据Bryce Zhang文章进行实践过程中遇到一些问题,解决后在此做相应的总结.大神请绕道,觉得低 ...

  7. 各种文件的mime类型

    扩展名:abs MIME类型:audio/x-mpeg 扩展名:ai MIME类型:application/postscript 扩展名:aif MIME类型:audio/x-aiff 扩展名:aif ...

  8. Spring MVC Rest服务 返回json报406错误的解决办法

    @ResponseBody & @RequestBody @RequestBody 将 HTTP 请求正文插入方法中,使用适合的HttpMessageConverter将请求体写入某个对象. ...

  9. 转载ali面

    引言 首先声明,不要再问LZ谁是林萧,林萧就是某著名程序员小说的主角名字. 写这篇文章的目的其实很简单,算是对之前LZ一篇文章的补充和完善. 之前LZ写过一篇<回答阿里社招面试如何准备,顺便谈谈 ...

  10. javascript学习内容--改变样式、取消设置、显示内容、隐藏内容

    <head> <style> body{font-size:12px;} #txt{ height:400px; width:600px; border:#333 solid ...