AAS代码运行-第11章-1
启动PySpark
export IPYTHON= # PySpark也可使用IPython shell
pyspark --master yarn --num-executors
发生如下错误:
/opt/cloudera/parcels/CDH-5.3.-.cdh5.3.3.p0./bin/../lib/spark/bin/pyspark: line : exec: ipython: not found
原因是没有按照ipython,在google所有一下安装的方法,到网站http://continuum.io/downloads#all上下载Linux-64位的Anaconda版本,
wget https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda-2.3.0-Linux-x86_64.sh
下载完成之后,运行如下命令进行安装:
bash Anaconda-2.3.-Linux-x86_64.sh
上述命令会进入一个类似shell的交互式环境,安装过程中都选择yes即可。安装完成之后,在安装ipython,Anaconda在交换式环境中输入:
pip install ipython
这时候会安装一大堆的包。安装完成之后会提示是否修改环境变量,输入yes,安装脚本自动将如下脚本写入~/.bashrc中的尾部
export PATH=/root/anaconda/bin:$PATH
运行source ~/.bashrc使新环境变量生效。
source ~/.bashrc
再次运行pyspark命令即可,如下:
pyspark --master yarn --num-executors
得到如下输出:
Type "copyright", "credits" or "license" for more information. IPython 3.2. -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
// :: INFO SecurityManager: Changing view acls to: root
// :: INFO SecurityManager: Changing modify acls to: root
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
// :: INFO Slf4jLogger: Slf4jLogger started
// :: INFO Remoting: Starting remoting
// :: INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@ip-172-31-25-243.us-west-2.compute.internal:50324]
// :: INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@ip-172-31-25-243.us-west-2.compute.internal:50324]
// :: INFO Utils: Successfully started service 'sparkDriver' on port .
// :: INFO SparkEnv: Registering MapOutputTracker
// :: INFO SparkEnv: Registering BlockManagerMaster
// :: INFO DiskBlockManager: Created local directory at /tmp/spark-local--7afc
// :: INFO MemoryStore: MemoryStore started with capacity 265.4 MB
// :: INFO HttpFileServer: HTTP File server directory is /tmp/spark-e2f4c4e3-dc9b-4db0-8fd4-1dcbb8819b05
// :: INFO HttpServer: Starting HTTP Server
// :: INFO Utils: Successfully started service 'HTTP file server' on port .
// :: INFO Utils: Successfully started service 'SparkUI' on port .
// :: INFO SparkUI: Started SparkUI at http://ip-172-31-25-243.us-west-2.compute.internal:4040
// :: INFO RMProxy: Connecting to ResourceManager at ip----.us-west-.compute.internal/172.31.25.243:
// :: INFO Client: Requesting a new application from cluster with NodeManagers
// :: INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster ( MB per container)
// :: INFO Client: Will allocate AM container, with MB memory including MB overhead
// :: INFO Client: Setting up container launch context for our AM
// :: INFO Client: Preparing resources for our AM container
// :: INFO Client: Setting up the launch environment for our AM container
// :: INFO SecurityManager: Changing view acls to: root
// :: INFO SecurityManager: Changing modify acls to: root
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
// :: INFO Client: Submitting application to ResourceManager
// :: INFO YarnClientImpl: Submitted application application_1436008024626_0001
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -
queue: root.root
start time:
final status: UNDEFINED
tracking URL: http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001/
user: root
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
// :: INFO YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka.tcp://sparkYarnAM@ip-172-31-25-246.us-west-2.compute.internal:45245/user/YarnAM#-1546613765]
// :: INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> ip----.us-west-.compute.internal, PROXY_URI_BASES -> http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001), /proxy/application_1436008024626_0001
// :: INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
// :: INFO Client: Application report for application_1436008024626_0001 (state: RUNNING)
// :: INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: ip----.us-west-.compute.internal
ApplicationMaster RPC port:
queue: root.root
start time:
final status: UNDEFINED
tracking URL: http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001/
user: root
// :: INFO YarnClientSchedulerBackend: Application application_1436008024626_0001 has started running.
// :: INFO NettyBlockTransferService: Server created on
// :: INFO BlockManagerMaster: Trying to register BlockManager
// :: INFO BlockManagerMasterActor: Registering block manager ip----.us-west-.compute.internal: with 265.4 MB RAM, BlockManagerId(<driver>, ip----.us-west-.compute.internal, )
// :: INFO BlockManagerMaster: Registered BlockManager
// :: INFO EventLoggingListener: Logging events to hdfs://ns-ha/user/spark/applicationHistory/application_1436008024626_0001
// :: INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-31-25-245.us-west-2.compute.internal:43360/user/Executor#1565208330] with ID 1
// :: INFO RackResolver: Resolved ip----.us-west-.compute.internal to /default
// :: INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-31-25-244.us-west-2.compute.internal:39613/user/Executor#-786460899] with ID 2
// :: INFO RackResolver: Resolved ip----.us-west-.compute.internal to /default
// :: INFO BlockManagerMasterActor: Registering block manager ip----.us-west-.compute.internal: with 530.3 MB RAM, BlockManagerId(, ip----.us-west-.compute.internal, )
// :: INFO BlockManagerMasterActor: Registering block manager ip----.us-west-.compute.internal: with 530.3 MB RAM, BlockManagerId(, ip----.us-west-.compute.internal, )
// :: INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: (ms)
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 1.2.-SNAPSHOT
/_/ Using Python version 2.7. (default, May ::)
SparkContext available as sc. In []:
AAS代码运行-第11章-1的更多相关文章
- AAS代码运行-第11章-2
hdfs dfs -ls /user/littlesuccess/AdvancedAnalysisWithSparkhdfs dfs -mkdir /user/littlesuccess/Advanc ...
- AAS代码运行-第4章
[root@node1 aas]# ls ch02 ch03 spark--bin-hadoop2. spark--bin-hadoop2..tgz [root@node1 aas]# cd spar ...
- 第11章 Windows线程池(1)_传统的Windows线程池
第11章 Windows线程池 11.1 传统的Windows线程池及API (1)线程池中的几种底层线程 ①可变数量的长任务线程:WT_EXECUTELONGFUNCTION ②Timer线程:调用 ...
- 高性能Linux服务器 第11章 构建高可用的LVS负载均衡集群
高性能Linux服务器 第11章 构建高可用的LVS负载均衡集群 libnet软件包<-依赖-heartbeat(包含ldirectord插件(需要perl-MailTools的rpm包)) l ...
- Linux就这个范儿 第11章 独霸网络的蜘蛛神功
Linux就这个范儿 第11章 独霸网络的蜘蛛神功 第11章 应用层 (Application):网络服务与最终用户的一个接口.协议有:HTTP FTP TFTP SMTP SNMP DNS表示层 ...
- 第11章 享元模式(Flyweight Pattern)
原文 第11章 享元模式(Flyweight Pattern) 概述: 面向对象的思想很好地解决了抽象性的问题,一般也不会出现性能上的问题.但是在某些情况下,对象的数量可能会太多,从而导致了运行时 ...
- 翻译连载 | 第 11 章:融会贯通 -《JavaScript轻量级函数式编程》 |《你不知道的JS》姊妹篇
原文地址:Functional-Light-JS 原文作者:Kyle Simpson-<You-Dont-Know-JS>作者 关于译者:这是一个流淌着沪江血液的纯粹工程:认真,是 HTM ...
- 【STM32H7教程】第11章 STM32H7移植SEGGER的硬件异常分析
完整教程下载地址:http://forum.armfly.com/forum.php?mod=viewthread&tid=86980 第11章 STM32H7移植SEGGER的硬 ...
- Java核心技术卷一基础知识-第11章-异常、断言、日志和调试-读书笔记
第11章 异常.断言.日志和调试 本章内容: * 处理错误 * 捕获异常 * 使用异常机制的技巧 * 使用断言 * 日志 * 调试技巧 * GUI程序排错技巧 * 使用调试器 11.1 处理错误 如果 ...
随机推荐
- SQL 个版本下载地址
备用: SQL Server 2016简体中文企业版 文件名:cn_sql_server_2016_enterprise_x64_dvd_8699450.iso 64位下载地址:ed2k://|fil ...
- asp.net Forms身份验证
Web.config中的配置<system.web><authentication mode="Forms"> <forms name="K ...
- LVS
1.LVS-NAT, DNAT(多目标) 2.LVS-DR(Direct Routing) 返回报文不经过Direct real server 不能跨越路由 调度算法:Scheduling 静态方法 ...
- 无法启动调试。未安装Silverlight Developer运行时。最新运行时可以从以下地址下载: http://go.microsoft.com/fwlink/?LinkId=146060.
无法启动调试.未安装Silverlight Developer运行时.最新运行时可以从以下地址下载: http://go.microsoft.com/fwlink/?LinkId=146060. 解决 ...
- 如何在自己的窗体(控件)中显示XAF的视图
Form form = new Form(); DevExpress.ExpressApp.View listView = Application.CreateListView(Application ...
- iOS自动化编译方案
本文主要来源以下Bryce Zhang博主的文章,感谢博主的无私分享,转载请注明出处,尊重原创 然,根据Bryce Zhang文章进行实践过程中遇到一些问题,解决后在此做相应的总结.大神请绕道,觉得低 ...
- 各种文件的mime类型
扩展名:abs MIME类型:audio/x-mpeg 扩展名:ai MIME类型:application/postscript 扩展名:aif MIME类型:audio/x-aiff 扩展名:aif ...
- Spring MVC Rest服务 返回json报406错误的解决办法
@ResponseBody & @RequestBody @RequestBody 将 HTTP 请求正文插入方法中,使用适合的HttpMessageConverter将请求体写入某个对象. ...
- 转载ali面
引言 首先声明,不要再问LZ谁是林萧,林萧就是某著名程序员小说的主角名字. 写这篇文章的目的其实很简单,算是对之前LZ一篇文章的补充和完善. 之前LZ写过一篇<回答阿里社招面试如何准备,顺便谈谈 ...
- javascript学习内容--改变样式、取消设置、显示内容、隐藏内容
<head> <style> body{font-size:12px;} #txt{ height:400px; width:600px; border:#333 solid ...