启动 ./spark-shell 命令报错
当使用./spark-shell 命令报错
Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$@476fde05, see the next exception for details.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
... more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /usr/local/development/spark-2.1.-bin-hadoop2./bin/metastore_db.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)
--------------------
Caused by: java.sql.SQLException: Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$@476fde05, see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
-------------------
Caused by: org.apache.hadoop.ipc.RemoteException: Cannot create directory /tmp/hive/root/5436b1aa-85e3--b505-b0bdc7444e46. Name node is in safe mode.
The reported blocks needs additional blocks to reach the threshold 0.9990 of total blocks .
The number of live datanodes has reached the minimum number . Safe mode will be turned off automatically once the thresholds have been reached.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenode
抛出了很多错误,甚至我退出spark-shell,再次进入抛出的错误还不一样,最后一个错误是Name node is in safe mode. 先解决这个
网上说这是hdfs的分布式文件系统的安全模式,当安全模式时文件系统中的内容不允许修改和删除,直到安全模式结束,安全模式就是系统在检查各个dataNode数据块的有效性
bin/hadoop dfsadmin -safemode leave //离开安全模式
用户可以通过dfsadmin -safemode value 来操作安全模式,参数value的说明如下:
//enter - 进入安全模式
//leave - 强制NameNode离开安全模式
[root@node1 sbin]# hdfs dfsadmin -safemode leave
Safe mode is OFF
//get - 返回安全模式是否开启的信息
[root@node1 sbin]# hdfs dfsadmin -safemode get
Safe mode is ON
//wait - 等待,一直到安全模式结束。
当离开安全模式再次spark-shell,抛出异常
Caused by: org.apache.derby.iapi.error.StandardException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$@476fde05, see the next exception for details.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
... more
Caused by: org.apache.derby.iapi.error.StandardException: Another instance of Derby may have already booted the database /usr/local/development/spark-2.1.-bin-hadoop2./bin/metastore_db.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)
可以看到,这个异常是说在/usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/ 中已经有另一个数据库(Derby)实例了,我一直都是使用mysql数据库,没用过Derby数据库,最后想想可能是我在安装spark的时候,连接了hive,那个时候我的hive还
配置mysql数据库,hive默认使用Derby数据库,所以当我启动spark连接hive的时候,就自动生成了Derby的实例。我看了'metastore_db' 文件夹的内容是Derby数据库的配置
解决方法:
删除/usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/ metastore_db 文件夹。
重新启动
./spark-shell
[root@node1 bin]# ./spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/07/05 00:13:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/05 00:13:29 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/07/05 00:13:29 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
17/07/05 00:13:40 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.177.120:4040
Spark context available as 'sc' (master = local[*], app id = local-1499184787668).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.1
/_/ Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information. scala> scala> sc
res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@178c4480
启动 ./spark-shell 命令报错的更多相关文章
- 执行Spark运行在yarn上的命令报错 spark-shell --master yarn-client
1.执行Spark运行在yarn上的命令报错 spark-shell --master yarn-client,错误如下所示: // :: ERROR SparkContext: Error init ...
- hadoop命令报错:权限问题
root用户执行hadoop命令报错: [root@vmocdp125 conf]# hadoop fs -ls /user/ [INFO] 17:50:42 main [RetryInvocatio ...
- windows下使用GNU make命令报错的解决方法
windows下使用GNU make命令报错的解决方法=> 错误信息:make: Interrupt/Exception caught (code = 0xc00000fd, addr = 0x ...
- win7环境下,vagrant,在启动虚拟机的时候报错io.rb:32:in `encode': incomplete "\xC8" on GBK (Encoding::InvalidByteSequenceError)
描述: 这几天在windows环境上,部署了vagrant,在启动虚拟机的时候报错: [c:\~]$ vagrant upBringing machine 'default' up with 'vir ...
- eclipse java项目中明明引入了jar包 为什么项目启动的时候不能找到jar包 项目中已经 引入了 com.branchitech.app 包 ,但时tomcat启动的时候还是报错? java.lang.ClassNotFoundException: com.branchitech.app.startup.AppStartupContextListener java.lang.ClassN
eclipse java项目中明明引入了jar包 为什么项目启动的时候不能找到jar包 项目中已经 引入了 com.branchitech.app 包 ,但时tomcat启动的时候还是报错?java. ...
- CURL命令报错:dyld: lazy symbol binding failed: Symbol not found: _SSL_load_error_strings解决办法
Mac OS X 10.11.6, curl 命令报错,错误如下: dyld: lazy symbol binding failed: Symbol not found: _SSL_load_erro ...
- Windows 8.1升级至Windows 10后,启动VisualSVN Server Manager报错:提供程序无法执行所尝试的操作 (0x80041024)的解决
1.1.Windows 8.1升级至Windows 10后,启动VisualSVN Server Manager报错:提供程序无法执行所尝试的操作 (0x80041024),VisualSVN Ser ...
- RedHat中敲sh-copy-id命令报错:-bash: ssh-copy-id: command not found
RedHat中敲sh-copy-id命令报错:-bash: ssh-copy-id: command not found 在多台Linux服务器SSH相互访问无需密码, 其中进入一台Linus中,对其 ...
- php artisan 命令报错,什么命令都是这个错误,cmd下运行也不行,又没看到语法错误
Laravel 5.1 以上的版本的框架需求PHP的版本是5.5以上的版本.如果你的PHP版本等级太低,将会出现上述的问题. 估计你要升级你的PHP版本了.
随机推荐
- android 错误
问题1 手机开发者选项 开启USB安装 问题2 怎么变成了两行 问题3 运行虚拟机报错 电脑没有启用虚拟技术或者没有安装Intel HAXM软件 sdk下找到 (或者官网下载 https://soft ...
- Codeforces Beta Round #4 (Div. 2 Only) C. Registration system【裸hash/map】
C. Registration system time limit per test 5 seconds memory limit per test 64 megabytes input standa ...
- HDU 1426 Sudoku Killer (回溯 + 剪枝)
本文链接:http://i.cnblogs.com/EditPosts.aspx?postid=5398818 题意: 给你一个 9*9 的矩阵,同一行相邻的两个元素用一个空格分开.其中1-9代表该位 ...
- ELK集群
kafka集群-------------------1. 下载wget http://mirror.rise.ph/apache/kafka/0.11.0.0/kafka_2.12-0.11.0.0. ...
- LAMP----linux+apache+mysql+php详细安装步骤之一APACHE篇(openldap等)
LAMP----linux+apache+mysql+php详细安装步骤之一APACHE篇(openldap等) linux详细版本为RHEL5.3 [root@localhost mail]# un ...
- 【安居客】资深PHP软件开发工程师
工作职责: 1.网站项目的开发和维护: 2.负责技术部软件开发架构设计: 3.负责生产环境.测试环境和生产环境服务器运维和优化: 4.负责研究较前沿和复杂的技术运用: 岗位要求: 1.熟悉 PHP 程 ...
- sql server 2008 R2 数据库文件大小限制
select @@version SQL2005 分五个版本,如下所列: 1.Enterprise(企业版) 2.Development(开发版) 3.Workgroup,(工作群版) 4.Stand ...
- Vue服务端渲染 VS Vue浏览器端渲染)
Vue 2.0 开始支持服务端渲染的功能,所以本文章也是基于vue 2.0以上版本.网上对于服务端渲染的资料还是比较少,最经典的莫过于Vue作者尤雨溪大神的 vue-hacker-news.本人在公司 ...
- 【Linux】CentOS7 添加常用源
CentOS 的官方源去掉了一些与版权有关的软件,因此想要安装这些软件或者手动下载安装,或者使用其他源. 下面是添加EPEL源和RPMforge源的步骤. 1.首先, 添加源之前要确定系统架构及版本 ...
- Geek改变世界
在10月24号的GeekPwn到来前,主办方 — 来自Keen Team的创始人大牛蛙希望我能为GeekPwn写点东西.作为GeekPwn的顾问,我也非常乐意为这次首秀做一点事情. 正如之前提到过的, ...