Hadoop & Spark & Hive & HBase

Hadoop:
http://hadoop.apache.org/docs/r2.6.4/hadoop-project-dist/hadoop-common/SingleCluster.html

bin/hdfs namenode -format
sbin/start-dfs.sh


 http://localhost:50070/
 



bin/hdfs dfs -mkdir /user
bin/hdfs dfs -mkdir /user/<username>


these are for testing:


bin/hdfs dfs -put etc/hadoop input
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar grep input output 'dfs[a-z.]+'
bin/hdfs dfs -cat output/*


testing results:


6       dfs.audit.logger
4       dfs.class
3       dfs.server.namenode.
2       dfs.period
2       dfs.audit.log.maxfilesize
2       dfs.audit.log.maxbackupindex
1       dfsmetrics.log
1       dfsadmin
1       dfs.servers
1       dfs.replication
1       dfs.file







YARN: 
ResourceManager

./sbin/start-yarn.sh


Http://localhost:8088/
 



HistoryServer



./sbin/mr-jobhistory-daemon.sh start historyserver





http://localhost:19888/
 




Spark:




http://spark.apache.org/docs/1.6.2/
  start: 


./sbin/start-master.sh


 http://localhost:8080/


 start worker:




./sbin/start-slaves.sh spark://<your-computer-name>:7077  


You will see:


Alive Workers: 1

 http://localhost:8080/



This is for testing:





./bin/spark-shell --master spark://<your-computer-name>:7077





You will see the scala shell.
use :q to quit.

To see the history:

http://spark.apache.org/docs/latest/monitoring.html

http://blog.chinaunix.net/uid-29454152-id-5641909.html

http://www.cloudera.com/documentation/cdh/5-1-x/CDH5-Installation-Guide/cdh5ig_spark_configure.html 

./sbin/start-history-server.sh 

http://localhost:18080/

Hive:

https://cwiki.apache.org/confluence/display/Hive/GettingStarted

https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2

http://www.360doc.com/content/16/0411/19/2795334_549791350.shtml

Bug:

in mysql 5.7 you should use :

jdbc:mysql://localhost:3306/hivedb?useSSL=false&createDatabaseIfNotExist=true

start hiveserver2:

 nohup hiveserver2 &

http://localhost:10002/

Bug：

User:  is not allowed to impersonate anonymous (state=,code=0)

https://community.hortonworks.com/questions/42483/user-hive-is-not-allowed-to-impersonate-anonymous.html

http://stackoverflow.com/questions/31228420/how-to-run-hive-on-spark-job-from-beeline-or-any-jdbc-client

See more:

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html

Hwi 界面Bug：

HWI WAR file not found at

pack the war file yourself, then copy it to the right place, then add needed setting into hive-site.xml

http://blog.csdn.net/gao634209276/article/details/51426371

http://blog.csdn.net/duguduchong/article/details/8852425

Problem: failed to create task or type componentdef
Or:
Could not create task or type of type: componentdef

sudo apt-get install libjasperreports-java

sudo apt-get install ant

_________________________________________________________________________________not finished

自定义配置：

http://blog.csdn.net/reesun/article/details/8556078

数据库连接软件：

默认 用户名就是登录账号 密码为空

http://blog.sina.com.cn/s/blog_76923bd80102wi3g.html

语法

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL

more info:

http://stackoverflow.com/questions/35476468/what-is-the-difference-between-the-hive-metastore-in-derby-vs-the-one-in-hive-wa

http://www.2cto.com/database/201408/325554.html

HBase

http://hbase.apache.org/book.html#quickstart

./bin/start-hbase.sh 

http://localhost:16010/

HBase & Hive

Hive & Shark & SparkSQL

Spark SQL架构如下图所示:

http://blog.csdn.net/wzy0623/article/details/52249187

 http://lib.csdn.net/article/spark/33925

phoenix

queryserver.py start

jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF

Or:

phoenix-sqlline.py localhost:2181

来自为知笔记(Wiz)