superset部署

superset功能概述:

• 丰富的数据可视化集

• 易于使用的界面，用于探索和可视化数据

• 创建和共享仪表板

• 与主要身份验证提供程序集成的企业级身份验证（通过Flask AppBuilder进行数据库，OpenID，LDAP，OAuth和REMOTE_USER）

• 可扩展的高粒度安全/权限模型，允许谁可以访问各个功能和数据集的复杂规则

• 一个简单的语义层，允许用户通过定义哪些字段应显示在哪个下拉列表以及哪些聚合和功能度量标准可供用户使用来控制数据源在UI中的显示方式

• 通过SQLAlchemy与大多数讲SQL的RDBMS集成

• 与Druid.io深度整合

部署:

官网:http://superset.apache.org/installation.html

一. 准备python3环境

#安装基础依赖

yum -y groupinstall "Development tools"

yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel

#superset用到的依赖

yum install gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64

wget https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tar.xz

#速度慢，直接去官网下载相应版本

mkdir -p /usr/local/python3

然后解压压缩包，进入该目录，安装Python3

tar -xvf  Python-3.6.2.tar.xz

cd Python-3.6.2

./configure --prefix=/usr/local/python3

make && make install

创建软链接:

ln -s /usr/local/python3/bin/python3 /usr/bin/python3

ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3

在命令行中输入python3测试:

[root@oldboy .superset]# python3 --version
Python 3.6.2

二 创建python3的虚拟环境

pip3 install virtualenv

mkdir /data

python3 -m venv venv

#创建成功会在/data目录下生成虚拟目录:

[root@oldboy .superset]# cd /data
[root@oldboy data]# ls
venv

#激活
. venv/bin/activate

#退出用

deactivate

三在虚拟环境中安装superset

(venv) [root@oldboy data]# pip3 install --upgrade setuptools pip

pip3 install superset

# Initialize the database
superset db upgrade

# Create an admin user (you will be prompted to set a username, first and last name before setting a password)
export FLASK_APP=superset
flask fab create-admin

# Load some data to play with
superset load_examples

# Create default roles and permissions
superset init

#启动,默认端口是8088

superset runserver -d

nohup superset runserver -d >> /dev/null 2>&1 &

单线程会有闪退的现象,用:

.  /data/venv/bin/activate

gunicorn -w 10  --timeout 120 -b  0.0.0.0:6666 --limit-request-line 0 --limit-request-field_size 0 --statsd-host localhost:8125 superset:app

或者同事的启动方法:

/data/venv/bin/python3 /data/venv/bin/gunicorn -w 10 --timeout 120 -b 0.0.0.0:6666 --limit-request-line 0 --limit-request-field_size 0 --statsd-host localhost:8125 superset:app > /dev/null 2>&1 &

#查看

netstat -lntup |grep 6666

#停止

pkill gunicorn

#默认数据库是sqlite,在配置文件里可以改为mysql等

/data/venv/lib/python3.6/site-packages/superset/config.py

连接mysql需要安装:

yum install mysql-devel
pip3 install mysqlclient

连接impala需要安装:

pip3 install impyla  需要重启服务

注意:

在安装中初始化数据库时会报错,这是superset的bug: 可以去https://github.com/apache/incubator-superset/issues看

1/ cannot import name '_maybe_box_datetimelike'

解决方法:
pip3 uninstall pandas
pip list | grep pandas
pip3 install pandas==0.23.4

2/ "Can't determine which FROM clause to join "
sqlalchemy.exc.InvalidRequestError: Can't determine which FROM clause to join from, there are multiple FROMS which can join to this entity. Try adding an explicit ON clause to help resolve the ambiguity.

解决方法:
pip3 install sqlalchemy==1.2.18

一点经验:

迁移的时候,如果有错,2个小时排不出来,赶紧重装.

重装如果不对,一定是版本不对. python版本和superset版本

pip3 freeze | grep superset  #看superset版本

注意安装的时候用pip3, 因为做了软连接

#sqlite数据库备份脚本

#! /bin/sh
date=$(date +%Y_%m_%d)
cp -f /root/.superset/superset.db /fengfeng/backup/
mv /fengfeng/backup/superset.db /fengfeng/backup/superset${date}.db
scp /fengfeng/backup/superset${date}.db 10.52.80.112:/fengfeng/backup
sleep 5
rm -f /fengfeng/backup/superset${date}.db
~

00 00 * * * /bin/sh /fengfeng/script/superset_backup.sh >/dev/null 2>&1

#在备份服务器上做个定时删除:

00 01 * * * /bin/sh /fengfeng/script/del_superset_db.sh >/dev/null 2>&1

find /fengfeng/backup -mtime +2 -name "superset*.db" -exec -Rf {} \;

nginx:

upstream opossuperset_backend{
server 10.52.2.243:6666;
keepalive 1000;
}

server{

location / {
proxy_pass http://opossuperset_backend;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $http_x_real_ip;
}
}