[AirFlow]AirFlow使用指南一 安装与启动
1. 安装
通过pip安装:
xiaosi@yoona:~$ pip install airflow
如果速度比较慢，可以使用下面提供的源进行安装:
xiaosi@yoona:~$ pip install -i https://pypi.tuna.tsinghua.edu.cn/simple airflow
如果出现下面提示，表示你的airflow安装成功了:
Successfully installed airflow alembic croniter dill flask flask-admin flask-cache flask-login flask-swagger flask-wtf funcsigs future gitpython gunicorn jinja2 lxml markdown pandas psutil pygments python-daemon python-dateutil python-nvd3 requests setproctitle sqlalchemy tabulate thrift zope.deprecation Mako python-editor click itsdangerous Werkzeug wtforms PyYAML ordereddict gitdb2 MarkupSafe pytz numpy docutils setuptools lockfile six python-slugify idna urllib3 certifi chardet smmap2 Unidecode
Cleaning up...
安装完成之后我的默认安装在~/.local/bin目录下
2. 配置
如果不修改路径，默认的配置为~/airflow
永久修改环境变量
echo "export AIRFLOW_HOME=/home/xiaosi/opt/airflow" >> /etc/profile
source /etc/profile
为了便于操作方便，进行如下配置:
echo "export PATH=/home/xiaosi/.local/bin:$PATH" >> /etc/profile
source /etc/profile
3. 初始化
初始化数据库:
xiaosi@yoona:~$ airflow initdb
[2017-08-02 16:39:22,319] {__init__.py:57} INFO - Using executor SequentialExecutor
[2017-08-02 16:39:22,432] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-08-02 16:39:22,451] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
DB: sqlite:////home/xiaosi/opt/airflow/airflow.db
[2017-08-02 16:39:22,708] {db.py:287} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current schema
INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
/home/xiaosi/.local/lib/python2.7/site-packages/alembic/util/messaging.py:69: UserWarning: Skipping unsupported ALTER for creation of implicit constraint
warnings.warn(msg)
INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_isntance
INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding extra to Log
INFO [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, add dagrun
INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, task_duration
INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, dagrun_config
INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, add password column to user
INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, dagrun start end
INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss
INFO [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection
INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table
INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table
INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, add TI state index
INFO [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, add task fails journal table
INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, add dag_stats table
INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, xcom dag task indices
INFO [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance
INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table
Done.
运行上述命令之后，会在$AIRFLOW_HOME目录下生成如下文件:
xiaosi@yoona:~/opt/airflow$ ll
总用量 88
drwxrwxr-x 2 xiaosi xiaosi 4096 8月 2 16:39 ./
drwxrwxr-x 26 xiaosi xiaosi 4096 7月 31 13:56 ../
-rw-rw-r-- 1 xiaosi xiaosi 11424 8月 2 16:38 airflow.cfg
-rw-r--r-- 1 xiaosi xiaosi 58368 8月 2 16:39 airflow.db
-rw-rw-r-- 1 xiaosi xiaosi 1554 8月 2 16:38 unittests.cfg
4. 修改默认数据库
找到$AIRFLOW_HOME/airflow.cfg配置文件，进行如下修改:
sql_alchemy_conn = mysql://root:root@localhost:3306/airflow
备注
数据库用户名与密码均为root，airflow使用的数据库为airflow．使用如下命令创建对应的数据库:
mysql> create database airfow;
Query OK, 1 row affected (0.00 sec)
重新初始化服务器数据库:
xiaosi@yoona:~$ airflow initdb
出现了如下错误:
xiaosi@yoona:~$ airflow initdb
Traceback (most recent call last):
File "/home/xiaosi/.local/bin/airflow", line 17, in <module>
from airflow import configuration
File "/home/xiaosi/.local/lib/python2.7/site-packages/airflow/__init__.py", line 30, in <module>
from airflow import settings
File "/home/xiaosi/.local/lib/python2.7/site-packages/airflow/settings.py", line 159, in <module>
configure_orm()
File "/home/xiaosi/.local/lib/python2.7/site-packages/airflow/settings.py", line 147, in configure_orm
engine = create_engine(SQL_ALCHEMY_CONN, **engine_args)
File "/home/xiaosi/.local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
return strategy.create(*args, **kwargs)
File "/home/xiaosi/.local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 80, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/home/xiaosi/.local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 110, in dbapi
return __import__('MySQLdb')
ImportError: No module named MySQLdb
解决方案:
MySQL是最流行的开源数据库之一，但在Python标准库中并没有集成MySQL接口程序，MySQLdb是一个第三方包，需独立下载并安装。
sudo apt-get install python-mysqldb
再次初始化:
xiaosi@yoona:~$ airflow initdb
[2017-08-02 17:22:21,169] {__init__.py:57} INFO - Using executor SequentialExecutor
[2017-08-02 17:22:21,282] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-08-02 17:22:21,302] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
DB: mysql://root:***@localhost:3306/airflow
[2017-08-02 17:22:21,553] {db.py:287} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current schema
INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_isntance
INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding extra to Log
INFO [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, add dagrun
INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, task_duration
INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, dagrun_config
INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, add password column to user
INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, dagrun start end
INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss
INFO [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection
INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table
INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table
INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, add TI state index
INFO [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, add task fails journal table
INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, add dag_stats table
INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, xcom dag task indices
INFO [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance
INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table
Done.
查看一下airflow数据库中做了哪些操作:
mysql> use airflow;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+-------------------+
| Tables_in_airflow |
+-------------------+
| alembic_version |
| chart |
| connection |
| dag |
| dag_pickle |
| dag_run |
| dag_stats |
| import_error |
| job |
| known_event |
| known_event_type |
| log |
| sla_miss |
| slot_pool |
| task_fail |
| task_instance |
| users |
| variable |
| xcom |
+-------------------+
19 rows in set (0.00 sec)
5. 启动
通过如下命令就可以启动后台管理界面，默认访问localhost:8080即可:
xiaosi@yoona:~$ airflow webserver
[2017-08-02 17:25:31,961] {__init__.py:57} INFO - Using executor SequentialExecutor
[2017-08-02 17:25:32,075] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-08-02 17:25:32,095] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
/home/xiaosi/.local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
[2017-08-02 17:25:32,469] [9703] {models.py:167} INFO - Filling up the DagBag from /home/xiaosi/opt/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2017-08-02 17:25:33,052] {__init__.py:57} INFO - Using executor SequentialExecutor
[2017-08-02 17:25:33,156] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-08-02 17:25:33,179] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2017-08-02 17:25:33 +0000] [9706] [INFO] Starting gunicorn 19.3.0
[2017-08-02 17:25:33 +0000] [9706] [INFO] Listening at: http://0.0.0.0:8080 (9706)
[2017-08-02 17:25:33 +0000] [9706] [INFO] Using worker: sync
...
呈现出的主界面如下:
[AirFlow]AirFlow使用指南一 安装与启动的更多相关文章
- ColumnStore完整验证指南之安装与启动停止
官方文档https://mariadb.com/kb/en/library/mariadb-columnstore/ 推荐使用最新1.2.x最新版本. 先决条件 yum -y install boos ...
- Airflow使用入门指南
Airflow能做什么 关注公众号, 查看更多 http://mp.weixin.qq.com/s/xPjXMc_6ssHt16J07BC7jA Airflow是一个工作流分配管理系统,通过有向非循环 ...
- [AirFlow]AirFlow使用指南二 DAG定义文件
1. Example """ Code that goes along with the Airflow tutorial located at: https://git ...
- [转] Spark快速入门指南 – Spark安装与基础使用
[From] https://blog.csdn.net/w405722907/article/details/77943331 Spark快速入门指南 – Spark安装与基础使用 2017年09月 ...
- vSphere Web Client使用指南之安装配置
vSphere Web Client使用指南之安装配置 vSphere Web Client是为忙碌的管理员提供的一款通用的.基于浏览器的VMware管理工具,能够监控并管理VMware基础设施.在摆 ...
- Mongodb在windows下的安装和启动
在windows下安装的参考官方地址:https://docs.mongodb.com/manual/tutorial/install-mongodb-on-windows/ Mongodb的安装与启 ...
- MySQL 安装和启动服务，“本地计算机 上的 MySQL 服务启动后停止。某些服务在未由其他服务或程序使用时将自动停止。”
MySQL 安装和启动服务,以及遇到的问题 MySQL版本: mysql-5.7.13-winx64.zip (免安装,解压放到程序文件夹即可,比如 C:\Program Files\mysql-5. ...
- Linux下memcached安装和启动方法
Linux下memcached安装和启动方法 1. 首先下载memcached 和 libevent 包. Memcached用到了libevent这个库用于Socket的处理.下面是下载的两个包文件 ...
- 高手指南PHP安装配置
高手指南PHP安装配置 2014-11-05 12:57:13 来源: 评论:0 次 点击:12 次 | 发布人:登陆查看 PHP的快速发展,它的功能越来越强大,运用它也变得很方便,下面我 ...
随机推荐
- JAVA面试题整理（4）-Netty
1.BIO.NIO和AIO 2.Netty 的各大组件 3.Netty的线程模型 4.TCP 粘包/拆包的原因及解决方法 5.了解哪几种序列化协议?包括使用场景和如何去选择 6.Netty的零拷贝实现 ...
- 一文弄懂神经网络中的反向传播法——BackPropagation【转】
本文转载自:https://www.cnblogs.com/charlotte77/p/5629865.html 一文弄懂神经网络中的反向传播法——BackPropagation 最近在看深度学习 ...
- 详解WIFI破解-Kali篇
转自: http://www.secbox.cn/hacker/wireless/4877.html 工具: 1:笔记本 2:USB无线网卡(必备) 3:kali系统 4:靠谱字典 第一种方法: 暴力 ...
- The Startup Manager FAQ
Main Features: 1. Login Items: Manageable list of applications that are launched automatically ever ...
- Springboot-mongodb MongoRepository接口 save方法 详解
问题: 我们都知道 mongodb 有两种添加数据的方式 一种 就是 save 方法 另外一种 insert 方法 这里两个方法 唯一的区别就是 insert:当主键"_id&qu ...
- UVa 11354 邦德（最小瓶颈路+LCA）
https://vjudge.net/problem/UVA-11354 题意: 有n个城市m条道路,每条道路有一个危险系数.先在有若干个询问,要求找到一条从s到t的路,使得途径所有边的最大危险系数最 ...
- LNMP环境配置SSL证书 lnmp ssl add
.4新增的添加SSL功能 lnmp ssl add 如果需要添加输入 y ,不添加输入 n 回车. 选择了添加SSL会提示 有两个选项,1 选项为使用自己准备好的SSL证书和key. > &g ...
- sql数据类型转换函数
1.CAST()CAST (<expression> AS <data_ type>[ length ]) 2.CONVERT()CONVERT (<data_ type ...
- Dubbo 和 Spring Cloud微服务架构 比较及相关差异
你真的了解微服务架构吗?听听八年阿里架构师怎样讲述Dubbo和Spring Cloud微服务架构. 微服务架构是互联网很热门的话题,是互联网技术发展的必然结果.它提倡将单一应用程序划分成一组小的服务, ...
- UVA-10972 RevolC FaeLoN （边双连通+缩点）
题目大意:将n个点,m条边的无向图变成强连通图,最少需要加几条有向边. 题目分析:所谓强连通,就是无向图中任意两点可互达.找出所有的边连通分量,每一个边连通分量都是强连通的,那么缩点得到bcc图,只需 ...