在airflow的BashOperator中执行docker容器中的脚本容易忽略的问题

dag模板

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators import ExternalTaskSensor
from airflow.operators import EmailOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(, , ,,,),
    'retries': ,
    'retryDelay': timedelta(seconds=),
    'end_date': datetime(, , )
}

dag = DAG('time_my',
    default_args=default_args,
    schedule_interval='0 0 * * *')

time_my_task_1 = BashOperator(
    task_id='time_my_task_1',
    dag=dag,
bash_command='set -e;docker exec -it testsuan /bin/bash -c "cd /algorithm-platform/algorithm_model_code/ && python time_my_task_1.py " '
)
time_my_task_1

在调度的时候日志报这样的错误

[-- ::,] {models.py:} INFO - Dependencies all met for <TaskInstance: time_my.time_my_task_1 --05T09::39.294890+: [queued]>
[-- ::,] {models.py:} INFO - Dependencies all met for <TaskInstance: time_my.time_my_task_1 --05T09::39.294890+: [queued]>
[-- ::,] {models.py:} INFO -
--------------------------------------------------------------------------------
Starting attempt  of
--------------------------------------------------------------------------------

[-- ::,] {models.py:} INFO - Executing <Task(BashOperator): time_my_task_1> on --05T09::39.294890+:
[-- ::,] {base_task_runner.py:} INFO - Running: ['bash', '-c', u'airflow run time_my time_my_task_1 2019-12-05T09:21:39.294890+08:00 --job_id 53877 --raw -sd DAGS_FOLDER/time_my.py --cfg_path /tmp/tmprAeiWr']
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 /usr/lib/python2./site-packages/requests/__init__.py:: RequestsDependencyWarning: urllib3 (1.25.) or chardet (3.0.) doesn't match a supported version!
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   RequestsDependencyWarning)
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {settings.py:} INFO - settings.configure_orm(): Using pool settings. pool_size=, pool_recycle=, pid=
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {__init__.py:} INFO - Using executor LocalExecutor
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {configuration.py:} WARNING - section/key [rest_api_plugin/log_loading] not found in config
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {rest_api_plugin.py:} WARNING - Initializing [rest_api_plugin/LOG_LOADING] with default value = False
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {configuration.py:} WARNING - section/key [rest_api_plugin/filter_loading_messages_in_cli_response] not found in config
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {rest_api_plugin.py:} WARNING - Initializing [rest_api_plugin/FILTER_LOADING_MESSAGES_IN_CLI_RESPONSE] with default value = True
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {configuration.py:} WARNING - section/key [rest_api_plugin/rest_api_plugin_http_token_header_name] not found in config
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {rest_api_plugin.py:} WARNING - Initializing [rest_api_plugin/REST_API_PLUGIN_HTTP_TOKEN_HEADER_NAME] with default value = rest_api_plugin_http_token
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {configuration.py:} WARNING - section/key [rest_api_plugin/rest_api_plugin_expected_http_token] not found in config
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {rest_api_plugin.py:} WARNING - Initializing [rest_api_plugin/REST_API_PLUGIN_EXPECTED_HTTP_TOKEN] with default value = None
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {models.py:} INFO - Filling up the DagBag from /usr/local/airflow/dags/time_my.py
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 /usr/lib/python2./site-packages/airflow/utils/helpers.py:: DeprecationWarning: Importing 'ExternalTaskSensor' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   DeprecationWarning)
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 /usr/lib/python2./site-packages/airflow/utils/helpers.py:: DeprecationWarning: Importing 'EmailOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   DeprecationWarning)
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 [-- ::,] {cli.py:} INFO - Running <TaskInstance: time_my.time_my_task_1 --05T09::39.294890+: [running]> on host node5
[-- ::,] {bash_operator.py:} INFO - Tmp dir root location:
 /tmp
[-- ::,] {bash_operator.py:} INFO - Exporting the following env vars:
AIRFLOW_CTX_TASK_ID=time_my_task_1
AIRFLOW_CTX_DAG_ID=time_my
AIRFLOW_CTX_EXECUTION_DATE=--05T09::39.294890+:
AIRFLOW_CTX_DAG_RUN_ID=manual__2019--05T09::39.294890+:
[-- ::,] {bash_operator.py:} INFO - Temporary script location: /tmp/airflowtmpoAfSER/time_my_task_1biUSjF
[-- ::,] {bash_operator.py:} INFO - Running command: set -e;docker exec -it testsuan /bin/bash -c "cd /algorithm-platform/algorithm_model_code/ && python time_my_task_1.py"
[-- ::,] {bash_operator.py:} INFO - Output:
[-- ::,] {bash_operator.py:} INFO - the input device is not a TTY
[-- ::,] {bash_operator.py:} INFO - Command exited with return code
[-- ::,] {models.py:} ERROR - Bash command failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line , in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/lib/python2.7/site-packages/airflow/operators/bash_operator.py", line , in execute
    raise AirflowException("Bash command failed")
AirflowException: Bash command failed
[-- ::,] {models.py:} INFO - Marking task as UP_FOR_RETRY
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 Traceback (most recent call last):
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   File "/usr/bin/airflow", line , in <module>
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1     args.func(args)
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line , in wrapper
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1     return f(*args, **kwargs)
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line , in run
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1     _run(args, dag, ti)
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line , in _run
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1     pool=args.pool,
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line , in wrapper
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1     return func(*args, **kwargs)
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   File "/usr/lib/python2.7/site-packages/airflow/models.py", line , in _run_raw_task
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1     result = task_copy.execute(context=context)
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1   File "/usr/lib/python2.7/site-packages/airflow/operators/bash_operator.py", line , in execute
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1     raise AirflowException("Bash command failed")
[-- ::,] {base_task_runner.py:} INFO - Job : Subtask time_my_task_1 airflow.exceptions.AirflowException: Bash command failed
[-- ::,] {logging_mixin.py:} INFO - [-- ::,] {jobs.py:} INFO - Task exited with return code 

其实问题就出在这

用定时任务执行docker命令的脚本的时候报错如上标题,tty(终端设备的统称): tty一词源于Teletypes,或teletypewriters。

这个的意思是说后台linux执行的时候没有终端设备。我们一般执行docker里的命令时候都喜欢加上-it 这个参数，这里的-it 就是表示终端设备。

所以，如果我们docker执行后台运行的任务或者程序直接去除 -it 这个参数就不会出现这个报错了！

修改后的DAG模板

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators import ExternalTaskSensor
from airflow.operators import EmailOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(, , ,,,),
    'retries': ,
    'retryDelay': timedelta(seconds=),
    'end_date': datetime(, , )
}

dag = DAG('time_my',
    default_args=default_args,
    schedule_interval='0 0 * * *')

time_my_task_1 = BashOperator(
    task_id='time_my_task_1',
    dag=dag,
bash_command='set -e;docker exec -i testsuan /bin/bash -c "cd /algorithm-platform/algorithm_model_code/ && python time_my_task_1.py " '
)
time_my_task_1

再次调度就不会出现错误了。