I have installed Airflow with celeryexecutor having 2 workers , 1 scheduler and 1 webserver. I am starting all of these components with systemd. All components start fine I can see the DAGs in web UI. When I execute any DAG first task itself fails without even starting it. Seems some issue in starting the task at worker. Below is the log I am getting in journalctl.
Worker error log :
ct 21 19:17:23 mydomain.mycompany python[9780]: [2019-10-21 19:17:23,425: DEBUG/MainProcess] Task accepted: airflow.executors.celery_executor.execute_command[526ed99b-f21f-40ba-8018-afbf0dc9a36a] pid:9800
Oct 21 19:17:23 mydomain.mycompany python[9780]: [2019-10-21 19:17:23,425: INFO/Worker-16] Executing command in Celery: airflow run sip_data_parsing_batch_execution_dag clear_configurations 2019-10-21T19:17:20.802983+00:00 --local -sd /opt/sip-orchestration/airflow_pai/dags/sip_data_parsing/sip_data_parsing_dag.py
Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 2618, in _execute
Oct 21 19:17:24 mydomain.mycompany python[9780]: self.task_runner.start()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/bash_task_runner.py", line 34, in start
Oct 21 19:17:24 mydomain.mycompany python[9780]: self.process = self.run_command(['bash', '-c'], join_args=True)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/base_task_runner.py", line 126, in run_command
Oct 21 19:17:24 mydomain.mycompany python[9780]: preexec_fn=os.setsid
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
Oct 21 19:17:24 mydomain.mycompany python[9780]: restore_signals, start_new_session)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child
Oct 21 19:17:24 mydomain.mycompany python[9780]: raise child_exception_type(errno_num, err_msg, err_filename)
Oct 21 19:17:24 mydomain.mycompany python[9780]: FileNotFoundError: [Errno 2] No such file or directory: 'bash': 'bash'
Oct 21 19:17:24 mydomain.mycompany python[9780]: During handling of the above exception, another exception occurred:
Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/bin/airflow", line 32, in <module>
Oct 21 19:17:24 mydomain.mycompany python[9780]: args.func(args)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
Oct 21 19:17:24 mydomain.mycompany python[9780]: return f(*args, **kwargs)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/bin/cli.py", line 490, in run
Oct 21 19:17:24 mydomain.mycompany python[9780]: _run(args, dag, ti)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/bin/cli.py", line 401, in _run
Oct 21 19:17:24 mydomain.mycompany python[9780]: run_job.run()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 203, in run
Oct 21 19:17:24 mydomain.mycompany python[9780]: self._execute()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 2654, in _execute
Oct 21 19:17:24 mydomain.mycompany python[9780]: self.on_kill()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 2658, in on_kill
Oct 21 19:17:24 mydomain.mycompany python[9780]: self.task_runner.on_finish()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/bash_task_runner.py", line 44, in on_finish
Oct 21 19:17:24 mydomain.mycompany python[9780]: super(BashTaskRunner, self).on_finish()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/base_task_runner.py", line 163, in on_finish
Oct 21 19:17:24 mydomain.mycompany python[9780]: subprocess.call(['sudo', 'rm', self._cfg_path], close_fds=True)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 287, in call
Oct 21 19:17:24 mydomain.mycompany python[9780]: with Popen(*popenargs, **kwargs) as p:
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
Oct 21 19:17:24 mydomain.mycompany python[9780]: restore_signals, start_new_session)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child
Oct 21 19:17:24 mydomain.mycompany python[9780]: raise child_exception_type(errno_num, err_msg, err_filename)
Oct 21 19:17:24 mydomain.mycompany python[9780]: FileNotFoundError: [Errno 2] No such file or directory: 'sudo': 'sudo'
Oct 21 19:17:24 mydomain.mycompany python[9780]: [2019-10-21 19:17:24,681: ERROR/Worker-16] execute_command encountered a CalledProcessError
Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 58, in execute_command
Oct 21 19:17:24 mydomain.mycompany python[9780]: close_fds=True, env=env)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
Oct 21 19:17:24 mydomain.mycompany python[9780]: raise CalledProcessError(retcode, cmd)
Oct 21 19:17:24 mydomain.mycompany python[9780]: subprocess.CalledProcessError: Command 'airflow run sip_data_parsing_batch_execution_dag clear_configurations 2019-10-21T19:17:20.802983+00:00 --local -sd /opt/sip-orchestration/airflow_pai/dags/sip_data_parsing/sip_data_parsing_dag.py' returned non-zero exit status 1.
Oct 21 19:17:24 mydomain.mycompany python[9780]: [2019-10-21 19:17:24,682: ERROR/Worker-16] None
Oct 21 19:17:24 mydomain.mycompany python[9780]: [2019-10-21 19:17:24,684: ERROR/MainProcess] Task airflow.executors.celery_executor.execute_command[526ed99b-f21f-40ba-8018-afbf0dc9a36a] raised unexpected: AirflowException('Celery command failed',)
Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/celery/app/trace.py", line 240, in trace_task
Oct 21 19:17:24 mydomain.mycompany python[9780]: R = retval = fun(*args, **kwargs)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
Oct 21 19:17:24 mydomain.mycompany python[9780]: return self.run(*args, **kwargs)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 63, in execute_command
Oct 21 19:17:24 mydomain.mycompany python[9780]: raise AirflowException('Celery command failed')
Oct 21 19:17:24 mydomain.mycompany python[9780]: airflow.exceptions.AirflowException: Celery command failed
Command I use to start components.
- sudo service airflow-worker start
sudo service airflow-scheduler start
sudo service airflow-webserver start
Airflow version - 1.10.1 Celery version - 3.1.17
