Multiple DAGs run on Airflow 2.0 unintentionally at a time

1
votes

I installed Airflow with postgreSQL for the first time. However, after activating DAG on the web, about 16 DAGs are created at the same time as shown in the picture below. enter image description here

As a test, I made a dag that enters the current date into the db for 10 seconds once a second. If intended, it should be generated one row per second, but the problem above creates multiple rows per second. enter image description here

Below is the DAG code.

import pendulum
from airflow.models import DAG
from airflow.utils.dates import days_ago
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

kst = pendulum.timezone("Asia/Seoul")

args = {'owner': 'jovyan', 'start_date': datetime(2021, 5, 3, 16,20,10, tzinfo=kst), 'run_as_user':'bskim'}
#args = {'owner': 'jovyan', 'start_date': days_ago(n=1)}
dag  = DAG(dag_id='talend_first_dag',
        default_args=args,
        schedule_interval='0/1 * * * *')

talend_command = "/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS_run.sh "

t1 = BashOperator(task_id='talend_first',
                bash_command=talend_command,
                dag=dag)


t1

Below is result "ps-ef |grep airflow".

(venv) [bskim@swproduct airflow_home]$ ps -ef |grep airflow
bskim    108569 103922  2 17:23 pts/0    00:00:30 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow webserver
bskim    108607 108569  0 17:23 pts/0    00:00:01 gunicorn: master [airflow-webserver]
bskim    109264 103922  2 17:28 pts/0    00:00:25 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109266 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109274 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109275 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109276 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109277 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109278 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109280 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109281 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109282 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109283 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109285 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109286 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109287 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109288 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109290 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109291 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109292 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109293 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109294 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109296 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109297 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109298 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109300 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109301 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109302 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109303 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109304 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109306 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109307 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109308 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109310 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109311 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109312 109264  0 17:28 pts/0    00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim    109313 109264  0 17:28 pts/0    00:00:02 airflow scheduler -- DagFileProcessorManager
bskim    110825 108607  1 17:36 pts/0    00:00:05 [ready] gunicorn: worker [airflow-webserver]
bskim    110921 108607  1 17:36 pts/0    00:00:05 [ready] gunicorn: worker [airflow-webserver]
bskim    111012 108607  1 17:37 pts/0    00:00:05 [ready] gunicorn: worker [airflow-webserver]
bskim    111109 108607  1 17:37 pts/0    00:00:05 [ready] gunicorn: worker [airflow-webserver]
bskim    112652 109310  0 17:41 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:26:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    112729 109312  0 17:41 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:27:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    112754 112652  1 17:41 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:26:00+00:00 --job-id 448 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpepxvlj4v --error-file /tmp/tmpupd5hmkg
bskim    112781 112729  1 17:41 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:27:00+00:00 --job-id 449 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmp2kswizo1 --error-file /tmp/tmpiifxr0x7
bskim    112981 109276  0 17:42 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:43:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113052 109277  0 17:42 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:44:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113055 112981  2 17:42 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:43:00+00:00 --job-id 450 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmppkeav48n --error-file /tmp/tmp920t73mg
bskim    113065 108607  3 17:42 pts/0    00:00:02 gunicorn: worker [airflow-webserver]
bskim    113090 109274  0 17:42 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:45:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113121 113052  2 17:42 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:44:00+00:00 --job-id 451 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmphhj4r3oh --error-file /tmp/tmpa8ad955w
bskim    113162 109275  0 17:42 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:46:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113168 113090  2 17:42 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:45:00+00:00 --job-id 452 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmp_vhz10e0 --error-file /tmp/tmpsgh_rus8
bskim    113199 109278  0 17:42 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:47:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113200 113162  2 17:42 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:46:00+00:00 --job-id 453 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmpy3rssjwn --error-file /tmp/tmp39hnh9o1
bskim    113286 109280  0 17:42 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:28:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    113287 109285  0 17:42 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:48:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113288 113199  3 17:42 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:47:00+00:00 --job-id 454 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmpo1_l0dct --error-file /tmp/tmpe0ynvaaa
bskim    113299 109281  0 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:29:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    113300 109283  0 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:49:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113301 113286  3 17:43 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:28:00+00:00 --job-id 456 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpblx5vzdt --error-file /tmp/tmpbzp5s9st
bskim    113303 113287  3 17:43 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:48:00+00:00 --job-id 455 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmpawabzixj --error-file /tmp/tmpr4rau1_t
bskim    113331 109286  1 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:30:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    113332 109282  1 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:50:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113334 113300  3 17:43 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:49:00+00:00 --job-id 457 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmppn54vxqi --error-file /tmp/tmpc36akon7
bskim    113336 113299  3 17:43 ?        00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:29:00+00:00 --job-id 458 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpffxckf4j --error-file /tmp/tmpqnx2vdom
bskim    113347 113332  3 17:43 ?        00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:50:00+00:00 --job-id 459 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmphexft182 --error-file /tmp/tmpq0_wfubu
bskim    113348 113331  3 17:43 ?        00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:30:00+00:00 --job-id 460 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpxp7m7131 --error-file /tmp/tmp7qpmc7kp
bskim    113351 109288  1 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:31:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    113352 109287  1 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:51:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113366 113360  6 17:43 ?        00:00:01 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -Xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim    113376 109290  1 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:32:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    113381 113352  3 17:43 ?        00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:51:00+00:00 --job-id 461 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmpg2lyaeig --error-file /tmp/tmpvpcv2k_d
bskim    113382 113351  3 17:43 ?        00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:31:00+00:00 --job-id 462 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmp62kc8itw --error-file /tmp/tmpvhebo3kj
bskim    113401 113398  6 17:43 ?        00:00:00 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -Xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim    113449 109293  1 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:33:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    113450 109291  2 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:52:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113462 113376  3 17:43 ?        00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:32:00+00:00 --job-id 463 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpj3908y63 --error-file /tmp/tmp7e2vqj77
bskim    113468 113464  6 17:43 ?        00:00:00 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -Xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim    113483 109292  2 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:34:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    113484 109300  2 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:53:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113488 113449  3 17:43 ?        00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:33:00+00:00 --job-id 464 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmp2d7zszd2 --error-file /tmp/tmpxhrfjto_
bskim    113490 113450  3 17:43 ?        00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:52:00+00:00 --job-id 465 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmp05cgdqd1 --error-file /tmp/tmpv8x5_4lc
bskim    113497 113491  5 17:43 ?        00:00:00 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -Xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim    113513 109294  3 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_idx_2_dag', 'talend_idx_2', '2021-05-03T09:35:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim    113514 109296  4 17:43 pts/0    00:00:00 airflow task supervisor: ['airflow', 'tasks', 'run', 'talend_first_dag', 'talend_first', '2021-05-03T10:54:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim    113515 113483  4 17:43 ?        00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:34:00+00:00 --job-id 466 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpmmkwhfic --error-file /tmp/tmpi5wzbio8
bskim    113518 113484  4 17:43 ?        00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:53:00+00:00 --job-id 467 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmp1b9oaxu0 --error-file /tmp/tmpyxqlwzvs
bskim    113532 113529  4 17:43 ?        00:00:00 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -Xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim    113535 103922  0 17:43 pts/0    00:00:00 grep --color=auto airflow

The executor is LocalExecutor, and even if it is set to SequentialExecutor, the result is the same.

I wasted two days because of this problem. I need your help!

1
airflow-schedulerairflow
This question isn't clear and very broad. Please focus your question and post the DAG code that you are asking about. The behavior is a result of a code if you don't share a code we can't help you. - Elad
All right. Give me five minutes. - everythx
You are getting many runs because your start_date is in the past and you didn't add catchup = False - Elad
Wow!!! You are my Savior!!! Thank you very much!! - everythx

1 Answers

0
votes

Airflow has a default True value for catchup_by_default setting so if you didn't change the defaults you must specify catchup=False in your DAGs if you don't want them to backfill runs from the given start_date.

dag  = DAG(dag_id='talend_first_dag',
        default_args=args,
        schedule_interval='0/1 * * * *',
        catchup=False)

Another possible solution (assuming you need catchup) is not to set the start_date in the past.