want to upload a file to s3 useing apache airflow [ DAG ] file

Question

i want to make a DAG file (apache airflow) for uploading a rar file to s3 bucket any one tried.? plz suggest ,

and i tried these things on my DAG file but there is showing some error

from airflow.operators import SimpleHttpOperator, HttpSensor,   , EmailOperator, S3KeySensor

The error is

/usr/local/lib/python3.6/dist-packages/airflow/utils/helpers.py:439: DeprecationWarning: Importing 'SimpleHttpOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.

DeprecationWarning) /usr/local/lib/python3.6/dist-packages/airflow/utils/helpers.py:439: DeprecationWarning: Importing 'HttpSensor' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.

DeprecationWarning) /usr/local/lib/python3.6/dist-packages/airflow/utils/helpers.py:439: DeprecationWarning: Importing 'EmailOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.

how to solve this issue.?

Daniel Huang Daniel Huang · Accepted Answer · 2020-03-11T07:10:50

This is simply a warning, not an error. A DeprecationWarning usually hints that something you're doing will work now, but may break in future versions. If your task is failing, ignore these messages and look for a proper error.

Code for operators have always been located under airflow.operators.[operator_module], but it was also made available under airflow.operators directly for convenience. For example, SimpleHttpOperator is defined in https://github.com/apache/airflow/blob/1.10.9/airflow/operators/http_operator.py, so importing it from airflow.operators.http_operator will definitely work. However, importing it from airflow.operators will also work due to the code that currently exists in https://github.com/apache/airflow/blob/1.10.9/airflow/operators/init.py#L97-L99, at least for now in your current version of Airflow. Basically, you can address these warnings by updating your imports to the following:

from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.email_operator import EmailOperator
from airflow.sensors.http_sensor import HttpSensor
from airflow.sensors.s3_key_sensor import S3KeySensor

Just heads up, currently only on the master branch, not yet in any released version, some of the third party operators and sensors have also been moved again. For example, S3KeySensor will be found under providers.amazon.aws.sensors.s3_key.py. As expected, importing from the "old" path will get you a similar deprecation message, https://github.com/apache/airflow/blob/97a429f9d0cf740c5698060ad55f11e93cb57b55/airflow/sensors/s3_key_sensor.py#L25-L28.

want to upload a file to s3 useing apache airflow [ DAG ] file

1 Answers