An operator defines an individual task that needs to be performed. Although each one can mention multiple tasks, it’s a good idea to keep one logical workflow in one file. DAGs are defined in Python files that are placed in Airflow’s DAG_FOLDER. A graph- it’s a very convenient way to view the process. Directed means the tasks are executed in some order.Īcyclic- as you cannot create loops (i.e. Airflow concepts DagĪn Airflow workflow is designed as a directed acyclic graph (DAG). It is generally best suited for regular operations which can be scheduled to run at specific times. ETL (extract, transform, load) jobs - extracting data from multiple sources, transforming for analysis and loading it into a data store.Python Based: Every part of the configuration is written in Python, including configuration of schedules and the scripts to run them.Įxample of use cases suitable for Airflow:.Web Interface: Airflow ships with a Flask app that tracks all the defined workflows, and allows you to easily start, stop, change workflows.This is one of the main reasons why this project was adopted and managed by Apache foundation.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |