Apply ⏱ 120 min

Airflow Tutorial for Beginners - Full Course in 2 Hours 2022

What You Will Learn

Create a Python project and install Apache Airflow
Initialize the Airflow database and start the Airflow web server
Define and run a simple DAG using the BashOperator and PythonOperator

Key Concepts

Airflow is an open-sourced tool for creating, scheduling, and monitoring workflows. It is based on Python and allows users to define tasks and their dependencies using a directed acyclic graph (DAG). The Airflow architecture includes components such as the web server, scheduler, and workers, which work together to manage the workflow. The task lifecycle includes stages such as “no status”, “scheduled”, “running”, and “success”, and tasks can be retried or rescheduled as needed. Airflow also provides a way to share data between tasks using XComs.

Code Examples

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator

default_args = {
    'owner': 'code2j',
    'depends_on_past': False,
    'start_date': datetime(2021, 7, 13),
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'our_first_dag',
    default_args=default_args,
    schedule_interval=timedelta(days=1),
)

task1 = BashOperator(
    task_id='first_task',
    bash_command='echo "Hello World, this is the first task!"',
    dag=dag,
)

This code defines a simple DAG with one task that runs a Bash command.

Lesson Summary

In this lesson, we learned how to create a Python project and install Apache Airflow. We also learned how to initialize the Airflow database and start the Airflow web server. We then defined and ran a simple DAG using the BashOperator and PythonOperator. The DAG consisted of one task that ran a Bash command, and we were able to view the task’s execution log in the Airflow web interface. We also learned about the Airflow architecture and the task lifecycle, including the different stages that a task can go through. Additionally, we learned how to share data between tasks using XComs. By the end of this lesson, we had a basic understanding of how to create and run a workflow in Airflow.

Practice Exercise

Create a new DAG that runs two tasks: one that runs a Bash command to print “Hello World”, and another that runs a Python function to print the current date and time. Use the BashOperator and PythonOperator to define the tasks, and make sure to specify the dependencies between the tasks.

What Is Next

In the next lesson, we will learn how to connect to external services such as PostgreSQL and AWS S3 using Airflow connections. We will also learn how to install Python packages in the Airflow environment and how to use Airflow hooks with examples.

Choose Language