January 16, 2021

299 words 2 mins read



Apache Airflow tutorial

repo name tuanavu/airflow-tutorial
repo link https://github.com/tuanavu/airflow-tutorial
language Jupyter Notebook
size (curr.) 157 kB
stars (curr.) 457
created 2018-10-27
license MIT License

Airflow tutorial

This is the code for Apache Airflow Tutorials playlist by Tuan Vu on Youtube


Part Title Git Tag
1 Introduction to Apache Airflow (blog post) v0.1
2 Set up airflow environment with docker (blog post) v0.2
3 Set up airflow environment using Google Cloud Composer (blog post) N/A
4 Writing your first pipeline (blog post) N/A
5 Airflow concept (blog post) N/A
6 Build a data pipeline using Google Cloud Bigquery (blog post) v0.6
7 Airflow variables (blog post) v0.7

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

  • Clone this repo
  • Install the prerequisites
  • Run the service
  • Check http://localhost:8080
  • Done! :tada:



Run the web service with docker

docker-compose up -d

# Build the image
# docker-compose up -d --build

Check http://localhost:8080/

  • docker-compose logs - Displays log output
  • docker-compose ps - List containers
  • docker-compose down - Stop containers

Other commands

If you want to run airflow sub-commands, you can do so like this:

  • docker-compose run --rm webserver airflow list_dags - List dags
  • docker-compose run --rm webserver airflow test [DAG_ID] [TASK_ID] [EXECUTION_DATE] - Test specific task

If you want to run/test python script, you can do so like this:

  • docker-compose run --rm webserver python /usr/local/airflow/dags/[PYTHON-FILE].py - Test python script

Connect to database

If you want to use Ad hoc query, make sure you’ve configured connections: Go to Admin -> Connections and Edit “postgres_default” set this values:

  • Host : postgres
  • Schema : airflow
  • Login : airflow
  • Password : airflow


comments powered by Disqus