The Complete Hands-On Introduction to Apache Airflow from Udemy

What's inside

Learning objectives

Create plugins to add functionalities to apache airflow.
Using docker with airflow and different executors
Master core functionalities such as dags, operators, tasks, workflows, etc
Understand and apply advanced concepts of apache airflow such as xcoms, branching and subdags.

The difference between sequential, local and celery executors, how do they work and how can you use them.
Use apache airflow in a big data ecosystem with hive, postgresql, elasticsearch etc.
Install and configure apache airflow
Think, answer and implement solutions using airflow to real data processing problems

Create plugins to add functionalities to apache airflow.
Using docker with airflow and different executors
Master core functionalities such as dags, operators, tasks, workflows, etc
Understand and apply advanced concepts of apache airflow such as xcoms, branching and subdags.
The difference between sequential, local and celery executors, how do they work and how can you use them.
Use apache airflow in a big data ecosystem with hive, postgresql, elasticsearch etc.
Install and configure apache airflow
Think, answer and implement solutions using airflow to real data processing problems

Syllabus

Just a quick introduction.

Prerequisites

Course Objectives

Who I am?

Development Environment

Overview of Airflow, how it works and why and when to use it.

Why Airflow?

What is Airflow?

Core Components

Core Concepts

Airflow is not...

Single Node Architecture

Multi Node Architecture

How does it work?

[Practice] Installing Apache Airflow

What is Docker?

The docker-compose file

Quiz Time!

Key Takeaways

The important views of the Airflow UI

The DAGs View

Run your first DAG

The Grid View

The Graph View

The Landing Times View

The Calendar View

The Gantt View

The Code View

Wrap up!

Quiz!

You will be able to create your own DAGs using all the basic components of Apache Airflow

The Project

Advices

What is a DAG?

DAG Skeleton

Define your first DAG

What is an Operator?

Providers

Create a Table

Create a connection

Implement the create table task

The secret weapon!

What is a Sensor?

Is the API available?

Implement the sensor is_api_available

Extract users

Implement extract users

Process users

Before running process_user

Implement process_user

What is a Hook?

Store users

Implement store_user

Order matters!

Your DAG in action!

DAG Scheduling

Backfilling: How does it work?

The New Way of Scheduling DAGs

Why do you need that feature?

What is a Dataset?

Adios schedule_interval!

Create the Producer DAG

Create the Consumer DAG

Track your Datasets with the new view!

Wait for many datasets

Dataset limitations

You will be able to choose the right executor associated with the right backend according to your needs.

What's an executor?

The default config

The Sequential Executor

The Local Executor

The Celery Executor

The current config

Add the DAG parallel_dag.py into the dags folder

Monitor your tasks with Flower

Remove DAG examples

Running tasks on Celery Workers

What is a queue?

Add a new Celery Worker

Create a queue to better distribute tasks

Send a task to a specific queue

Concurrency, the parameters you must know!

You will be able to use some advanced concepts in Airflow to make your DAG even more dynamic.

Adios repetitive patterns

Add the DAG group_dag.py

How to use SubDAGs?

[Practice] Group tasks with SubDAGs!

Adios SubDAGs, welcome TaskGroups!

Group tasks with TaskGroups!

Add the DAG xcom_dag.py

Sharing data between tasks with XComs

[Practice] XComs in action!

Choosing a specific path in your DAG

[Practice] Executing a task according to a condition

Trigger rules or how tasks get triggered

Fixing the BranchPythonOperator with trigger rules

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Expands your knowledge of Apache Airflow, a vital tool in ETL management

Taught by Marc Lamberti, a recognized expert in Apache Airflow

Builds a foundation for using plugins to expand Airflow functionality

Demonstrates effective use of Docker with Airflow and various executors

Develops skills in creating, customizing, and managing Apache Airflow DAGs

Reinforces knowledge of Apache Airflow's core functionalities

Reviews summary

Hands-on airflow course

According to students, this in-depth course provides a state-of-art look at Apache Airflow. Beyond the beginner level material, this course teaches real-world applications of Airflow and expert practices. Students say that the instructor is experienced with the material.

Learners praise the advanced material including real-world applications.

"The instructor seems to have a lot of experience with this tool, you can check out his blog + I recommend buying a new course from Marc: "the-ultimate-hands-on-course-to-master-apache-airflow". It goes much more in depth and shows state-of-the-art options to deploy Airflow with KubernetesExecutor."

The course instructor is praised by students for their deep knowledge of the material.

Students say that this course covers advanced material.

"A very good introduction to Airflow."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in The Complete Hands-On Introduction to Apache Airflow with these activities:

Review the book 'Designing Data-Intensive Applications'

Show steps

Review key concepts in data engineering and big data to strengthen your understanding of the topics covered in this course.

View Secret Colors on Amazon

Show steps

Read the book's introduction and first three chapters.
Summarize the main points of each chapter in your own words.
Identify any areas where you need further clarification.
Discuss the book's concepts with a study partner or online forum.

Solve Apache Airflow coding challenges

Show steps

Sharpen your Apache Airflow skills by solving coding challenges that test your understanding of core concepts and syntax.

Browse courses on Apache Airflow

Show steps

Find coding challenges on platforms like LeetCode or HackerRank.
Attempt to solve the challenges on your own.
Review the solutions and identify areas for improvement.

Follow online tutorials on Apache Airflow

Show steps

Supplement your course learning with online tutorials that provide step-by-step guidance on specific Apache Airflow topics.

Show steps

Identify areas where you need additional support.
Find reputable online tutorials that cover those topics.
Follow the tutorials carefully and implement the examples.

Four other activities

Expand to see all activities and additional details

Show all seven activities

Create a data pipeline using Apache Airflow

Show steps

Build a practical understanding of Apache Airflow by implementing a data pipeline for a real-world scenario.

Show steps

Choose a data source and define the data processing tasks.
Design the Airflow DAG and define the operators.
Implement the DAG and test its functionality.
Deploy the pipeline and monitor its performance.

Attend Apache Airflow meetups or conferences

Show steps

Connect with other Apache Airflow users and experts to exchange knowledge and learn about best practices.

Show steps

Find Apache Airflow meetups or conferences in your area.
Register for the event and attend the sessions.
Network with other attendees and share your experiences.

Build a personal Apache Airflow project

Show steps

Apply your Apache Airflow knowledge by building a project that solves a real-world problem or explores a specific topic.

Show steps

Define the scope and goals of your project.
Design the architecture and implementation plan.
Develop and test your project.
Deploy and maintain your project.

Contribute to the Apache Airflow community

Show steps

Gain hands-on experience and give back to the Apache Airflow community by contributing to projects or forums.

Show steps

Join the Apache Airflow community forums.
Identify areas where you can contribute, such as answering questions or reporting issues.
Submit your contributions and engage with other community members.

Career center

Learners who complete The Complete Hands-On Introduction to Apache Airflow will develop knowledge and skills that may be useful to these careers:

Project Manager

A Project Manager plans, executes, and closes projects. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Project Manager by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Project Management career.

See salaries and explore the career path for Project Manager

Data Scientist

A Data Scientist uses scientific methods to extract knowledge and insights from data. They use Apache Airflow to build out data pipelines and manage data. This course will allow you to build the foundation you need to become a Data Scientist by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Science career.

See salaries and explore the career path for Data Scientist

Machine Learning Engineer

A Machine Learning Engineer designs, builds, and maintains machine learning systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Machine Learning Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Machine Learning Engineering career.

See salaries and explore the career path for Machine Learning Engineer

Business Analyst

A Business Analyst analyzes business processes and develops solutions to improve efficiency. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Business Analyst by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Business Analyst career.

See salaries and explore the career path for Business Analyst

Data Architect

A Data Architect designs, builds, and maintains data systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Data Architect by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Architect career.

See salaries and explore the career path for Data Architect

Software Engineer

A Software Engineer designs, develops, and maintains software systems. They work within a variety of industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Software Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Software Engineering career.

See salaries and explore the career path for Software Engineer

Data Analyst

A Data Analyst transforms raw data into insights that companies can use to make better decisions. They build out dashboards and data visualizations, which can be built using Apache Airflow. This course will allow you to build the foundation you need to become a Data Analyst by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Analyst career.

See salaries and explore the career path for Data Analyst

Database Administrator

A Database Administrator designs, builds, and maintains database systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Database Administrator by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Database Administrator career.

See salaries and explore the career path for Database Administrator

Product Manager

A Product Manager plans, develops, and launches products. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Product Manager by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Product Management career.

See salaries and explore the career path for Product Manager

Data Engineer

A Data Engineer designs, builds, and maintains data pipelines. They work within big data ecosystems to build out data infrastructure. Often, Data Engineers will also develop solutions using Apache Airflow. This course will allow you to build the foundation you need to become a Data Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Engineering career.

See salaries and explore the career path for Data Engineer

Cloud Engineer

A Cloud Engineer designs, builds, and maintains cloud computing systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Cloud Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Cloud Engineering career.

See salaries and explore the career path for Cloud Engineer

ETL Developer

An ETL Developer designs, builds, and maintains ETL (Extract, Transform, Load) systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become an ETL Developer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your ETL Developer career.

See salaries and explore the career path for ETL Developer

Big Data Engineer

A Big Data Engineer designs, builds, and maintains big data systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Big Data Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Big Data Engineering career.

See salaries and explore the career path for Big Data Engineer

Data Integration Specialist

A Data Integration Specialist integrates data from different sources into a single system. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Data Integration Specialist by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Integration Specialist career.

See salaries and explore the career path for Data Integration Specialist

Data Analytics Manager

A Data Analytics Manager plans, executes, and manages data analytics projects. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Data Analytics Manager by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Analytics Management career.

See salaries and explore the career path for Data Analytics Manager