Apache Airflow: The Hands-On Guide from Udemy

Apache Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.

It is scalable, dynamic, extensible, and modulable.

Without any doubt, mastering Airflow is becoming a must-have and an attractive skill for anyone working with data.

What you will learn in the course:

Fundamentals of Airflow are explained such as what Airflow is, how the scheduler and the web server work
The Forex Data Pipeline project is an incredible way to discover many operators in Airflow and deal with Slack, Spark, Hadoop, and more
Mastering your DAGs is a top priority, and you can play with timezones, unit test your DAGs, structure your DAG folder, and much more.
Scaling Airflow through different executors such as the Local Executor, the Celery Executor, and the Kubernetes Executor will be explained in detail. You will discover how to specialize your workers, add new workers, and what happens when a node crashes.
A Kubernetes cluster of 3 nodes will be set up with Rancher, Airflow, and the Kubernetes Executor local to run your data pipelines.
Advanced concepts will be shown through practical examples such as templating your DAGs, how to make your DAG dependent on another, what are Subdags and deadlocks, and more.
You will set up a Kubernetes cluster in the cloud with AWS EKS and Rancher to use Airflow and the Kubernetes Executor.
Monitoring Airflow is extremely important. That's why you will know how to do it with Elasticsearch and Grafana.
Security will also be addressed to make your Airflow instance compliant with your company. Specifying roles and permissions for your users with RBAC, preventing them from accessing the Airflow UI with authentication and password, data encryption, and more.

In addition:

Many practical exercises are given along the course so that you will have occasions to apply what you learn.
Best practices are stated when needed to give you the best ways of using Airflow.
Quiz are available to assess your comprehension at the end of each section.
Answering your questions fast is my top priority, and I will do my best for you.

I put a lot of effort into giving you the best content, and I hope you will enjoy it as much as I wanted to do it.

At the end of the course, you will be more confident than ever in using Airflow.

I wish you a great success.

Marc Lamberti

What's inside

Learning objectives

Coding production grade data pipelines by mastering airflow through hands-on examples
How to follow best practices with apache airflow
How to scale airflow with the local, celery and kubernetes wxecutors
How to set up monitoring with elasticsearch and grafana

How to secure airflow with authentication, crypto and the rbac ui
Core and advanced concepts with pros and limitations
Mastering dags with timezones, unit testing, backfill and catchup
Organising the dag folder and keep things clean

Coding production grade data pipelines by mastering airflow through hands-on examples
How to follow best practices with apache airflow
How to scale airflow with the local, celery and kubernetes wxecutors
How to set up monitoring with elasticsearch and grafana
How to secure airflow with authentication, crypto and the rbac ui
Core and advanced concepts with pros and limitations
Mastering dags with timezones, unit testing, backfill and catchup
Organising the dag folder and keep things clean

Syllabus

Introduction

Important Prerequisites

The Roadmap

Who I am?

Traffic lights

Read about what's good

what should give you pause

and possible dealbreakers

Teaches Apache Airflow, which is an in-demand skill for data professionals

Covers core and advanced concepts of Airflow, providing a comprehensive foundation

Provides practical examples and hands-on exercises to reinforce learning

Explores best practices for using Airflow, ensuring efficient and effective data pipelines

Offers a detailed guide to setting up and running Airflow on Kubernetes, enabling scalability and flexibility

Provides instruction on security measures for Airflow, ensuring data privacy and compliance

Reviews summary

Practical airflow mastery with hands-on labs

According to learners, this course is a highly practical and comprehensive guide to Apache Airflow. Students particularly value the hands-on labs and real-world project examples, which cover everything from DAG mastering to scaling with Celery and Kubernetes executors. Many commend the instructor for clear explanations and prompt Q&A responses, making complex topics like AWS EKS deployment accessible. While the course provides an excellent foundation for production readiness, some learners with less prior experience found the initial environment setup challenging or the pace sometimes fast. Overall, it's highly recommended for those serious about data orchestration.

Instructor is responsive and provides clear explanations.

"The instructor's quick responses on questions were a huge plus."

"Excellent course! The instructor's responsiveness in Q&A is amazing."

"Marc explains complex concepts clearly and practically."

Covers a wide range of Airflow concepts and advanced topics.

"A very comprehensive course. The sections on different executors (Celery, Kubernetes) were well-explained."

"Fantastic course! The coverage of DAGs, templating, and cross-DAG dependencies was thorough."

"As an experienced developer, I found the deep dive into Kubernetes Executor and EKS particularly useful."

Strong emphasis on practical exercises and real-world projects.

"Absolutely brilliant! The hands-on labs, especially setting up Airflow on AWS EKS with Kubernetes, were incredibly valuable."

"This course delivers exactly what it promises: a hands-on guide... The practical exercises are key to understanding Airflow deeply."

"The course is incredibly practical. The detailed explanations of DAGs, operators, and sensors are spot on. It provides a solid understanding of how to build robust data pipelines."

Course pace may be fast for learners new to the prerequisites.

"Some parts moved a bit fast for a complete beginner, but with patience and re-watching, it's manageable."

"Occasionally, the pace was a little too fast."

"The course assumes too much prior knowledge of Docker and Kubernetes."

Initial environment setup can be complex and time-consuming.

"The course content was good but the lab setup was very challenging. It took me more time to troubleshoot environment issues than to actually learn the concepts."

"Honestly, the setup process was a nightmare. The course assumes too much prior knowledge of Docker and Kubernetes."

"The installation instructions could be more robust for different OS, as I faced some issues."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Apache Airflow: The Hands-On Guide with these activities:

Explore Airflow Best Practices

Show steps

Become familiar with Airflow best practices to improve your workflow efficiency.

Browse courses on Airflow

Show steps

Review official Airflow documentation on best practices
Explore community forums and articles to gather additional insights

Run a data pipeline with the Local Executor

Show steps

Run a data pipeline with the 'Local Executor' to familiarize yourself with its functionality and execution.

Browse courses on Data Pipeline

Show steps

Install the Airflow CLI if not already installed
Run 'airflow initdb' to initialize the Airflow database
Start the Airflow webserver and scheduler with 'airflow webserver' and 'airflow scheduler'
Run 'airflow example_dags run_example_dags' to create and run example DAGs

Collaborative Airflow Pipeline Design

Show steps

Engage with peers to share knowledge, troubleshoot, and refine your Airflow pipeline designs.

Browse courses on Airflow

Show steps

Join an online community or forum dedicated to Airflow
Participate in discussions and ask questions to gain insights
Collaborate on pipeline designs and provide feedback to others

Three other activities

Expand to see all activities and additional details

Show all six activities

Hands-on Airflow Dataset Transformation

Show steps

Enhance your practical skills by working on a hands-on project involving data transformation using Airflow.

Browse courses on Airflow

Show steps

Identify a suitable dataset for transformation
Design and implement an Airflow workflow for the transformation
Test and refine your workflow to ensure accuracy and efficiency

Configure Airflow with the Kubernetes Executor

Show steps

Configure Airflow with the 'Kubernetes Executor' to learn about distributed task execution and container orchestration.

Show steps

Install Kubernetes on a server or cluster
Deploy Airflow with the Kubernetes Executor using Helm or Docker Compose
Create and run a DAG that uses the Kubernetes Executor
Monitor your DAGs and Kubernetes cluster

Personal Data Pipeline Project

Show steps

Solidify your understanding by building your own data pipeline using Airflow.

Browse courses on Airflow

Show steps

Choose a project idea that aligns with your learning objectives
Design and implement the pipeline using Airflow operators and sensors
Monitor and maintain your pipeline to ensure its reliability and efficiency

Career center

Learners who complete Apache Airflow: The Hands-On Guide will develop knowledge and skills that may be useful to these careers:

Data Engineer

Data Engineers are responsible for building and maintaining large-scale data pipelines and data infrastructure. The Apache Airflow: The Hands-On Guide course provides you with the skills and knowledge needed to develop and deploy production-grade data pipelines using Apache Airflow. This course covers everything from the basics of Airflow to advanced topics such as scaling, monitoring, and security. With this training, you'll be well-equipped to take this role.

See salaries and explore the career path for Data Engineer

Data Analyst

Data Analysts use data to solve business problems and make better decisions. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to work with data at scale. This course covers topics such as data engineering, data warehousing, and data visualization. With this training, you'll be well-positioned to contribute to data-driven decision making in your organization.

See salaries and explore the career path for Data Analyst

Machine Learning Engineer

Machine Learning Engineers build and deploy machine learning models. The Apache Airflow: The Hands-On Guide course can help data professionals to get started with deploying machine learning models into production. This course covers topics such as data engineering, model deployment, and monitoring. With this training, you'll be well-prepared to use Airflow to automate the machine learning lifecycle.

See salaries and explore the career path for Machine Learning Engineer

Cloud Architect

Cloud Architects design and manage cloud computing solutions. The Apache Airflow: The Hands-On Guide course can provide you with the skills and knowledge needed to build and manage data pipelines in the cloud. This course covers topics such as cloud computing, data engineering, and data warehousing. With this training, you'll be well-equipped to take on this role.

See salaries and explore the career path for Cloud Architect

Software Engineer

Software Engineers design, develop, and maintain software systems. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to build and maintain data pipelines. This course covers topics such as software development, data engineering, and data warehousing. With this training, you'll be well-prepared to contribute to the development of data-driven applications.

See salaries and explore the career path for Software Engineer

Database Administrator

Database Administrators manage and maintain databases. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to work with data at scale. This course covers topics such as data engineering, data warehousing, and data visualization. With this training, you'll be well-positioned to manage and maintain data pipelines.

See salaries and explore the career path for Database Administrator

Data Scientist

Data Scientists use data to solve business problems and make better decisions. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to work with data at scale. This course covers topics such as data engineering, data warehousing, and data visualization. With this training, you'll be well-prepared to contribute to data-driven decision making in your organization.

See salaries and explore the career path for Data Scientist

Business Analyst

Business Analysts use data to solve business problems and make better decisions. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to work with data at scale. This course covers topics such as data engineering, data warehousing, and data visualization. With this training, you'll be well-positioned to contribute to data-driven decision making in your organization.

See salaries and explore the career path for Business Analyst

Project Manager

Project Managers plan and execute projects. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to manage data pipeline projects. This course covers topics such as project management, data engineering, and data warehousing. With this training, you'll be well-prepared to lead data pipeline projects to success.

See salaries and explore the career path for Project Manager

Data Architect

Data Architects design and manage data systems. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to design and manage data pipelines. This course covers topics such as data architecture, data engineering, and data warehousing. With this training, you'll be well-prepared to take on this role.

See salaries and explore the career path for Data Architect

Data Warehouse Engineer

Data Warehouse Engineers build and maintain data warehouses. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to build and maintain data pipelines for data warehouses. This course covers topics such as data engineering, data warehousing, and data visualization. With this training, you'll be well-prepared to take on this role.

See salaries and explore the career path for Data Warehouse Engineer

Data Visualization Engineer

Data Visualization Engineers create visual representations of data. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to build and maintain data pipelines for data visualization. This course covers topics such as data engineering, data warehousing, and data visualization. With this training, you'll be well-prepared to take on this role.

See salaries and explore the career path for Data Visualization Engineer

DevOps Engineer

DevOps Engineers automate and manage the software development process. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to automate and manage data pipelines. This course covers topics such as DevOps, data engineering, and data warehousing. With this training, you'll be well-prepared to take on this role.

See salaries and explore the career path for DevOps Engineer

Systems Analyst

Systems Analysts design and implement computer systems. The Apache Airflow: The Hands-On Guide course can help you develop the skills and knowledge needed to design and implement data pipelines. This course covers topics such as systems analysis, data engineering, and data warehousing. With this training, you'll be well-prepared to take on this role.

See salaries and explore the career path for Systems Analyst

Network Administrator

Network Administrators manage and maintain computer networks. The Apache Airflow: The Hands-On Guide course may help you develop the skills and knowledge needed to manage and maintain data pipelines. This course covers topics such as networking, data engineering, and data warehousing. With this training, you'll be well-prepared to take on this role.

See salaries and explore the career path for Network Administrator