We may earn an affiliate commission when you visit our partners.
The Complete Hands-On Introduction to Apache Airflow
Marc Lamberti
Marc Lamberti
Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.
Read more
Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.
Read more
Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.
In this course you are going to learn everything you need to start using Apache Airflow through theory and pratical videos. Starting from very basic notions such as, what is Airflow and how it works, we will dive into advanced concepts such as, how to create plugins and make real dynamic pipelines.
Register for this course and see more details by visiting:
OpenCourser.com/course/0rxwnd/the
What's inside
Learning objectives
- Create plugins to add functionalities to apache airflow.
- Using docker with airflow and different executors
- Master core functionalities such as dags, operators, tasks, workflows, etc
- Understand and apply advanced concepts of apache airflow such as xcoms, branching and subdags.
- The difference between sequential, local and celery executors, how do they work and how can you use them.
- Use apache airflow in a big data ecosystem with hive, postgresql, elasticsearch etc.
- Install and configure apache airflow
- Think, answer and implement solutions using airflow to real data processing problems
- Create plugins to add functionalities to apache airflow.
- Using docker with airflow and different executors
- Master core functionalities such as dags, operators, tasks, workflows, etc
- Understand and apply advanced concepts of apache airflow such as xcoms, branching and subdags.
- The difference between sequential, local and celery executors, how do they work and how can you use them.
- Use apache airflow in a big data ecosystem with hive, postgresql, elasticsearch etc.
- Install and configure apache airflow
- Think, answer and implement solutions using airflow to real data processing problems
Syllabus
Just a quick introduction.
Prerequisites
Course Objectives
Who I am?
Read more
Syllabus
Just a quick introduction.
Prerequisites
Course Objectives
Who I am?
Read more
Development Environment
Overview of Airflow, how it works and why and when to use it.
Why Airflow?
What is Airflow?
Core Components
Core Concepts
Airflow is not...
Single Node Architecture
Multi Node Architecture
How does it work?
[Practice] Installing Apache Airflow
What is Docker?
The docker-compose file
Quiz Time!
Key Takeaways
The important views of the Airflow UI
The DAGs View
Run your first DAG
The Grid View
The Graph View
The Landing Times View
The Calendar View
The Gantt View
The Code View
Wrap up!
Quiz!
You will be able to create your own DAGs using all the basic components of Apache Airflow
The Project
Advices
What is a DAG?
DAG Skeleton
Define your first DAG
What is an Operator?
Providers
Create a Table
Create a connection
Implement the create table task
The secret weapon!
What is a Sensor?
Is the API available?
Implement the sensor is_api_available
Extract users
Implement extract users
Process users
Before running process_user
Implement process_user
What is a Hook?
Store users
Implement store_user
Order matters!
Your DAG in action!
DAG Scheduling
Backfilling: How does it work?
The New Way of Scheduling DAGs
Why do you need that feature?
What is a Dataset?
Adios schedule_interval!
Create the Producer DAG
Create the Consumer DAG
Track your Datasets with the new view!
Wait for many datasets
Dataset limitations
You will be able to choose the right executor associated with the right backend according to your needs.
What's an executor?
The default config
The Sequential Executor
The Local Executor
The Celery Executor
The current config
Add the DAG parallel_dag.py into the dags folder
Monitor your tasks with Flower
Remove DAG examples
Running tasks on Celery Workers
What is a queue?
Add a new Celery Worker
Create a queue to better distribute tasks
Send a task to a specific queue
Concurrency, the parameters you must know!
You will be able to use some advanced concepts in Airflow to make your DAG even more dynamic.
Adios repetitive patterns
Add the DAG group_dag.py
How to use SubDAGs?
[Practice] Group tasks with SubDAGs!
Adios SubDAGs, welcome TaskGroups!
Group tasks with TaskGroups!
Add the DAG xcom_dag.py
Sharing data between tasks with XComs
[Practice] XComs in action!
Choosing a specific path in your DAG
[Practice] Executing a task according to a condition
Trigger rules or how tasks get triggered
Fixing the BranchPythonOperator with trigger rules
Good to know
Good to know
Know what's good ,
what to watch for , and
possible dealbreakers
Expands your knowledge of Apache Airflow, a vital tool in ETL management
Taught by Marc Lamberti, a recognized expert in Apache Airflow
Builds a foundation for using plugins to expand Airflow functionality
Demonstrates effective use of Docker with Airflow and various executors
Develops skills in creating, customizing, and managing Apache Airflow DAGs
Reinforces knowledge of Apache Airflow's core functionalities
Save this course
Save The Complete Hands-On Introduction to Apache Airflow to your list so you can find it easily later:
Save
Save
Reviews summary
Hands-on airflow course
According to students, this in-depth course provides a state-of-art look at Apache Airflow. Beyond the beginner level material, this course teaches real-world applications of Airflow and expert practices. Students say that the instructor is experienced with the material.
Learners praise the advanced material including real-world applications.
"The instructor seems to have a lot of experience with this tool, you can check out his blog + I recommend buying a new course from Marc: "the-ultimate-hands-on-course-to-master-apache-airflow". It goes much more in depth and shows state-of-the-art options to deploy Airflow with KubernetesExecutor."
The course instructor is praised by students for their deep knowledge of the material.
"The instructor seems to have a lot of experience with this tool, you can check out his blog + I recommend buying a new course from Marc: "the-ultimate-hands-on-course-to-master-apache-airflow". It goes much more in depth and shows state-of-the-art options to deploy Airflow with KubernetesExecutor."
Students say that this course covers advanced material.
"A very good introduction to Airflow."
"The instructor seems to have a lot of experience with this tool, you can check out his blog + I recommend buying a new course from Marc: "the-ultimate-hands-on-course-to-master-apache-airflow". It goes much more in depth and shows state-of-the-art options to deploy Airflow with KubernetesExecutor."
Activities
Be better prepared
before
your course. Deepen your understanding
during
and
after
it. Supplement your coursework and achieve mastery of the topics covered
in The Complete Hands-On Introduction to Apache Airflow with these
activities:
Review the book 'Designing Data-Intensive Applications'
Show steps
Review key concepts in data engineering and big data to strengthen your understanding of the topics covered in this course.
View
Secret Colors
on Amazon
Show steps
-
Read the book's introduction and first three chapters.
-
Summarize the main points of each chapter in your own words.
-
Identify any areas where you need further clarification.
-
Discuss the book's concepts with a study partner or online forum.
Solve Apache Airflow coding challenges
Show steps
Sharpen your Apache Airflow skills by solving coding challenges that test your understanding of core concepts and syntax.
Browse courses on
Apache Airflow
Show steps
-
Find coding challenges on platforms like LeetCode or HackerRank.
-
Attempt to solve the challenges on your own.
-
Review the solutions and identify areas for improvement.
Follow online tutorials on Apache Airflow
Show steps
Supplement your course learning with online tutorials that provide step-by-step guidance on specific Apache Airflow topics.
Show steps
-
Identify areas where you need additional support.
-
Find reputable online tutorials that cover those topics.
-
Follow the tutorials carefully and implement the examples.
Four other activities
Expand to see all activities and additional details
Show all seven activities
Create a data pipeline using Apache Airflow
Show steps
Build a practical understanding of Apache Airflow by implementing a data pipeline for a real-world scenario.
Show steps
-
Choose a data source and define the data processing tasks.
-
Design the Airflow DAG and define the operators.
-
Implement the DAG and test its functionality.
-
Deploy the pipeline and monitor its performance.
Attend Apache Airflow meetups or conferences
Show steps
Connect with other Apache Airflow users and experts to exchange knowledge and learn about best practices.
Show steps
-
Find Apache Airflow meetups or conferences in your area.
-
Register for the event and attend the sessions.
-
Network with other attendees and share your experiences.
Build a personal Apache Airflow project
Show steps
Apply your Apache Airflow knowledge by building a project that solves a real-world problem or explores a specific topic.
Show steps
-
Define the scope and goals of your project.
-
Design the architecture and implementation plan.
-
Develop and test your project.
-
Deploy and maintain your project.
Contribute to the Apache Airflow community
Show steps
Gain hands-on experience and give back to the Apache Airflow community by contributing to projects or forums.
Show steps
-
Join the Apache Airflow community forums.
-
Identify areas where you can contribute, such as answering questions or reporting issues.
-
Submit your contributions and engage with other community members.
Review the book 'Designing Data-Intensive Applications'
Show steps
Review key concepts in data engineering and big data to strengthen your understanding of the topics covered in this course.
View
Secret Colors
on Amazon
Show steps
Show steps
Show steps
- Read the book's introduction and first three chapters.
- Summarize the main points of each chapter in your own words.
- Identify any areas where you need further clarification.
- Discuss the book's concepts with a study partner or online forum.
Solve Apache Airflow coding challenges
Show steps
Sharpen your Apache Airflow skills by solving coding challenges that test your understanding of core concepts and syntax.
Browse courses on
Apache Airflow
Show steps
Show steps
Show steps
- Find coding challenges on platforms like LeetCode or HackerRank.
- Attempt to solve the challenges on your own.
- Review the solutions and identify areas for improvement.
Follow online tutorials on Apache Airflow
Show steps
Supplement your course learning with online tutorials that provide step-by-step guidance on specific Apache Airflow topics.
Show steps
Show steps
Show steps
- Identify areas where you need additional support.
- Find reputable online tutorials that cover those topics.
- Follow the tutorials carefully and implement the examples.
Four other activities
Expand to see all activities and additional details
Show all seven activities
Create a data pipeline using Apache Airflow
Show steps
Build a practical understanding of Apache Airflow by implementing a data pipeline for a real-world scenario.
Show steps
Show steps
Show steps
- Choose a data source and define the data processing tasks.
- Design the Airflow DAG and define the operators.
- Implement the DAG and test its functionality.
- Deploy the pipeline and monitor its performance.
Attend Apache Airflow meetups or conferences
Show steps
Connect with other Apache Airflow users and experts to exchange knowledge and learn about best practices.
Show steps
Show steps
Show steps
- Find Apache Airflow meetups or conferences in your area.
- Register for the event and attend the sessions.
- Network with other attendees and share your experiences.
Build a personal Apache Airflow project
Show steps
Apply your Apache Airflow knowledge by building a project that solves a real-world problem or explores a specific topic.
Show steps
Show steps
Show steps
- Define the scope and goals of your project.
- Design the architecture and implementation plan.
- Develop and test your project.
- Deploy and maintain your project.
Contribute to the Apache Airflow community
Show steps
Gain hands-on experience and give back to the Apache Airflow community by contributing to projects or forums.
Show steps
Show steps
Show steps
- Join the Apache Airflow community forums.
- Identify areas where you can contribute, such as answering questions or reporting issues.
- Submit your contributions and engage with other community members.
Career center
Learners who complete The Complete Hands-On Introduction to Apache Airflow will develop knowledge and skills
that may be useful to these careers:
Project Manager
Project Manager
A Project Manager plans, executes, and closes projects. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Project Manager by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Project Management career.
Data Scientist
Data Scientist
A Data Scientist uses scientific methods to extract knowledge and insights from data. They use Apache Airflow to build out data pipelines and manage data. This course will allow you to build the foundation you need to become a Data Scientist by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Science career.
Machine Learning Engineer
Machine Learning Engineer
A Machine Learning Engineer designs, builds, and maintains machine learning systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Machine Learning Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Machine Learning Engineering career.
Business Analyst
Business Analyst
A Business Analyst analyzes business processes and develops solutions to improve efficiency. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Business Analyst by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Business Analyst career.
Data Architect
Data Architect
A Data Architect designs, builds, and maintains data systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Data Architect by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Architect career.
Software Engineer
Software Engineer
A Software Engineer designs, develops, and maintains software systems. They work within a variety of industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Software Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Software Engineering career.
Data Analyst
Data Analyst
A Data Analyst transforms raw data into insights that companies can use to make better decisions. They build out dashboards and data visualizations, which can be built using Apache Airflow. This course will allow you to build the foundation you need to become a Data Analyst by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Analyst career.
Database Administrator
Database Administrator
A Database Administrator designs, builds, and maintains database systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Database Administrator by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Database Administrator career.
Product Manager
Product Manager
A Product Manager plans, develops, and launches products. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Product Manager by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Product Management career.
Data Engineer
Data Engineer
A Data Engineer designs, builds, and maintains data pipelines. They work within big data ecosystems to build out data infrastructure. Often, Data Engineers will also develop solutions using Apache Airflow. This course will allow you to build the foundation you need to become a Data Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Engineering career.
Cloud Engineer
Cloud Engineer
A Cloud Engineer designs, builds, and maintains cloud computing systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Cloud Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Cloud Engineering career.
ETL Developer
ETL Developer
An ETL Developer designs, builds, and maintains ETL (Extract, Transform, Load) systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become an ETL Developer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your ETL Developer career.
Big Data Engineer
Big Data Engineer
A Big Data Engineer designs, builds, and maintains big data systems. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Big Data Engineer by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Big Data Engineering career.
Data Integration Specialist
Data Integration Specialist
A Data Integration Specialist integrates data from different sources into a single system. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Data Integration Specialist by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Integration Specialist career.
Data Analytics Manager
Data Analytics Manager
A Data Analytics Manager plans, executes, and manages data analytics projects. They work within various industries, and often use Apache Airflow. This course will allow you to build the foundation you need to become a Data Analytics Manager by helping you build out the skills needed to use Airflow to create plugins, use Docker with Airflow, use core functionalities, and use advanced concepts. This course may help you get into or advance your Data Analytics Management career.
For more career information including salaries, visit:
OpenCourser.com/course/0rxwnd/the
Reading list
We've selected five books
that we think will supplement your
learning. Use these to
develop background knowledge, enrich your coursework, and gain a
deeper understanding of the topics covered in
The Complete Hands-On Introduction to Apache Airflow.
Provides a broader perspective on data-intensive applications, including architectural patterns and best practices that can be applied to Apache Airflow.
Provides an introduction to Python and its libraries for data analysis, which are essential for working with Apache Airflow.
Comprehensive guide to using Airflow to build and manage data pipelines. It covers topics such as data ingestion, data processing, and data analysis. It valuable resource for anyone who wants to use Airflow to build and manage data pipelines.
Comprehensive guide to using Airflow to build and manage data pipelines. It covers topics such as data ingestion, data processing, and data analysis. It valuable resource for anyone who wants to use Airflow to build and manage data pipelines.
Guide to using Apache Airflow for deep learning. It covers topics such as using Airflow to build and manage deep learning pipelines, using Airflow with different deep learning tools and frameworks, and deploying Airflow in production.
For more information about how these books relate to this course, visit:
OpenCourser.com/course/0rxwnd/the
Share
Similar courses
Here are nine courses similar to
The Complete Hands-On Introduction to Apache Airflow.
Apache Airflow: The Hands-On Guide
OpenCourser.com/course/mxps76/apache
Apache Airflow: The Hands-On Guide
Most relevant
Productionalizing Data Pipelines with Apache Airflow 1
OpenCourser.com/course/zeeh8u/productionalizing
Productionalizing Data Pipelines with Apache Airflow 1
Most relevant
Introduction to Airflow
OpenCourser.com/course/lb4584/introduction
Introduction to Airflow
Most relevant
Building Pipelines for Workflow Orchestration Using Google Composer
OpenCourser.com/course/hliavq/building
Building Pipelines for Workflow Orchestration Using...
Most relevant
Orchestrating a TFX Pipeline with Airflow
OpenCourser.com/course/ybwy8f/orchestrating
Orchestrating a TFX Pipeline with Airflow
Most relevant
Apache Airflow on AWS EKS: The Hands-On Guide
OpenCourser.com/course/riugn3/apache
Apache Airflow on AWS EKS: The Hands-On Guide
Most relevant
Building ETL and Data Pipelines with Bash, Airflow and Kafka
OpenCourser.com/course/01t16b/building
Building ETL and Data Pipelines with Bash, Airflow and...
Most relevant
Architecting Serverless Big Data Solutions Using Google Dataflow
OpenCourser.com/course/auqu3x/architecting
Architecting Serverless Big Data Solutions Using Google...
Most relevant
Prep for Microsoft Azure Data Engineer Associate Cert DP-203
OpenCourser.com/course/312svh/prep
Prep for Microsoft Azure Data Engineer Associate Cert DP...
Most relevant
Effort
3.5 total hours
Via
Udemy
Instructor
Marc Lamberti
Language
English
Good to know
Expands your knowledge of Apache Airflow, a vital tool in ETL management
Taught by Marc Lamberti, a recognized expert in Apache Airflow
Builds a foundation for using plugins to expand Airflow functionality
Demonstrates effective use of Docker with Airflow and various executors
Develops skills in creating, customizing, and managing Apache Airflow DAGs
Reinforces knowledge of Apache Airflow's core functionalities
Our mission
OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.
Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.
Find this site helpful? Tell a friend about us.
Affiliate disclosure
We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.
Your purchases help us maintain our catalog and keep our servers humming without ads.
Thank you for supporting OpenCourser.
© 2016 - 2024 OpenCourser