OpenCourser brand icon
Sign in
We may earn an affiliate commission when you visit our partners.
Course image
Udacity logo

Automate Data Pipelines

Sean Murdock, Matt Swaffer, Ben Goldberg, Amanda Moran, and Valerie Scarlata

Create streamlined data pipelines with Airflow and learn best practices with Udacity's Automated Data Pipelines Training Course. Enroll today & grow your career

Prerequisite details

Read more

Create streamlined data pipelines with Airflow and learn best practices with Udacity's Automated Data Pipelines Training Course. Enroll today & grow your career

Prerequisite details

To optimize your success in this program, we've created a list of prerequisites and recommendations to help you prepare for the curriculum. Prior to enrolling, you should have the following knowledge:

  • Data modeling basics
  • Intermediate Python
  • Database fundamentals
  • Intermediate SQL
  • Amazon web services basics
  • Command line interface basics

You will also need to be able to communicate fluently and professionally in written and spoken English.

Enroll now

What's inside

Syllabus

Welcome to Automating Data Pipelines. In this lesson, you'll be introduced to the topic, prerequisites for the course, and the environment and tools you'll be using to build data pipelines.
Read more
In this lesson, you'll learn about the components of a data pipeline including Directed Acyclic Graphs (DAGs). You'll practice creating data pipelines with DAGs and Apache Airflow
This lesson creates connections between Airflow and AWS first by creating credentials, then copying S3 data, leveraging connections and hooks, and building S3 data to the Redshift DAG.
Students will learn how to track data lineage and set up data pipeline schedules, partition data to optimize pipelines, investigating Data Quality issues, and write tests to ensure data quality.
In this last lesson, students will learn how to build Pipelines with maintainability and reusability in mind. They will also learn about pipeline monitoring.
Students work on a music streaming company’s data infrastructure by creating and automating a set of data pipelines with Airflow, monitoring and debugging production pipelines

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Examines important data engineering concepts like data modeling, SQL, and AWS, which are crucial for data engineering
Leverages Airflow, a widely-used orchestration tool in the industry for building data pipelines
Suitable for junior data engineers or aspiring data professionals looking to enhance their data engineering skills
Taught by seasoned instructors with extensive experience in data engineering and data pipelines, ensuring high-quality instruction
Provides hands-on experience through interactive exercises and a capstone project, allowing learners to apply their knowledge practically
Covers essential topics in data pipeline engineering, including data lineage, data quality, and pipeline monitoring

Save this course

Save Automate Data Pipelines to your list so you can find it easily later:
Save

Activities

Coming soon We're preparing activities for Automate Data Pipelines. These are activities you can do either before, during, or after a course.

Career center

Learners who complete Automate Data Pipelines will develop knowledge and skills that may be useful to these careers:

Reading list

We've selected three books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Automate Data Pipelines.
Cover image
Secret Colors
Save
Provides a deep dive into the design and implementation of data-intensive applications, including topics such as data modeling, data storage, and data processing.
Secret Colors
Paperback
Check
Secret Colors
Kindle Edition
Check
Cover image
Kafka: The Definitive Guide
Save
Provides a comprehensive overview of data pipelines using Kafka. It covers the basics of data pipelines, including data sources, transformations, and destinations. It also discusses more advanced topics, such as scheduling, monitoring, and debugging data pipelines.
Kafka: The Definitive Guide: Real-Time Data and...
Paperback
$$
Kafka: The Definitive Guide
Kindle Edition
$$
Cover image
Hadoop: The Definitive Guide
Save
Provides a comprehensive guide to building data pipelines with Apache Hadoop, covering topics such as data ingestion, transformation, and data storage.
Hadoop: The Definitive Guide: Storage and Analysis...
Paperback
$$
Hadoop: The Definitive Guide
Kindle Edition
$$
Hadoop: The Definitive Guide
Paperback
$$
Hadoop: The Definitive Guide
Paperback
$$

Share

Help others find this course page by sharing it with your friends and followers:
Facebook
LinkedIn
Reddit
X
Link
Email

Similar courses

Here are nine courses similar to Automate Data Pipelines.
Productionalizing Data Pipelines with Apache Airflow 1
Most relevant
Introduction to Airflow
Most relevant
The Complete Hands-On Introduction to Apache Airflow
Most relevant
Building ETL and Data Pipelines with Bash, Airflow and...
Most relevant
Apache Airflow on AWS EKS: The Hands-On Guide
Most relevant
Building Pipelines for Workflow Orchestration Using...
Advanced Data Engineering
Apache Airflow: The Hands-On Guide
Orchestrating a TFX Pipeline with Airflow
Course image
Effort
4 weeks
Level
Intermediate
Via
Udacity
Instructors
Sean Murdock
Matt Swaffer
Ben Goldberg
Amanda Moran
Valerie Scarlata
Language
English
Good to know
Examines important data engineering concepts like data modeling, SQL, and AWS, which are crucial for data engineering
Leverages Airflow, a widely-used orchestration tool in the industry for building data pipelines
Suitable for junior data engineers or aspiring data professionals looking to enhance their data engineering skills
Taught by seasoned instructors with extensive experience in data engineering and data pipelines, ensuring high-quality instruction
Provides hands-on experience through interactive exercises and a capstone project, allowing learners to apply their knowledge practically
Covers essential topics in data pipeline engineering, including data lineage, data quality, and pipeline monitoring
Share and help others discover this course
Facebook LinkedIn X Reddit Link Email
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2024 OpenCourser