We may earn an affiliate commission when you visit our partners.
Noah Gift
and
Alfredo Deza
In this advanced course, you will gain practical expertise in scaling data engineering systems using cutting-edge tools and techniques. This course is designed for data scientists, data engineers, and anyone with a foundational understanding of data handling who desires to escalate their skills to handle larger, more complex datasets efficiently.
Throughout the course, you'll master the application of technologies such as Celery with RabbitMQ for scalable data consumption, Apache Airflow for optimized workflow management, and Vector and Graph databases for robust data management at scale.
Read more
In this advanced course, you will gain practical expertise in scaling data engineering systems using cutting-edge tools and techniques. This course is designed for data scientists, data engineers, and anyone with a foundational understanding of data handling who desires to escalate their skills to handle larger, more complex datasets efficiently.
Throughout the course, you'll master the application of technologies such as Celery with RabbitMQ for scalable data consumption, Apache Airflow for optimized workflow management, and Vector and Graph databases for robust data management at scale.
Read more
In this advanced course, you will gain practical expertise in scaling data engineering systems using cutting-edge tools and techniques. This course is designed for data scientists, data engineers, and anyone with a foundational understanding of data handling who desires to escalate their skills to handle larger, more complex datasets efficiently.
Throughout the course, you'll master the application of technologies such as Celery with RabbitMQ for scalable data consumption, Apache Airflow for optimized workflow management, and Vector and Graph databases for robust data management at scale.
The course will culminate with hands-on projects that offer real-world experience, where you'll put your acquired skills to test in solving data engineering challenges. You will not only learn to create scalable data systems but also to analyze their performance and make necessary adjustments for optimum results.
This invaluable experience in advanced data engineering techniques will prepare you for the demanding tasks of handling massive datasets, streamlining complex workflows, and optimizing data operations for businesses of any scale.
Register for this course and see more details by visiting:
OpenCourser.com/course/swu1x4/advanced
What's inside
Syllabus
Queues and Databases-RabbitMQ and MySQL
This week you will learn about databases and queues. You will find out the purpose and components of RabbitMQ including its use of queues and integration with Celery. Through hands-on exercises, they will gain experience connecting Celery to RabbitMQ within a Flask application and implementing task patterns like fire and forget and result retrieval. The course also covers core MySQL skills like interacting via the command line interface, manipulating databases, and integrating with Python web apps. By the end, students will have a foundational understanding of RabbitMQ, Celery, and MySQL that allows them to start building modern, asynchronous applications backed by a database.
Read more
Syllabus
Queues and Databases-RabbitMQ and MySQL
This week you will learn about databases and queues. You will find out the purpose and components of RabbitMQ including its use of queues and integration with Celery. Through hands-on exercises, they will gain experience connecting Celery to RabbitMQ within a Flask application and implementing task patterns like fire and forget and result retrieval. The course also covers core MySQL skills like interacting via the command line interface, manipulating databases, and integrating with Python web apps. By the end, students will have a foundational understanding of RabbitMQ, Celery, and MySQL that allows them to start building modern, asynchronous applications backed by a database.
Read more
Optimizing Workflow Management at Scale with Apache Airflow
Achieving Scalability with Vector, Graph, and Key/Value Databases
This week we explore vector and graph databases, powerful tools for managing and extracting insights from large, complex datasets. As data volumes continue to grow, scalability is crucial. We'll learn how vector and graph databases can efficiently store data while maintaining relationships, enabling more advanced analytics. Through real-world examples, you'll see how these databases unlock scalability for machine learning, fraud detection, social networks, and more.
Real-world Advanced Data Engineering Projects
In this final week, you will work on advanced real-world data engineering projects, applying everything you've learned. You'll encounter complex data challenges and devise solutions using the latest tools and techniques. This is an opportunity to bring together data engineering concepts covered throughout the course and implement them holistically to deliver impactful outcomes.
Good to know
Good to know
Know what's good ,
what to watch for , and
possible dealbreakers
Explores RabbitMQ and MySQL databases, industry-standard tools widely used in modern web applications
Covers Apache Airflow, a leading workflow management platform, strengthening your ability to orchestrate complex data pipelines efficiently
Introduces Vector and Graph databases, specialized tools that unlock scalability and enhanced data management capabilities for complex datasets, extending your skillset to handle diverse data scenarios
Through hands-on projects, learners can apply their acquired knowledge to real-world data engineering challenges, enhancing their practical problem-solving skills
Taught by Noah Gift and Alfredo Deza, experienced data engineering professionals, providing learners with access to valuable insights and industry best practices
Suitable for data engineers, data scientists, and those with a grounding in data handling, aligning with their professional growth and skill development
Save this course
Save Advanced Data Engineering to your list so you can find it easily later:
Save
Save
Activities
Be better prepared
before
your course. Deepen your understanding
during
and
after
it. Supplement your coursework and achieve mastery of the topics covered
in Advanced Data Engineering with these
activities:
Organize your course materials
Show steps
Organizing your course materials will help you stay on top of the content and make it easier to review later.
Browse courses on
Organization
Show steps
-
Create folders for each module or topic covered in the course.
-
File away lecture notes, readings, assignments, and other materials in the appropriate folders.
-
Regularly review your organized materials to reinforce your learning.
Review basics
Show steps
Reviewing the basic concepts will help you better understand the advanced topics covered in the course.
Browse courses on
Databases
Show steps
-
Go over notes from previous courses or textbooks on databases and queues.
-
Set up a MySQL database and practice basic operations like creating tables, inserting data, and querying data.
-
Install and configure RabbitMQ and Celery, and practice sending and receiving messages.
Organize and review course resources
Show steps
Review course materials to build a foundational understanding of data engineering concepts such as queues, databases, and optimization.
Show steps
-
Gather and organize class materials
-
Identify key concepts and topics to review
-
Review lecture notes, assignments, and quizzes
Four other activities
Expand to see all activities and additional details
Show all seven activities
Connect with experienced data engineers
Show steps
Connecting with experienced data engineers will provide you with valuable insights and guidance to enhance your learning.
Browse courses on
Mentorship
Show steps
-
Attend industry events and meetups related to data engineering.
-
Reach out to data engineers on LinkedIn or other professional networking platforms.
-
Ask for informational interviews or mentorship opportunities.
Follow tutorials on Apache Airflow
Show steps
Apache Airflow is a powerful tool for managing complex workflows. Following tutorials will help you gain hands-on experience with it.
Browse courses on
Apache Airflow
Show steps
-
Find tutorials on Apache Airflow from reputable sources.
-
Follow the tutorials step-by-step and try to implement the concepts in your own projects.
-
Join online forums or communities to connect with other Apache Airflow users and learn from their experiences.
Solve data engineering problems on Leetcode or HackerRank
Show steps
Solving data engineering problems on platforms like Leetcode or HackerRank will help you develop your problem-solving skills and practical knowledge.
Show steps
-
Sign up for an account on Leetcode or HackerRank.
-
Start solving data engineering problems under timed conditions.
-
Review your solutions and learn from your mistakes.
-
Aim to solve problems of varying difficulty levels to challenge yourself.
Build a data pipeline using Airflow
Show steps
Building a data pipeline using Airflow will give you practical experience in applying the concepts you learn in the course.
Browse courses on
Apache Airflow
Show steps
-
Define the requirements and scope of your data pipeline.
-
Design the architecture of your pipeline, including data sources, transformations, and destinations.
-
Use Airflow to orchestrate the execution of your pipeline.
-
Monitor and maintain your pipeline to ensure it runs smoothly and efficiently.
Organize your course materials
Show steps
Organizing your course materials will help you stay on top of the content and make it easier to review later.
Browse courses on
Organization
Show steps
Show steps
Show steps
- Create folders for each module or topic covered in the course.
- File away lecture notes, readings, assignments, and other materials in the appropriate folders.
- Regularly review your organized materials to reinforce your learning.
Review basics
Show steps
Reviewing the basic concepts will help you better understand the advanced topics covered in the course.
Browse courses on
Databases
Show steps
Show steps
Show steps
- Go over notes from previous courses or textbooks on databases and queues.
- Set up a MySQL database and practice basic operations like creating tables, inserting data, and querying data.
- Install and configure RabbitMQ and Celery, and practice sending and receiving messages.
Organize and review course resources
Show steps
Review course materials to build a foundational understanding of data engineering concepts such as queues, databases, and optimization.
Show steps
Show steps
Show steps
- Gather and organize class materials
- Identify key concepts and topics to review
- Review lecture notes, assignments, and quizzes
Four other activities
Expand to see all activities and additional details
Show all seven activities
Connect with experienced data engineers
Show steps
Connecting with experienced data engineers will provide you with valuable insights and guidance to enhance your learning.
Browse courses on
Mentorship
Show steps
Show steps
Show steps
- Attend industry events and meetups related to data engineering.
- Reach out to data engineers on LinkedIn or other professional networking platforms.
- Ask for informational interviews or mentorship opportunities.
Follow tutorials on Apache Airflow
Show steps
Apache Airflow is a powerful tool for managing complex workflows. Following tutorials will help you gain hands-on experience with it.
Browse courses on
Apache Airflow
Show steps
Show steps
Show steps
- Find tutorials on Apache Airflow from reputable sources.
- Follow the tutorials step-by-step and try to implement the concepts in your own projects.
- Join online forums or communities to connect with other Apache Airflow users and learn from their experiences.
Solve data engineering problems on Leetcode or HackerRank
Show steps
Solving data engineering problems on platforms like Leetcode or HackerRank will help you develop your problem-solving skills and practical knowledge.
Show steps
Show steps
Show steps
- Sign up for an account on Leetcode or HackerRank.
- Start solving data engineering problems under timed conditions.
- Review your solutions and learn from your mistakes.
- Aim to solve problems of varying difficulty levels to challenge yourself.
Build a data pipeline using Airflow
Show steps
Building a data pipeline using Airflow will give you practical experience in applying the concepts you learn in the course.
Browse courses on
Apache Airflow
Show steps
Show steps
Show steps
- Define the requirements and scope of your data pipeline.
- Design the architecture of your pipeline, including data sources, transformations, and destinations.
- Use Airflow to orchestrate the execution of your pipeline.
- Monitor and maintain your pipeline to ensure it runs smoothly and efficiently.
Career center
Learners who complete Advanced Data Engineering will develop knowledge and skills
that may be useful to these careers:
Cloud Data Engineer
Cloud Data Engineer
A Cloud Data Engineer designs, builds, and maintains data systems in the cloud. This course can help you develop the skills needed to build and manage scalable data systems, optimize data pipelines, and troubleshoot data issues in the cloud. The course covers topics such as cloud data platforms, cloud data services, and cloud data security, which are all essential for success in this role.
Data Scientist
Data Scientist
A Data Scientist uses data to solve business problems. This course can help you develop the skills needed to analyze data, build machine learning models, and communicate insights to stakeholders. The course covers topics such as data analysis, machine learning, and data visualization, which are all essential for success in this role.
Machine Learning Engineer
Machine Learning Engineer
A Machine Learning Engineer designs, builds, and deploys machine learning models. This course can help you develop the skills needed to build and manage scalable machine learning systems, optimize model performance, and troubleshoot model issues. The course covers topics such as machine learning algorithms, model training, and model deployment, which are all essential for success in this role.
Data Architect
Data Architect
A Data Architect designs and implements data management solutions that meet the needs of an organization. This course can help you develop the skills needed to design and build scalable data systems, manage complex data workflows, and optimize data operations. The course covers topics such as data modeling, data integration, and data governance, which are all essential for success in this role.
Database Administrator
Database Administrator
A Database Administrator designs, builds, and maintains databases. This course can help you develop the skills needed to design and build scalable databases, optimize database performance, and troubleshoot database issues. The course covers topics such as database design, database optimization, and database security, which are all essential for success in this role.
Data Engineer
Data Engineer
A Data Engineer designs, builds, and maintains data pipelines that transform raw data into usable information. This course can help you develop the skills needed to build and manage scalable data systems, optimize data pipelines, and troubleshoot data issues. The course covers topics such as data ingestion, data cleaning, and data transformation, which are all essential for success in this role.
Big Data Engineer
Big Data Engineer
A Big Data Engineer designs, builds, and maintains data systems that handle large volumes of data. This course can help you develop the skills needed to build and manage scalable data systems, optimize data pipelines, and troubleshoot data issues. The course covers topics such as big data architectures, big data technologies, and big data analytics, which are all essential for success in this role.
Data Analyst
Data Analyst
A Data Analyst analyzes data to identify trends and patterns. This course can help you develop the skills needed to analyze data, build data visualizations, and communicate insights to stakeholders. The course covers topics such as data analysis techniques, data visualization tools, and data storytelling, which are all essential for success in this role.
Business Intelligence Analyst
Business Intelligence Analyst
A Business Intelligence Analyst uses data to improve business decision-making. This course can help you develop the skills needed to analyze data, build data visualizations, and communicate insights to stakeholders. The course covers topics such as business intelligence tools, business intelligence techniques, and business intelligence reporting, which are all essential for success in this role.
Computer Scientist
Computer Scientist
A Computer Scientist researches and develops new computer technologies. This course can help you develop the skills needed to research and develop new data engineering technologies, design and implement new data engineering algorithms, and evaluate the performance of new data engineering systems. The course covers topics such as data engineering theory, data engineering algorithms, and data engineering systems, which are all essential for success in this role.
Software Engineer
Software Engineer
A Software Engineer designs, builds, and maintains software applications. This course can help you develop the skills needed to design and build scalable software applications, optimize software performance, and troubleshoot software issues. The course covers topics such as software design, software development, and software testing, which are all essential for success in this role.
Project Manager
Project Manager
A Project Manager plans, executes, and closes projects. This course can help you develop the skills needed to plan and execute data engineering projects, manage data engineering teams, and budget for data engineering projects. The course covers topics such as project planning, project execution, and project closure, which are all essential for success in this role.
Information Technology Manager
Information Technology Manager
An Information Technology Manager plans, implements, and manages information technology systems. This course can help you develop the skills needed to plan and implement data engineering systems, manage data engineering teams, and budget for data engineering projects. The course covers topics such as information technology planning, information technology management, and information technology budgeting, which are all essential for success in this role.
Data Privacy Officer
Data Privacy Officer
A Data Privacy Officer develops and implements data privacy policies and procedures. This course can help you develop the skills needed to develop and implement data privacy policies and procedures, manage data privacy teams, and budget for data privacy projects. The course covers topics such as data privacy planning, data privacy implementation, and data privacy auditing, which are all essential for success in this role.
Data Governance Manager
Data Governance Manager
A Data Governance Manager develops and implements data governance policies and procedures. This course can help you develop the skills needed to develop and implement data governance policies and procedures, manage data governance teams, and budget for data governance projects. The course covers topics such as data governance planning, data governance implementation, and data governance auditing, which are all essential for success in this role.
For more career information including salaries, visit:
OpenCourser.com/course/swu1x4/advanced
Reading list
We've selected eight books
that we think will supplement your
learning. Use these to
develop background knowledge, enrich your coursework, and gain a
deeper understanding of the topics covered in
Advanced Data Engineering.
Save
A comprehensive guide to designing and building scalable data-intensive applications. Covers system architecture, data modeling, and performance optimization. Provides valuable insights and best practices for handling complex data challenges.
Comprehensive guide to using Python for data analysis and manipulation.
Provides a practical guide to data cleaning, transformation, and analysis using Python and Pandas. Covers data manipulation, data exploration, and data visualization. Serves as a valuable reference for implementing data engineering tasks in Python.
A concise reference guide for MySQL, covering its SQL syntax, data types, and database management commands. Provides quick access to essential MySQL information and serves as a handy companion for data engineers working with MySQL.
Provides a comprehensive overview of using MapReduce for data-intensive text processing.
Provides a practical introduction to big data analytics.
Save
Provides a practical introduction to graph databases.
Provides a practical introduction to key-value databases.
For more information about how these books relate to this course, visit:
OpenCourser.com/course/swu1x4/advanced
Share
Similar courses
Here are nine courses similar to
Advanced Data Engineering.
Advanced Data Engineering
OpenCourser.com/course/9zvusz/advanced
Advanced Data Engineering
Most relevant
Vector Databases & Embeddings for Developers
OpenCourser.com/course/tgrwtu/vector
Vector Databases & Embeddings for Developers
The R Programming Environment
OpenCourser.com/course/55q1x5/the
The R Programming Environment
Rust Data Engineering
OpenCourser.com/course/vbuvsm/rust
Rust Data Engineering
Analyze Datasets and Train ML Models using AutoML
OpenCourser.com/course/ksk9bd/analyze
Analyze Datasets and Train ML Models using AutoML
Pandas Arrays and Data Structures
OpenCourser.com/course/zdwf6a/pandas
Pandas Arrays and Data Structures
Handling Streaming Data with Azure Databricks Using Spark Structured Streaming
OpenCourser.com/course/2q4nr7/handling
Handling Streaming Data with Azure Databricks Using Spark...
Optimize ML Models and Deploy Human-in-the-Loop Pipelines
OpenCourser.com/course/80vzx7/optimize
Optimize ML Models and Deploy Human-in-the-Loop Pipelines
Creating New BigQuery Datasets and Visualizing Insights
OpenCourser.com/course/52tejo/creating
Creating New BigQuery Datasets and Visualizing Insights
Effort
5 weeks, 5 hours/week
Level
Intermediate
Via
Coursera
Institution
Duke University
Instructors
Noah Gift
Alfredo Deza
Language
English
This course belongs to a
collection
called
Large Language Model Operations (LLMOps). It's comprised of five courses:
- Databricks to Local LLMs
- Open Source LLMOps Solutions
- Operationalizing LLMs on Azure
- GenAI and LLMs on AWS
- Advanced Data Engineering (viewing)
Good to know
Explores RabbitMQ and MySQL databases, industry-standard tools widely used in modern web applications
Covers Apache Airflow, a leading workflow management platform, strengthening your ability to orchestrate complex data pipelines efficiently
Introduces Vector and Graph databases, specialized tools that unlock scalability and enhanced data management capabilities for complex datasets, extending your skillset to handle diverse data scenarios
Through hands-on projects, learners can apply their acquired knowledge to real-world data engineering challenges, enhancing their practical problem-solving skills
Taught by Noah Gift and Alfredo Deza, experienced data engineering professionals, providing learners with access to valuable insights and industry best practices
Suitable for data engineers, data scientists, and those with a grounding in data handling, aligning with their professional growth and skill development
Our mission
OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.
Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.
Find this site helpful? Tell a friend about us.
Affiliate disclosure
We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.
Your purchases help us maintain our catalog and keep our servers humming without ads.
Thank you for supporting OpenCourser.
© 2016 - 2024 OpenCourser