We may earn an affiliate commission when you visit our partners.
Noah Gift
Welcome to the third course in the Building Cloud Computing Solutions at Scale Specialization! In this course, you will learn how to apply Data Engineering to real-world projects using the Cloud computing concepts introduced in the first two courses of this series. By the end of this course, you will be able to develop Data Engineering applications and use software development best practices to create data engineering applications. These will include continuous deployment, code quality tools, logging, instrumentation and monitoring. Finally, you will use Cloud-native technologies to tackle complex data engineering solutions.
Read more
Welcome to the third course in the Building Cloud Computing Solutions at Scale Specialization! In this course, you will learn how to apply Data Engineering to real-world projects using the Cloud computing concepts introduced in the first two courses of this series. By the end of this course, you will be able to develop Data Engineering applications and use software development best practices to create data engineering applications. These will include continuous deployment, code quality tools, logging, instrumentation and monitoring. Finally, you will use Cloud-native technologies to tackle complex data engineering solutions.
Read more
Welcome to the third course in the Building Cloud Computing Solutions at Scale Specialization! In this course, you will learn how to apply Data Engineering to real-world projects using the Cloud computing concepts introduced in the first two courses of this series. By the end of this course, you will be able to develop Data Engineering applications and use software development best practices to create data engineering applications. These will include continuous deployment, code quality tools, logging, instrumentation and monitoring. Finally, you will use Cloud-native technologies to tackle complex data engineering solutions.
This course is ideal for beginners as well as intermediate students interested in applying Cloud computing to data science, machine learning and data engineering. Students should have beginner level Linux and intermediate level Python skills. For your project in this course, you will build a serverless data engineering pipeline in a Cloud platform: Amazon Web Services (AWS), Azure or Google Cloud Platform (GCP).
Register for this course and see more details by visiting:
OpenCourser.com/course/bi4ac5/cloud
Three deals to help you save
We found three deals and offers that may be relevant to this course.
Save money when you learn.
All coupon codes, vouchers, and discounts are applied automatically unless otherwise noted.
Valid until December 12
Annual Subscription Savings
Unlock programs from Google, Microsoft, and IBM and join 77% of learners who reported positive career outcomes.
Only $239!
Valid for a limited time only
Future-proof your career
Access O'Reilly books, live events, courses, and more. Save with an annual subscription.
Take
15%
off
What's inside
Syllabus
Getting Started with Cloud Data Engineering
This week, you will learn about the methodologies involved in Data Engineering. You will also learn to evaluate best practices for dealing with the end of Moore’s Law, develop distributed systems that apply software engineering best practices and evaluate best practices for implementing solutions with Big Data. You will apply these practices to build a GPU programming project using Numba and the CUDA SDK.
Read more
Syllabus
Getting Started with Cloud Data Engineering
This week, you will learn about the methodologies involved in Data Engineering. You will also learn to evaluate best practices for dealing with the end of Moore’s Law, develop distributed systems that apply software engineering best practices and evaluate best practices for implementing solutions with Big Data. You will apply these practices to build a GPU programming project using Numba and the CUDA SDK.
Read more
Examining Principles of Data Engineering
This week, you will learn what Data Engineering is and how to use software engineering best practices in Data Engineering. You will then apply this knowledge by building a command-line data processing tool.
Building Data Engineering Pipelines
This week, you will learn serverless data engineering techniques and data governance best practices. You will then apply this knowledge by building a serverless Data Engineering system.
Applying Key Data Engineering Tasks
This week, you will learn about key Data Engineering tasks including ETL, Cloud Databases and Cloud Storage. You will then apply this knowledge by building a serverless AWS lambda function that labels an image using the AWS Rekognition API.
Good to know
Good to know
Know what's good ,
what to watch for , and
possible dealbreakers
Sets up a strong foundation for intermediate learners
Builds a strong foundation for beginners
May require foundational skills in Linux and Python
Instructs students with clear learning objectives
Provides comprehensive coverage of cloud data engineering
Teaches industry best practices for data engineering
Save this course
Save Cloud Data Engineering to your list so you can find it easily later:
Save
Save
Reviews summary
Mixed opinions about cloud data engineering course
Students are largely divided in their opinions about this Cloud Data Engineering course with just over half being positive in their reviews. Positive feedback highlights the engaging assignments, well-designed content, and important information presented. Negative reviews criticize a lack of hands-on labs, cohesive structure, superficial explanations, and errors in the material.
Valuable information, well-designed, and engaging.
"Very well designed course for data engineering"
"Very important information and concepts was shared."
"Great introduction to the concepts, very good practical examples, with relevant technologies and platforms."
Lacks hands-on activities, cohesive structure, and clear explanations.
"no inbuilt labs"
"The course, unfortunately, lacks a cohesive structure"
"Explanations provided were superficial and lacked the required technical depth."
Activities
Be better prepared
before
your course. Deepen your understanding
during
and
after
it. Supplement your coursework and achieve mastery of the topics covered
in Cloud Data Engineering with these
activities:
Organize and Review Course Materials
Show steps
Prepare for the course by organizing and reviewing the provided materials.
Show steps
-
Review the course syllabus and schedule
-
Obtain and organize lecture notes, readings, and assignments
-
Preview the materials and identify areas for clarification
-
Create a study plan and schedule
Review cloud computing concepts
Show steps
Refreshing your knowledge of cloud computing concepts will provide a stronger foundation for this course.
Browse courses on
Cloud Computing Concepts
Show steps
-
Review the syllabus for the first two courses in this series to recap the concepts covered.
-
Visit the course website to review the course syllabus and any introductory materials.
-
Read through the textbooks for the first two courses to brush up on key concepts.
Follow Cloud Data Engineering Best Practices Tutorials
Show steps
Enhance your knowledge and skills by following guided tutorials from industry experts.
Show steps
-
Identify reputable sources for Cloud Data Engineering tutorials
-
Select tutorials that align with your learning objectives
-
Follow the tutorials step-by-step and experiment with the provided examples
-
Implement the learned techniques in your own projects
Nine other activities
Expand to see all activities and additional details
Show all 12 activities
Solve Cloud Data Engineering Coding Challenges
Show steps
Reinforce Data Engineering concepts by solving hands-on coding challenges.
Show steps
-
Find online coding challenges related to Cloud Data Engineering
-
Attempt to solve the challenges independently
-
Review solutions and identify areas for improvement
-
Practice regularly to enhance problem-solving skills
Build a Serverless Data Engineering Pipeline
Show steps
Build a serverless data engineering pipeline to gain practical experience and apply course concepts.
Show steps
-
Choose a cloud platform (AWS, Azure, or GCP)
-
Design and architect the pipeline
-
Implement the pipeline using serverless technologies
-
Test and validate the pipeline
-
Deploy and monitor the pipeline
Practice building data engineering pipelines
Show steps
Practicing building data engineering pipelines will help you develop the skills necessary for this course.
Browse courses on
Data Engineering
Show steps
-
Set up a development environment with the necessary tools and software.
-
Find a dataset to work with.
-
Design and implement a data engineering pipeline to process the data.
Follow tutorials on data engineering best practices
Show steps
Following tutorials on data engineering best practices will help you learn the best approaches to use in your projects.
Show steps
-
Search for tutorials on data engineering best practices.
-
Follow the steps in the tutorials to learn how to implement best practices in your own projects.
Join Data Engineering Study Groups
Show steps
Collaborate and learn from peers to enhance your understanding and retention.
Show steps
-
Find or create a study group with other learners
-
Establish regular meeting times for discussion and review
-
Share knowledge, insights, and resources
-
Work on projects together to apply course concepts
Develop Data Engineering Cheat Sheets
Show steps
Create your own reference materials to aid in your understanding and recall.
Show steps
-
Identify key concepts and techniques covered in the course
-
Create concise and informative cheat sheets summarizing these topics
-
Use diagrams, examples, and concise explanations
-
Review and update the cheat sheets regularly
Read documentation and contribute to open source data engineering projects
Show steps
Reading documentation and contributing to open source projects will allow you to learn from the work of others and contribute to the data engineering community.
Browse courses on
Data Engineering
Show steps
-
Find open source data engineering projects that interest you.
-
Read the documentation for the projects to learn about their design and implementation.
-
Identify areas where you can contribute to the projects.
-
Submit your contributions to the project.
Build a serverless data engineering pipeline
Show steps
Building a serverless data engineering pipeline will allow you to apply the concepts you learn in this course to a real-world project.
Browse courses on
Data Engineering
Show steps
-
Choose a cloud platform (AWS, Azure, or GCP) to build your pipeline on.
-
Design and implement your pipeline using serverless technologies.
-
Test and deploy your pipeline.
Participate in a data engineering competition
Show steps
Participating in a data engineering competition will allow you to test your skills against other data engineers and learn from their approaches.
Browse courses on
Data Engineering
Show steps
-
Find a data engineering competition to participate in.
-
Form a team or work individually on the competition.
-
Develop and implement a data engineering solution to the competition problem.
-
Submit your solution and compete for prizes or recognition.
Organize and Review Course Materials
Show steps
Prepare for the course by organizing and reviewing the provided materials.
Show steps
Show steps
Show steps
- Review the course syllabus and schedule
- Obtain and organize lecture notes, readings, and assignments
- Preview the materials and identify areas for clarification
- Create a study plan and schedule
Review cloud computing concepts
Show steps
Refreshing your knowledge of cloud computing concepts will provide a stronger foundation for this course.
Browse courses on
Cloud Computing Concepts
Show steps
Show steps
Show steps
- Review the syllabus for the first two courses in this series to recap the concepts covered.
- Visit the course website to review the course syllabus and any introductory materials.
- Read through the textbooks for the first two courses to brush up on key concepts.
Follow Cloud Data Engineering Best Practices Tutorials
Show steps
Enhance your knowledge and skills by following guided tutorials from industry experts.
Show steps
Show steps
Show steps
- Identify reputable sources for Cloud Data Engineering tutorials
- Select tutorials that align with your learning objectives
- Follow the tutorials step-by-step and experiment with the provided examples
- Implement the learned techniques in your own projects
Nine other activities
Expand to see all activities and additional details
Show all 12 activities
Solve Cloud Data Engineering Coding Challenges
Show steps
Reinforce Data Engineering concepts by solving hands-on coding challenges.
Show steps
Show steps
Show steps
- Find online coding challenges related to Cloud Data Engineering
- Attempt to solve the challenges independently
- Review solutions and identify areas for improvement
- Practice regularly to enhance problem-solving skills
Build a Serverless Data Engineering Pipeline
Show steps
Build a serverless data engineering pipeline to gain practical experience and apply course concepts.
Show steps
Show steps
Show steps
- Choose a cloud platform (AWS, Azure, or GCP)
- Design and architect the pipeline
- Implement the pipeline using serverless technologies
- Test and validate the pipeline
- Deploy and monitor the pipeline
Practice building data engineering pipelines
Show steps
Practicing building data engineering pipelines will help you develop the skills necessary for this course.
Browse courses on
Data Engineering
Show steps
Show steps
Show steps
- Set up a development environment with the necessary tools and software.
- Find a dataset to work with.
- Design and implement a data engineering pipeline to process the data.
Follow tutorials on data engineering best practices
Show steps
Following tutorials on data engineering best practices will help you learn the best approaches to use in your projects.
Show steps
Show steps
Show steps
- Search for tutorials on data engineering best practices.
- Follow the steps in the tutorials to learn how to implement best practices in your own projects.
Join Data Engineering Study Groups
Show steps
Collaborate and learn from peers to enhance your understanding and retention.
Show steps
Show steps
Show steps
- Find or create a study group with other learners
- Establish regular meeting times for discussion and review
- Share knowledge, insights, and resources
- Work on projects together to apply course concepts
Develop Data Engineering Cheat Sheets
Show steps
Create your own reference materials to aid in your understanding and recall.
Show steps
Show steps
Show steps
- Identify key concepts and techniques covered in the course
- Create concise and informative cheat sheets summarizing these topics
- Use diagrams, examples, and concise explanations
- Review and update the cheat sheets regularly
Read documentation and contribute to open source data engineering projects
Show steps
Reading documentation and contributing to open source projects will allow you to learn from the work of others and contribute to the data engineering community.
Browse courses on
Data Engineering
Show steps
Show steps
Show steps
- Find open source data engineering projects that interest you.
- Read the documentation for the projects to learn about their design and implementation.
- Identify areas where you can contribute to the projects.
- Submit your contributions to the project.
Build a serverless data engineering pipeline
Show steps
Building a serverless data engineering pipeline will allow you to apply the concepts you learn in this course to a real-world project.
Browse courses on
Data Engineering
Show steps
Show steps
Show steps
- Choose a cloud platform (AWS, Azure, or GCP) to build your pipeline on.
- Design and implement your pipeline using serverless technologies.
- Test and deploy your pipeline.
Participate in a data engineering competition
Show steps
Participating in a data engineering competition will allow you to test your skills against other data engineers and learn from their approaches.
Browse courses on
Data Engineering
Show steps
Show steps
Show steps
- Find a data engineering competition to participate in.
- Form a team or work individually on the competition.
- Develop and implement a data engineering solution to the competition problem.
- Submit your solution and compete for prizes or recognition.
Career center
Learners who complete Cloud Data Engineering will develop knowledge and skills
that may be useful to these careers:
Data Engineer
Data Engineer
A Data Engineer designs and develops data pipelines and systems to manage and analyze large volumes of data. They work closely with data scientists and other stakeholders to ensure that data is accessible, reliable, and secure. This course can help you develop the skills you need to succeed as a Data Engineer, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
Data Scientist
Data Scientist
A Data Scientist uses data to solve business problems. They collect, clean, and analyze data to identify trends and patterns. They then use this information to develop models and make predictions. This course can help you develop the skills you need to succeed as a Data Scientist, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
Machine Learning Engineer
Machine Learning Engineer
A Machine Learning Engineer designs and develops machine learning models. They work closely with data scientists and other stakeholders to ensure that models are accurate, reliable, and scalable. This course can help you develop the skills you need to succeed as a Machine Learning Engineer, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
Cloud Architect
Cloud Architect
A Cloud Architect designs and develops cloud-based solutions. They work closely with customers to understand their business needs and then design and implement cloud solutions that meet those needs. This course can help you develop the skills you need to succeed as a Cloud Architect, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
Software Engineer
Software Engineer
A Software Engineer designs, develops, and maintains software systems. They work closely with customers and other stakeholders to understand their needs and then design and implement software solutions that meet those needs. This course can help you develop the skills you need to succeed as a Software Engineer, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
Data Analyst
Data Analyst
A Data Analyst collects, cleans, and analyzes data to identify trends and patterns. They then use this information to develop insights and make recommendations. This course can help you develop the skills you need to succeed as a Data Analyst, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
Business Analyst
Business Analyst
A Business Analyst works with businesses to identify and solve problems. They use data and other information to analyze business processes and make recommendations. This course can help you develop the skills you need to succeed as a Business Analyst, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
Project Manager
Project Manager
A Project Manager plans, executes, and closes projects. They work closely with stakeholders to define project goals and objectives and then develop and implement project plans. This course can help you develop the skills you need to succeed as a Project Manager, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
Product Manager
Product Manager
A Product Manager develops and launches new products. They work closely with customers and other stakeholders to identify and define product requirements. This course can help you develop the skills you need to succeed as a Product Manager, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
Sales Engineer
Sales Engineer
A Sales Engineer helps customers to understand and purchase products and services. They work closely with customers to identify their needs and then develop and implement solutions that meet those needs. This course can help you develop the skills you need to succeed as a Sales Engineer, including data engineering principles, best practices, and tools. You will also learn how to build and deploy data engineering pipelines in the cloud.
For more career information including salaries, visit:
OpenCourser.com/course/bi4ac5/cloud
Reading list
We've selected ten books
that we think will supplement your
learning. Use these to
develop background knowledge, enrich your coursework, and gain a
deeper understanding of the topics covered in
Cloud Data Engineering.
Save
Takes a deep dive into various aspects of big data, including storage, processing, and analysis. It will be a useful reference in examining principles of data engineering.
Covers theoretical and practical aspects of data-intensive text processing using Hadoop and MapReduce. It will be useful as a background reading in text processing for data engineering applications.
Focuses on using Python for data analysis. It will be a useful reference for data engineering applications.
Is an introduction to machine learning using Python libraries. It will be useful as a reference for data engineering applications.
Covers more theoretical aspects of designing scalable systems, and useful as a background reference for data engineering principles.
Save
Is an introduction to database systems. It will be a useful reference for those looking to explore the fundamentals of data storage and retrieval.
Save
Is an introduction to the Spark framework. It will be useful as a background reference or for additional reading.
Save
Covers cloud computing concepts and architectures. It will be a useful reference for those interested in exploring the background of cloud computing beyond the course.
Save
Is an introduction to deep learning. Like Machine Learning Yearning, it is not directly related to data engineering, but may be useful for the course project.
Covers foundations of data science and useful as a background text or reference.
For more information about how these books relate to this course, visit:
OpenCourser.com/course/bi4ac5/cloud
Share
Similar courses
Here are nine courses similar to
Cloud Data Engineering.
Cloud Virtualization, Containers and APIs
OpenCourser.com/course/a2dhqp/cloud
Cloud Virtualization, Containers and APIs
Most relevant
Cloud Computing Foundations
OpenCourser.com/course/6643kn/cloud
Cloud Computing Foundations
Most relevant
Cloud Machine Learning Engineering and MLOps
OpenCourser.com/course/47wowm/cloud
Cloud Machine Learning Engineering and MLOps
Most relevant
Cloud Computing Foundations
OpenCourser.com/course/1kuqig/cloud
Cloud Computing Foundations
Most relevant
Develop Clustering Models with Azure ML Designer
OpenCourser.com/course/fh1xi5/develop
Develop Clustering Models with Azure ML Designer
AWS Certified Solutions Architect - Associate (SAA-C03)
OpenCourser.com/course/rcccw9/aws
AWS Certified Solutions Architect - Associate (SAA-C03)
Modernizing Data Lakes and Data Warehouses with GCP
OpenCourser.com/course/c2f2ok/modernizing
Modernizing Data Lakes and Data Warehouses with GCP
AWS Certified Solutions Architect - Associate (SAA-C03)
OpenCourser.com/course/xusn4e/aws
AWS Certified Solutions Architect - Associate (SAA-C03)
Platform Engineering: The Big Picture
OpenCourser.com/course/g5yoi4/platform
Platform Engineering: The Big Picture
Effort
4 weeks of study, 3-5 hours/week
Level
Intermediate
Via
Coursera
Institution
Duke University
Instructor
Noah Gift
Language
English
Transcripts
Español
Français
Português
Deutsch
Italiano
中文
العربية
한국어
日本語
हिंदी
Nederlands
Svenska
Pусский
Türkçe
Polski
Ελληνικά
українська мова
Bahasa Indonesia
ไทย
қазақша
This course belongs to a
collection
called
Building Cloud Computing Solutions at Scale. It's comprised of four courses:
- Cloud Machine Learning Engineering and MLOps
- Cloud Virtualization, Containers and APIs
- Cloud Data Engineering (viewing)
- Cloud Computing Foundations
Good to know
Sets up a strong foundation for intermediate learners
Builds a strong foundation for beginners
May require foundational skills in Linux and Python
Instructs students with clear learning objectives
Provides comprehensive coverage of cloud data engineering
Teaches industry best practices for data engineering
Our mission
OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.
Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.
Find this site helpful? Tell a friend about us.
Affiliate disclosure
We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.
Your purchases help us maintain our catalog and keep our servers humming without ads.
Thank you for supporting OpenCourser.
© 2016 - 2024 OpenCourser