In this course, you will be introduced to the data engineering lifecycle, from data generation in source systems, data ingestion, data transformation, storage, all the way to serving data to downstream stakeholders. You’ll take note of the key undercurrents that affect all stages of the lifecycle, and start developing a framework for how to think like a data engineer.
Read more
In this course, you will be introduced to the data engineering lifecycle, from data generation in source systems, data ingestion, data transformation, storage, all the way to serving data to downstream stakeholders. You’ll take note of the key undercurrents that affect all stages of the lifecycle, and start developing a framework for how to think like a data engineer.
In this course, you will be introduced to the data engineering lifecycle, from data generation in source systems, data ingestion, data transformation, storage, all the way to serving data to downstream stakeholders. You’ll take note of the key undercurrents that affect all stages of the lifecycle, and start developing a framework for how to think like a data engineer.
To gain hands-on practice, you’ll gather stakeholder needs, translate those needs into system requirements, and choose tools and technologies to build systems that provide business value. You’ll also build an end-to-end data system that encompasses all stages of the data engineering lifecycle, dive into some of the details of batch and streaming pipelines to serve data for a product recommendation system, and apply principles of good data architecture to assess the security, performance, reliability, and scalability of a web app hosted on AWS.
Good to know
Save this course
Save
Activities
Compile course materials
Show steps
Gather and organize all relevant resources to better engage with course materials.
Show steps
-
Gather notes, assignments, and quizzes
-
Organize materials chronologically or by topic
-
Review materials for clarity and understanding
Follow a data ingestion tutorial
Show steps
Enhance data ingestion understanding and skills by following a guided tutorial.
Show steps
-
Identify a reputable data ingestion tutorial
-
Follow the tutorial step-by-step
-
Apply the concepts to a personal project
Join a data engineering study group
Show steps
Enhance learning through collaborative problem-solving and knowledge-sharing with peers in a study group.
Show steps
-
Find or create a study group
-
Establish regular meeting times
-
Discuss course materials, share resources
-
Work together on assignments or projects
Four other activities
Expand to see all activities and additional details
Show all seven activities
Practice data transformation exercises
Show steps
Solidify data transformation skills by completing targeted exercises regularly.
Show steps
-
Identify data transformation techniques
-
Find or create practice datasets
-
Apply data transformation techniques
Attend a data engineering workshop
Show steps
Acquire practical, specialized knowledge by engaging in hands-on, in-person data engineering workshops.
Browse courses on
Data Engineering
Show steps
-
Research relevant workshops
-
Register and attend the workshop
-
Actively participate in hands-on activities
-
Network with experts and peers
Build a basic data processing pipeline
Show steps
Apply data engineering principles and skills by constructing a data processing pipeline from scratch.
Show steps
-
Define the input and output data sources
-
Choose appropriate data transformation techniques
-
Implement the pipeline using a suitable programming language
-
Test and refine the pipeline
Participate in a data hackathon
Show steps
Accelerate learning and expand practical skills by participating in a data hackathon related to data engineering.
Show steps
-
Identify relevant data hackathons
-
Form or join a team
-
Develop innovative solutions to data-related challenges
-
Present the solution and compete for recognition
Compile course materials
Show steps
Gather and organize all relevant resources to better engage with course materials.
Show steps
- Gather notes, assignments, and quizzes
- Organize materials chronologically or by topic
- Review materials for clarity and understanding
Follow a data ingestion tutorial
Show steps
Enhance data ingestion understanding and skills by following a guided tutorial.
Show steps
- Identify a reputable data ingestion tutorial
- Follow the tutorial step-by-step
- Apply the concepts to a personal project
Join a data engineering study group
Show steps
Enhance learning through collaborative problem-solving and knowledge-sharing with peers in a study group.
Show steps
- Find or create a study group
- Establish regular meeting times
- Discuss course materials, share resources
- Work together on assignments or projects
Practice data transformation exercises
Show steps
Solidify data transformation skills by completing targeted exercises regularly.
Show steps
- Identify data transformation techniques
- Find or create practice datasets
- Apply data transformation techniques
Attend a data engineering workshop
Show steps
Acquire practical, specialized knowledge by engaging in hands-on, in-person data engineering workshops.
Browse courses on
Data Engineering
Show steps
- Research relevant workshops
- Register and attend the workshop
- Actively participate in hands-on activities
- Network with experts and peers
Build a basic data processing pipeline
Show steps
Apply data engineering principles and skills by constructing a data processing pipeline from scratch.
Show steps
- Define the input and output data sources
- Choose appropriate data transformation techniques
- Implement the pipeline using a suitable programming language
- Test and refine the pipeline
Participate in a data hackathon
Show steps
Accelerate learning and expand practical skills by participating in a data hackathon related to data engineering.
Show steps
- Identify relevant data hackathons
- Form or join a team
- Develop innovative solutions to data-related challenges
- Present the solution and compete for recognition
Career center
Reading list
Share
Similar courses
OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.
Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.
Find this site helpful? Tell a friend about us.
We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.
Your purchases help us maintain our catalog and keep our servers humming without ads.
Thank you for supporting OpenCourser.
