We may earn an affiliate commission when you visit our partners.
Course image
Joe Reis

In this course, you will explore various types of source systems, learn how they generate and update data, and troubleshoot common issues you might encounter when trying to connect to these systems in the real world. You’ll dive into the details of common ingestion patterns and implement batch and streaming pipelines. You’ll automate and orchestrate your data pipelines using infrastructure as code and pipeline as code tools. You’ll also explore AWS and open-source tools for monitoring your data systems and data quality.

Enroll now

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Builds foundation for working with streaming data pipelines
Designed for beginners who need to learn data ingestion and orchestration
Provides hands-on experience with AWS and open-source tools for monitoring data systems and data quality
Taught by Joe Reis, an instructor with experience in data engineering and architecture
May require prerequisites in data analysis and programming

Save this course

Save Source Systems, Data Ingestion, and Pipelines to your list so you can find it easily later:
Save

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Source Systems, Data Ingestion, and Pipelines with these activities:
Review SQL concepts
Review the basics of SQL to ensure a strong foundation for data ingestion and ETL processes.
Browse courses on SQL
Show steps
  • Read through SQL tutorials and documentation.
  • Practice writing basic SQL queries.
Review pre-requisite skills
Brushing up on these skills will provide a strong foundation for the topics covered in this course.
Browse courses on Python
Show steps
  • Review basic Python syntax, data structures, and algorithms
  • Practice data cleaning and manipulation techniques
  • Review data analysis and visualization techniques
Review data pipeline fundamentals
Refresh foundational knowledge in data pipelines to enhance understanding of advanced concepts covered in the course.
Browse courses on Data Pipelines
Show steps
  • Review concepts such as data sources, data ingestion, data transformations, and data storage.
  • Explore different types of data pipelines and their use cases.
13 other activities
Expand to see all activities and additional details
Show all 16 activities
Review the basics of data ingestion
Strengthen your foundation by reviewing the fundamentals of data ingestion before starting the course.
Browse courses on Data Ingestion
Show steps
  • Read articles or blog posts about data ingestion concepts.
  • Refer to online tutorials or videos on data ingestion techniques.
Join a study group or online forum
Engaging with peers will provide different perspectives and enhance your learning.
Browse courses on Data Pipelines
Show steps
  • Join an online forum or discussion group related to data pipelines
  • Participate in discussions, ask questions, and share your knowledge
  • Attend virtual meetups or conferences to connect with other learners
Explore open-source data pipeline tools
Familiarizing yourself with these tools will enhance your understanding of data pipeline implementation.
Browse courses on Apache Airflow
Show steps
  • Follow tutorials on Apache Airflow, Luigi, or Prefect
  • Build a simple data pipeline using one of these tools
Practice using common data ingestion patterns
Practice using common data ingestion patterns to deepen understanding and solidify skills.
Show steps
  • Identify different data ingestion patterns such as batch, streaming, and change data capture.
  • Implement these patterns using AWS services or open-source tools.
Practice data transformation tasks
Repetitive practice will improve your proficiency in transforming data.
Browse courses on Data Transformation
Show steps
  • Solve data transformation challenges on platforms like LeetCode or HackerRank
Build a small-scale data pipeline using AWS services
Gain hands-on experience by building a practical data pipeline and exploring AWS services.
Browse courses on Data Pipelines
Show steps
  • Choose a data source and define the data pipeline requirements.
  • Select appropriate AWS services for data ingestion, processing, and storage.
  • Configure and manage AWS resources to build the data pipeline.
  • Test and evaluate the pipeline's performance.
Attend data engineering meetups or conferences
Expand your professional network and gain insights by engaging in industry events.
Browse courses on Data Engineering
Show steps
  • Identify and register for relevant data engineering meetups or conferences.
  • Attend sessions, participate in discussions, and connect with professionals.
Build data ingestion pipelines using Python
Practice implementing real-world data ingestion scenarios using Python scripts.
Show steps
  • Find sample datasets and APIs for data sources.
  • Write Python scripts to extract, transform, and load data.
  • Run and debug the pipelines.
Explore AWS and open-source tools for data monitoring
Gain hands-on experience with tools used in the industry to monitor data systems.
Show steps
  • Identify AWS services and open-source tools for data monitoring and alerting.
  • Implement these tools to monitor data pipelines and data quality.
Create and debug pipelines
Solidify your skills in creating data pipelines by building and debugging your own.
Browse courses on Data Pipelines
Show steps
  • Design a data pipeline for a specific use case.
  • Break down the pipeline into smaller, manageable tasks.
  • Implement the pipeline using Python or a preferred programming language.
  • Test and debug the pipeline to ensure it meets the requirements.
Develop a data pipeline using infrastructure as code tools
Create a fully functional data pipeline using infrastructure as code tools, simulating a real-world scenario.
Show steps
  • Design a data pipeline architecture using infrastructure as code tools like Terraform or CloudFormation.
  • Implement the pipeline using these tools, automating infrastructure provisioning and management.
  • Deploy and test the pipeline to ensure its functionality.
Design and implement a data pipeline
Hands-on experience in designing and implementing a data pipeline will solidify your understanding.
Browse courses on Data Pipelines
Show steps
  • Define the requirements and scope of the data pipeline
  • Choose appropriate data sources, transformation tools, and storage solutions
  • Implement the data pipeline using a chosen platform like Apache Airflow or Luigi
  • Test and deploy the data pipeline
Write a blog post on data pipeline best practices
Enhance your understanding of data pipelines by sharing your knowledge and insights through a written article.
Browse courses on Data Pipelines
Show steps
  • Identify and research common best practices for data pipeline development.
  • Structure your blog post with a clear introduction, body, and conclusion.
  • Write in-depth about the best practices, providing examples and explanations.

Career center

Learners who complete Source Systems, Data Ingestion, and Pipelines will develop knowledge and skills that may be useful to these careers:

Reading list

We haven't picked any books for this reading list yet.

Share

Help others find this course page by sharing it with your friends and followers:

Similar courses

Here are nine courses similar to Source Systems, Data Ingestion, and Pipelines.
DevSecOps: Adding Security Testing Tools to Pipelines
Most relevant
Implement LangChain Solutions in Your Data Workflow
Most relevant
Enabling Security Governance and Compliance in DevSecOps
Building Data Pipelines with Luigi 3 and Python
Developing on the Google Cloud Using Datalab and Cloud...
Architecting Serverless Big Data Solutions Using Google...
Network Troubleshooting and Tools
Introduction to Data Engineering
Devops: Jenkins Pipeline As Code: All you need to know A ...
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2024 OpenCourser