We may earn an affiliate commission when you visit our partners.
Spark, Hadoop, and Snowflake for Data Engineering
Noah Gift
and
Kennedy Behrman
Noah Gift
and
Kennedy Behrman
In this course, you will:
Read more
In this course, you will:
Read more
In this course, you will:
- Explore essential data engineering platforms (Hadoop, Spark, and Snowflake) and learn how to optimize and manage them
- Delve into Databricks, a powerful platform for executing data analytics and machine learning tasks
- Hone your Python data science skills with PySpark
- Discover the key concepts of MLflow, an open-source platform for managing the end-to-end machine learning lifecycle, and learn how to integrate it with Databricks
- Gain methodologies to help you improve your project management and workflow skills for data engineering, including applying Kaizen, DevOps, and Data Ops best practices
This course is designed for learners who want to pursue or advance their career in data science or data engineering, or for software developers or engineers who want to grow their data management skill set. With quizzes to test your knowledge throughout, this comprehensive course will help guide your learning journey to become a proficient data engineer, ready to tackle the challenges of today's data-driven world.
Register for this course and see more details by visiting:
OpenCourser.com/course/fe5c2t/spark
What's inside
Learning objectives
- Optimize and manage hadoop, spark, and snowflake platforms
- Execute data analytics and machine learning tasks using databricks
- Enhance python data science skills with pyspark
- Manage end-to-end machine learning lifecycle with mlflow
- Apply kaizen, devops, and dataops methodologies for data engineering
- Optimize and manage hadoop, spark, and snowflake platforms
- Execute data analytics and machine learning tasks using databricks
- Enhance python data science skills with pyspark
- Manage end-to-end machine learning lifecycle with mlflow
- Apply kaizen, devops, and dataops methodologies for data engineering
Syllabus
Module 1: Overview and Introduction to PySpark (7 hours)
- 10 videos (Total 25 minutes)
- Meet your Co-Instructor: Kennedy Behrman (0 minutes, Preview module)
Read more
Syllabus
Module 1: Overview and Introduction to PySpark (7 hours)
- 10 videos (Total 25 minutes)
- Meet your Co-Instructor: Kennedy Behrman (0 minutes, Preview module)
Read more
- Meet your Co-Instructor: Noah Gift (1 minute)
- Overview of Big Data Platforms (1 minute)
- Getting Started with Hadoop (1 minute)
- Getting Started with Spark (1 minute)
- Introduction to Resilient Distributed Datasets (RDD) (2 minutes)
- Resilient Distributed Datasets (RDD) Demo (4 minutes)
- Introduction to Spark SQL (1 minute)
- PySpark Dataframe Demo: Part 1 (3 minutes)
- PySpark Dataframe Demo: Part 2 (7 minutes)
- 9 readings (Total 90 minutes)
- Welcome to Data Engineering Platforms with Python! (10 minutes)
- What is Apache Hadoop? (10 minutes)
- What is Apache Spark? (10 minutes)
- Use Apache Spark in Azure Databricks (optional) (10 minutes)
- Choosing between Hadoop and Spark (10 minutes)
- What are RDDs? (10 minutes)
- Getting Started: Creating RDD's with PySpark (10 minutes)
- Spark SQL, Dataframes and Datasets (10 minutes)
- PySpark and Spark SQL (10 minutes)
- 7 quizzes (Total 210 minutes)
- PySpark (30 minutes)
- Big Data Platforms (30 minutes)
- Apache Hadoop Concepts (30 minutes)
- Apache Spark Concepts (30 minutes)
- RDD Concepts (30 minutes)
- Spark SQL Concepts (30 minutes)
- PySpark Dataframe Concepts (30 minutes)
- 2 discussion prompts (Total 20 minutes)
- Meet and Greet (optional) (10 minutes)
- Let Us Know if Something's Not Working (10 minutes)
- 2 ungraded labs (Total 120 minutes)
- Practice: Creating RDD's with PySpark (60 minutes)
- Practice: Reading Data into Dataframes (60 minutes)
Module 2: Snowflake (4 hours)
- 8 videos (Total 27 minutes)
- What is Snowflake? (2 minutes, Preview module)
- Snowflake Layers (2 minutes)
- Snowflake Web UI (3 minutes)
- Navigating Snowflake (3 minutes)
- Creating a Table in Snowflake (5 minutes)
- Snowflake Warehouses (3 minutes)
- Writing to Snowflake (3 minutes)
- Reading from Snowflake (2 minutes)
- 5 readings (Total 50 minutes)
- Accessing Snowflake (10 minutes)
- Detailed View Inside Snowflake (10 minutes)
- Snowsight: The Snowflake Web Interface (10 minutes)
- Working with Warehouses (10 minutes)
- Python Connector Documentation (10 minutes)
- 6 quizzes (Total 180 minutes)
- Snowflake (30 minutes)
- Snowflake Architecture (30 minutes)
- Snowflake Layers (30 minutes)
- Navigating Snowflake (30 minutes)
- Creating a Table (30 minutes)
- Writing to Snowflake (30 minutes)
Module 3: Azure Databricks and MLFlow (5 hours)
- 16 videos (Total 71 minutes)
- Accessing Databricks (0 minutes, Preview module)
- Spark Notebooks with Databricks (4 minutes)
- Using Data with Databricks (4 minutes)
- Working with Workspaces in Databricks (3 minutes)
- Advanced Capabilities of Databricks (1 minute)
- PySpark Introduction on Databricks (7 minutes)
- Exploring Databricks Azure Features (3 minutes)
- Using the DBFS to AutoML Workflow (4 minutes)
- Load, Register and Deploy ML Models (2 minutes)
- Databricks Model Registry (2 minutes)
- Model Serving on Databricks (2 minutes)
- What is MLOps? (12 minutes)
- Exploring Open-Source MLFlow Frameworks (5 minutes)
- Running MLFlow with Databricks (6 minutes)
- End to End Databricks MLFlow (4 minutes)
- Databricks Autologging with MLFlow (4 minutes)
- 7 readings (Total 70 minutes)
- What is Azure Databricks? (10 minutes)
- Introduction to Databricks Machine Learning (10 minutes)
- What is the Databricks File System (DBFS)? (10 minutes)
- Serverless Compute with Databricks (10 minutes)
- MLOps Workflow on Azure Databricks (10 minutes)
- Run MLFlow Projects on Azure Databricks (10 minutes)
- Databricks Autologging (10 minutes)
- 4 quizzes (Total 120 minutes)
- DataBricks (30 minutes)
- PySpark SQL (30 minutes)
- PySpark DataFrames (30 minutes)
- MLFlow with Databricks (30 minutes)
- 1 ungraded lab (Total 60 minutes)
- ETL-Part-1: Keyword Extractor Tool to HashTag Tool (60 minutes)
Module 4: DataOps and Operations Methodologies (12 hours)
- 21 videos (Total 502 minutes)
- Kaizen Methodology for Data (4 minutes, Preview module)
- Introducing GitHub CodeSpaces (9 minutes)
- Compiling Python in GitHub Codespaces (18 minutes)
- Walking through Sagemaker Studio Lab (28 minutes)
- Pytest Master Class (Optional) (166 minutes)
- What is DevOps? (2 minutes)
- DevOps Key Concepts (35 minutes)
- Continuous Integration Overview (32 minutes)
- Build an NLP in Cloud9 with Python (43 minutes)
- Build a Continuously Deployed Containerized FastAPI Microservice (43 minutes)
- Hugo Continuous Deploy on AWS (18 minutes)
- Container Based Continuous Delivery (8 minutes)
- What is DataOps? (1 minute)
- DataOps and MLOps with Snowflake (61 minutes)
- Building Cloud Pipelines with Step Functions and Lambda (16 minutes)
- What is a Data Lake? (2 minutes)
- Data Warehouse vs. Feature Store (2 minutes)
- Big Data Challenges (1 minute)
- Types of Big Data Processing (1 minute)
- Real-World Data Engineering Pipeline (2 minutes)
- Data Feedback Loop (0 minutes)
- 6 readings (Total 60 minutes)
- GitHub Codespaces Overview (10 minutes)
- Getting Started with Amazon SageMaker Studio Lab (10 minutes)
- Teaching MLOps at Scale with GitHub (Optional) (10 minutes)
- Getting Started with DevOps and Cloud Computing (10 minutes)
- Benefits of Serverless ETL Technologies (10 minutes)
- Next Steps (10 minutes)
- DataOps and Operations Methodologies (30 minutes)
- Kaizen Methodology (30 minutes)
- DevOps (30 minutes)
- DataOps (30 minutes)
- ETL-Part2: SQLite ETL Destination (60 minutes)
Good to know
Good to know
Know what's good ,
what to watch for , and
possible dealbreakers
Develops foundational and advanced skills for those in data science and engineering fields
Taught by experienced professionals in the industry
Explores a range of essential data engineering platforms and technologies
Covers project management and workflow optimization practices
Requires some prior programming experience, limiting accessibility for absolute beginners
Save this course
Save Spark, Hadoop, and Snowflake for Data Engineering to your list so you can find it easily later:
Save
Save
Activities
Be better prepared
before
your course. Deepen your understanding
during
and
after
it. Supplement your coursework and achieve mastery of the topics covered
in Spark, Hadoop, and Snowflake for Data Engineering with these
activities:
Review Concepts in PySpark Study Materials
Show steps
Reviewing these topics will bring back important concepts in PySpark at the start of the course to stimulate your thinking.
Browse courses on
Pyspark
Show steps
-
Review big data platforms, such as Hadoop and Spark
-
Go over PySpark dataframes
-
Examine RDDs, Spark SQL, and dataframe concepts
Work Through PySpark Practice Exercises
Show steps
These exercises will initiate the implementation of concepts and solidifies your understanding of PySpark.
Browse courses on
Pyspark
Show steps
-
Load PySpark and setup the environment
-
Implement PySpark dataframe operations
-
Execute PySpark SQL queries
Join a Mentoring Program for Data Engineering
Show steps
Enhance your understanding and practical skills by sharing your knowledge and guiding others in their data engineering journey.
Browse courses on
Mentorship
Show steps
-
Identify a mentoring program in data engineering
-
Apply and get matched with a mentee
-
Provide guidance and support to your mentee
11 other activities
Expand to see all activities and additional details
Show all 14 activities
Read "Spark: The Definitive Guide"
Show steps
Gain a comprehensive understanding of Spark's architecture, programming model, and advanced techniques.
View
Spark: The Definitive Guide: Big Data...
on Amazon
Show steps
-
Obtain a copy of the book
-
Read through the relevant chapters
-
Take notes and highlight important concepts
Explore Snowflake Documentation and Tutorials
Show steps
Self-guided exploration of resources will grant you more profound insights into the features and functionalities of Snowflake.
Browse courses on
Snowflake
Show steps
-
Familiarize yourself with Snowflake architecture and components
-
Review best practices for Snowflake data management
-
Experiment with Snowflake's scripting and programming capabilities
Run PySpark examples
Show steps
Reinforce your understanding of PySpark by running the examples provided in the course materials.
Browse courses on
Pyspark
Show steps
-
Navigate to the PySpark examples directory.
-
Run the examples using the provided commands.
-
Observe the output and compare it to the expected results.
Follow Databricks Academy Tutorials
Show steps
Supplement your learning by following structured tutorials from Databricks Academy to enhance your practical skills.
Browse courses on
Databricks
Show steps
-
Identify relevant tutorials
-
Follow the tutorials step-by-step
-
Complete the exercises and quizzes
Create a Resource Collection on Snowflake
Show steps
Build your knowledge base by gathering and organizing resources related to Snowflake's features and capabilities.
Browse courses on
Snowflake
Show steps
-
Identify relevant resources (e.g., documentation, tutorials, articles)
-
Organize the resources into a structured format
-
Share the resource collection with others
Attend a Data Analytics Workshop
Show steps
Participate in a hands-on workshop to gain practical experience and deepen your understanding of data analytics concepts.
Browse courses on
Data Analytics
Show steps
-
Identify a relevant workshop
-
Register and attend the workshop
-
Engage actively in the hands-on exercises
Spark SQL Practice Problems
Show steps
Reinforce your understanding of Spark SQL syntax and operations by attempting to solve practice problems.
Show steps
-
Access the practice problems
-
Attempt to solve the problems on your own
-
Review the solutions provided
Write a Blog Post on Data Engineering with Hadoop
Show steps
Solidify your understanding by explaining concepts related to Hadoop and data engineering in a blog post.
Browse courses on
Hadoop
Show steps
-
Choose a specific topic within Hadoop and data engineering
-
Research and gather information
-
Write a well-structured and informative blog post
-
Publish and promote your blog post
Build a Data Pipeline Prototype in Databricks
Show steps
Hands-on implementation of these concepts will greatly enhance your comprehension and real-world readiness.
Browse courses on
Azure Databricks
Show steps
-
Establish a Databricks environment
-
Design and develop your data pipeline architecture
-
Implement data ingestion, transformation, and visualization
Build a Data Pipeline with PySpark
Show steps
Apply your skills to a practical project involving data extraction, transformation, and loading using PySpark.
Browse courses on
Pyspark
Show steps
-
Define the project scope and objectives
-
Design the data pipeline architecture
-
Implement the pipeline using PySpark
-
Test and evaluate the pipeline
Contribute to an Open-Source Data Engineering Project
Show steps
Enhance your skills and contribute to the data engineering community by participating in open-source projects.
Browse courses on
Open Source
Show steps
-
Identify an open-source data engineering project
-
Explore the project's codebase and documentation
-
Identify an area where you can contribute
-
Submit a pull request with your contribution
Review Concepts in PySpark Study Materials
Show steps
Reviewing these topics will bring back important concepts in PySpark at the start of the course to stimulate your thinking.
Browse courses on
Pyspark
Show steps
Show steps
Show steps
- Review big data platforms, such as Hadoop and Spark
- Go over PySpark dataframes
- Examine RDDs, Spark SQL, and dataframe concepts
Work Through PySpark Practice Exercises
Show steps
These exercises will initiate the implementation of concepts and solidifies your understanding of PySpark.
Browse courses on
Pyspark
Show steps
Show steps
Show steps
- Load PySpark and setup the environment
- Implement PySpark dataframe operations
- Execute PySpark SQL queries
Join a Mentoring Program for Data Engineering
Show steps
Enhance your understanding and practical skills by sharing your knowledge and guiding others in their data engineering journey.
Browse courses on
Mentorship
Show steps
Show steps
Show steps
- Identify a mentoring program in data engineering
- Apply and get matched with a mentee
- Provide guidance and support to your mentee
11 other activities
Expand to see all activities and additional details
Show all 14 activities
Read "Spark: The Definitive Guide"
Show steps
Gain a comprehensive understanding of Spark's architecture, programming model, and advanced techniques.
View
Spark: The Definitive Guide: Big Data...
on Amazon
Show steps
Show steps
Show steps
- Obtain a copy of the book
- Read through the relevant chapters
- Take notes and highlight important concepts
Explore Snowflake Documentation and Tutorials
Show steps
Self-guided exploration of resources will grant you more profound insights into the features and functionalities of Snowflake.
Browse courses on
Snowflake
Show steps
Show steps
Show steps
- Familiarize yourself with Snowflake architecture and components
- Review best practices for Snowflake data management
- Experiment with Snowflake's scripting and programming capabilities
Run PySpark examples
Show steps
Reinforce your understanding of PySpark by running the examples provided in the course materials.
Browse courses on
Pyspark
Show steps
Show steps
Show steps
- Navigate to the PySpark examples directory.
- Run the examples using the provided commands.
- Observe the output and compare it to the expected results.
Follow Databricks Academy Tutorials
Show steps
Supplement your learning by following structured tutorials from Databricks Academy to enhance your practical skills.
Browse courses on
Databricks
Show steps
Show steps
Show steps
- Identify relevant tutorials
- Follow the tutorials step-by-step
- Complete the exercises and quizzes
Create a Resource Collection on Snowflake
Show steps
Build your knowledge base by gathering and organizing resources related to Snowflake's features and capabilities.
Browse courses on
Snowflake
Show steps
Show steps
Show steps
- Identify relevant resources (e.g., documentation, tutorials, articles)
- Organize the resources into a structured format
- Share the resource collection with others
Attend a Data Analytics Workshop
Show steps
Participate in a hands-on workshop to gain practical experience and deepen your understanding of data analytics concepts.
Browse courses on
Data Analytics
Show steps
Show steps
Show steps
- Identify a relevant workshop
- Register and attend the workshop
- Engage actively in the hands-on exercises
Spark SQL Practice Problems
Show steps
Reinforce your understanding of Spark SQL syntax and operations by attempting to solve practice problems.
Show steps
Show steps
Show steps
- Access the practice problems
- Attempt to solve the problems on your own
- Review the solutions provided
Write a Blog Post on Data Engineering with Hadoop
Show steps
Solidify your understanding by explaining concepts related to Hadoop and data engineering in a blog post.
Browse courses on
Hadoop
Show steps
Show steps
Show steps
- Choose a specific topic within Hadoop and data engineering
- Research and gather information
- Write a well-structured and informative blog post
- Publish and promote your blog post
Build a Data Pipeline Prototype in Databricks
Show steps
Hands-on implementation of these concepts will greatly enhance your comprehension and real-world readiness.
Browse courses on
Azure Databricks
Show steps
Show steps
Show steps
- Establish a Databricks environment
- Design and develop your data pipeline architecture
- Implement data ingestion, transformation, and visualization
Build a Data Pipeline with PySpark
Show steps
Apply your skills to a practical project involving data extraction, transformation, and loading using PySpark.
Browse courses on
Pyspark
Show steps
Show steps
Show steps
- Define the project scope and objectives
- Design the data pipeline architecture
- Implement the pipeline using PySpark
- Test and evaluate the pipeline
Contribute to an Open-Source Data Engineering Project
Show steps
Enhance your skills and contribute to the data engineering community by participating in open-source projects.
Browse courses on
Open Source
Show steps
Show steps
Show steps
- Identify an open-source data engineering project
- Explore the project's codebase and documentation
- Identify an area where you can contribute
- Submit a pull request with your contribution
Career center
Learners who complete Spark, Hadoop, and Snowflake for Data Engineering will develop knowledge and skills
that may be useful to these careers:
For more career information including salaries, visit:
OpenCourser.com/course/fe5c2t/spark
Reading list
We haven't picked any books for this reading list yet.
For more information about how these books relate to this course, visit:
OpenCourser.com/course/fe5c2t/spark
Share
Similar courses
Here are nine courses similar to
Spark, Hadoop, and Snowflake for Data Engineering.
Spark, Hadoop, and Snowflake for Data Engineering
OpenCourser.com/course/qg0bfd/spark
Spark, Hadoop, and Snowflake for Data Engineering
Most relevant
Machine Learning with Apache Spark
OpenCourser.com/course/s1yvpd/machine
Machine Learning with Apache Spark
Most relevant
Apache Spark for Data Engineering and Machine Learning
OpenCourser.com/course/p1xkqd/apache
Apache Spark for Data Engineering and Machine Learning
Most relevant
Data Engineering Essentials using SQL, Python, and PySpark
OpenCourser.com/course/t0sfb9/data
Data Engineering Essentials using SQL, Python, and PySpark
Most relevant
Data Engineering and Machine Learning using Spark
OpenCourser.com/course/wi74bl/data
Data Engineering and Machine Learning using Spark
Most relevant
Data Engineering using Kafka and Spark Structured Streaming
OpenCourser.com/course/01yxz7/data
Data Engineering using Kafka and Spark Structured...
Most relevant
Developing Spark Applications Using Scala & Cloudera
OpenCourser.com/course/y90x0l/developing
Developing Spark Applications Using Scala & Cloudera
Most relevant
Big Data Computing with Spark
OpenCourser.com/course/mqnvdu/big
Big Data Computing with Spark
Most relevant
Building Deep Learning Models on Databricks
OpenCourser.com/course/fk4s98/building
Building Deep Learning Models on Databricks
Most relevant
Effort
3 to 6 hours per week for 4 weeks
Level
Introductory
Via
edX
Institution
Pragmatic AI Labs
Instructors
Noah Gift
Kennedy Behrman
Session
March 31, 2024
- March 31, 2025
Language
English
Transcript
English
Good to know
Develops foundational and advanced skills for those in data science and engineering fields
Taught by experienced professionals in the industry
Explores a range of essential data engineering platforms and technologies
Covers project management and workflow optimization practices
Requires some prior programming experience, limiting accessibility for absolute beginners
Our mission
OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.
Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.
Find this site helpful? Tell a friend about us.
Affiliate disclosure
We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.
Your purchases help us maintain our catalog and keep our servers humming without ads.
Thank you for supporting OpenCourser.
© 2016 - 2024 OpenCourser