Building Your First ETL Pipeline Using Azure Databricks from Pluralsight

In this course, you will learn about the Spark based Azure Databricks platform, see how to setup the environment, quickly build extract, transform, and load steps of your data pipelines, orchestrate it end-to-end, and run it automatically and reliably.

With an exponential growth in data volumes, increase in types of data sources, faster data processing needs and dynamically changing business requirements, traditional ETL tools are facing the challenge to keep up to the needs of modern data pipelines. While Apache Spark is very popular for big data processing and can help us overcome these challenges, managing the Spark environment is no cakewalk.

In this course, Building Your First ETL Pipeline Using Azure Databricks, you will gain the ability to use the Spark based Databricks platform running on Microsoft Azure, and leverage its features to quickly build and orchestrate an end-to-end ETL pipeline. And all this while learning about collaboration options and optimizations that it brings, but without worrying about the infrastructure management.

First, you will learn about the fundamentals of Spark, about the Databricks platform and features, and how it is runs on Microsoft Azure.

Next, you will discover how to setup the environment, like workspace, clusters and security, and build each phase of extract, transform and load separately, to implement the dimensional model.

Finally, you will explore how to orchestrate that using Databricks jobs and Azure Data Factory, followed by other features, like Databricks APIs and Delta Lake, to help you build automated and reliable data pipelines.

When you’re finished with this course, you will have the skills and knowledge of Azure Databricks platform needed to build and orchestrate an end-to-end ETL pipeline.

What's inside

Syllabus

Course Overview

Getting Started with Azure Databricks

Setting up Your Databricks Environment

Extracting Data from Multiple Sources

Transforming and Cleaning Data

Loading Data

Orchestrating ETL Pipeline

Building Better Pipelines on Databricks

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Incorporates the latest version of Apache Spark and Azure Databricks, making it highly relevant for working professionals

Taught by Mohit Batra, an experienced instructor in Big Data technologies

Provides a comprehensive understanding of the end-to-end ETL pipeline using Azure Databricks

Develops practical skills in building, orchestrating, and automating ETL pipelines using Spark-based Databricks

Covers cloud-based data engineering techniques, which are highly sought after in the industry

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Building Your First ETL Pipeline Using Azure Databricks with these activities:

Review Apache Spark documentation

Show steps

Review the Apache Spark documentation to refresh your knowledge of the platform and its capabilities.

Browse courses on Azure Databricks

Show steps

Visit the Apache Spark website.
Read the Apache Spark documentation.
Review the Apache Spark examples.

Review course syllabus

Show steps

Review the course syllabus to familiarize yourself with the topics that will be covered and the expectations for the course.

Browse courses on Azure Databricks

Show steps

Read through the syllabus carefully.
Identify the key topics that will be covered.
Note any prerequisites or co-requisites for the course.

Review 'Learning Spark: Lightning-Fast Data Analytics'

Show steps

This book provides a comprehensive overview of Apache Spark, the open-source cluster computing framework for big data analytics, and will help you understand the fundamentals of Spark and how to use it for data processing and analytics.

View Learning Spark: Lightning-Fast Big Data Analysis on Amazon

Show steps

Read chapters 1-3 to gain an understanding of the basics of Spark.
Complete the exercises in chapters 1-3 to practice using Spark.
Review the key concepts covered in chapters 1-3.

Five other activities

Expand to see all activities and additional details

Show all eight activities

Complete Databricks Academy tutorials

Show steps

The Databricks Academy provides a series of free, interactive tutorials that will help you learn how to use Azure Databricks and Apache Spark for data analytics.

Browse courses on Azure Databricks

Show steps

Sign up for a free Databricks Academy account.
Complete the 'Introduction to Azure Databricks' tutorial.
Complete the 'Introduction to Apache Spark' tutorial.
Complete at least one additional tutorial that is relevant to your interests.

Attend an Azure Databricks workshop

Show steps

Attending an Azure Databricks workshop can help you learn more about the platform and how to use it for data analytics.

Browse courses on Azure Databricks

Show steps

Find an Azure Databricks workshop that is relevant to your interests.
Register for the workshop.
Attend the workshop and participate in the activities.
Follow up with the workshop instructors or other participants after the workshop.

Practice Spark SQL queries

Show steps

Practice writing Spark SQL queries to extract, transform, and load data. This will help you develop your skills in data manipulation and analysis.

Browse courses on Azure Databricks

Show steps

Create a Spark DataFrame from a CSV file.
Use Spark SQL to perform data manipulation operations, such as filtering, sorting, and joining.
Use Spark SQL to perform data analysis operations, such as aggregation and statistical analysis.
Save the results of your queries to a new CSV file.

Mentor other students in the course

Show steps

Mentoring other students can help you solidify your understanding of the course material and develop your leadership skills.

Show steps

Join the course discussion forum.
Answer questions from other students.
Provide feedback on other students' work.
Lead a study group for other students.

Create a data pipeline using Azure Databricks

Show steps

Create a data pipeline using Azure Databricks to extract, transform, and load data from a source to a destination. This will help you develop your skills in data engineering and data management.

Browse courses on Azure Databricks

Show steps

Define the source and destination of your data.
Create a Spark DataFrame from the source data.
Use Spark SQL to perform data manipulation operations, such as filtering, sorting, and joining.
Use Spark SQL to perform data analysis operations, such as aggregation and statistical analysis.
Save the results of your pipeline to the destination.

Career center

Learners who complete Building Your First ETL Pipeline Using Azure Databricks will develop knowledge and skills that may be useful to these careers:

Data Analyst

Data Analysts gather, analyze, interpret, and present data to help businesses make informed decisions. This course may be useful for aspiring Data Analysts, as it covers the skills needed to extract, transform, and load data into a data warehouse or data lake, which are essential tasks for data analysis.

See salaries and explore the career path for Data Analyst

Data Scientist

Data Scientists use data to build predictive models and gain insights for businesses. This course may be useful for aspiring Data Scientists, as it provides a foundation in the Azure Databricks platform, which is widely used for data science and machine learning.

See salaries and explore the career path for Data Scientist

Data Engineer

Data Engineers design, build, and maintain data pipelines that collect, clean, and transform data from various sources to support data analysis and decision-making. This course may be helpful for aspiring Data Engineers, as it provides hands-on experience with the Azure Databricks platform, which is widely used for building data pipelines.

See salaries and explore the career path for Data Engineer

Data Architect

Data Architects design and manage the architecture of data systems, ensuring that they meet the business needs and technical requirements. This course may be useful for aspiring Data Architects, as it provides a foundation in the Azure Databricks platform, which is widely used for building and managing data pipelines and data warehouses.

See salaries and explore the career path for Data Architect

Software Engineer

Software Engineers design, develop, and maintain software applications. This course may be useful for Software Engineers who want to specialize in data engineering or data science, as it provides hands-on experience with the Azure Databricks platform, which is widely used for building data pipelines and data analysis applications.

See salaries and explore the career path for Software Engineer

Business Analyst

Business Analysts analyze business processes and data to identify inefficiencies and opportunities for improvement. This course may be useful for aspiring Business Analysts, as it provides skills in data extraction, transformation, and analysis, which are essential for understanding business processes and making data-driven decisions.

See salaries and explore the career path for Business Analyst

Database Administrator

Database Administrators manage and maintain databases, ensuring that they are available, reliable, and secure. This course may be useful for aspiring Database Administrators, as it provides hands-on experience with the Azure Databricks platform, which can be used to manage and administer data warehouses and data lakes.

See salaries and explore the career path for Database Administrator

Project Manager

Project Managers plan, execute, and deliver projects. This course may be useful for aspiring Project Managers who want to specialize in data engineering or data science projects, as it provides a foundation in the Azure Databricks platform, which is widely used for building data pipelines and data analysis applications.

See salaries and explore the career path for Project Manager

Product Manager

Product Managers define the vision and roadmap for products. This course may be useful for aspiring Product Managers who want to specialize in data-driven products, as it provides skills in data extraction, transformation, and analysis, which are essential for understanding customer needs and building successful products.

See salaries and explore the career path for Product Manager

ETL Engineer

ETL Engineers design, develop, test, and maintain ETL (extract, transform, and load) systems that move data from multiple sources into a central data warehouse or data lake. This course may be useful for aspiring ETL Engineers, as it covers the fundamentals of Spark, the Databricks platform, and how to build and orchestrate end-to-end ETL pipelines using these technologies.

See salaries and explore the career path for ETL Engineer

Technical Writer

Technical Writers create technical documentation, such as user manuals, white papers, and training materials. This course may be useful for aspiring Technical Writers who want to specialize in data engineering or data science, as it provides a foundation in the Azure Databricks platform, which is widely used for building data pipelines and data analysis applications.

See salaries and explore the career path for Technical Writer

Sales Engineer

Sales Engineers help customers understand and purchase technical products and services. This course may be useful for aspiring Sales Engineers who want to specialize in data engineering or data science, as it provides a foundation in the Azure Databricks platform, which is widely used for building data pipelines and data analysis applications.

See salaries and explore the career path for Sales Engineer

Consultant

Consultants provide advice and guidance to clients on a variety of business and technical topics. This course may be useful for aspiring Consultants who want to specialize in data engineering or data science, as it provides a foundation in the Azure Databricks platform, which is widely used for building data pipelines and data analysis applications.

See salaries and explore the career path for Consultant

Educator

Educators teach students at all levels, from elementary school to university.

See salaries and explore the career path for Educator

Financial Analyst

Financial Analysts provide financial advice and guidance to individuals and organizations.

See salaries and explore the career path for Financial Analyst