Handling Batch Data with Apache Spark on Databricks from Pluralsight

This course will teach you how to transform and aggregate batch data using Apache Spark on the Azure Databricks platform using selection, filter, and aggregation queries, built-in and user-defined functions, and perform windowing and join operations on batch data.

Azure Databricks allows you to work with big data processing and queries using the Apache Spark unified analytics engine. Azure Databricks allows to work with a variety of batch sources and makes it seamless to analyze, visualize, and process data on the Azure Cloud Platform. In this course, Handling Batch Data with Apache Spark on Databricks, you will learn how to perform transformations and aggregations on batch data with selection, filtering, grouping, and ordering queries that use the DataFrame API. You will understand the difference between narrow transformations and wide transformations in Spark which will help you figure out why certain transformations are more efficient than others. You will also see how you can execute these same transformations by executing SQL queries on your data. Next, you will learn how you can implement your own custom user-defined functions to process your data. You will write code on Azure Databricks notebooks to define and register your UDFs and use them to transform your data. You will also understand how to define and use different flavors of vectorized UDFs for data processing and learn how vectorized UDFs are often more efficient than regular UDFs. Along the way, you will also see how you can read from Azure Cosmos DB as a source for your batch data. Finally, you will see how you can repartition your data in memory to improve processing performance, you will use window functions to compute statistics on your data and you will combine data frames using union and join operations. When you’re finished with this course you will have the skills and ability to perform advanced transformations and aggregations on batch data, including defining and using user-defined functions for processing.

What's inside

Syllabus

Course Overview

Transforming Data Using DataFrames

Transforming Data Using Spark SQL

Applying User-defined Functions to Transform Data

Processing Data Using Joins and Window Functions

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Develops data processing capabilities, which are core skills for data analysts and engineers

Explores concepts of batch data manipulation with Apache Spark and Azure Databricks, which is standard in industry

Offers practical experience through Azure Databricks notebooks, valuable for hands-on learners

Instructed by Janani Ravi, a recognized expert in data analytics

Requires prior experience with data processing concepts and programming

May require additional resources, such as a subscription fee, to access Azure Databricks notebooks

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Handling Batch Data with Apache Spark on Databricks with these activities:

Review basic DataFrame API

Show steps

Reviewing the fundamentals of the DataFrame API will prepare you to build on this knowledge and perform complex transformations and aggregations on batch data in the course.

Browse courses on DataFrames

Show steps

Read the Spark documentation on DataFrames
Review examples and tutorials on using DataFrames in Spark

Read 'Learning Spark: Lightning-Fast Data Analysis'

Show steps

This book provides a comprehensive overview of Spark, from its core concepts to advanced techniques. Reading it will complement the course material and deepen your understanding of Spark.

View Learning Spark: Lightning-Fast Big Data Analysis on Amazon

Show steps

Read the chapters relevant to the course topics
Work through the examples and exercises in the book

Explore Spark's Window Functions

Show steps

Understanding how to use Spark's Window Functions will equip you with powerful techniques for performing complex aggregations and computations on your data.

Show steps

Read the Spark documentation on Window Functions
Follow a tutorial on using Window Functions in Spark
Try creating your own queries using Window Functions

Show all three activities

Career center

Learners who complete Handling Batch Data with Apache Spark on Databricks will develop knowledge and skills that may be useful to these careers:

Data Scientist

A data scientist, using both programming and statistical analysis, uncovers knowledge from data. Data science is used to create predictive models that can, for example, predict weather patterns or determine sales outcomes. This course is a great introduction to transforming and aggregating data using Apache Spark on Azure Databricks, skills that are essential to data science.

See salaries and explore the career path for Data Scientist

Data Engineer

A data engineer is responsible for designing, building, and maintaining data pipelines. This course may be useful for data engineers who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate batch data.

See salaries and explore the career path for Data Engineer

Data Architect

A data architect designs and builds the architecture for data systems. This course may be useful for data architects who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate batch data.

See salaries and explore the career path for Data Architect

Data Visualization Analyst

A data visualization analyst creates visual representations of data. This course may be useful for data visualization analysts who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate data in order to create visualizations that can be used to communicate insights to stakeholders.

See salaries and explore the career path for Data Visualization Analyst

Machine Learning Engineer

A machine learning engineer constructs and deploys machine learning models. This course may be useful for machine learning engineers who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate batch data for use in machine learning models.

See salaries and explore the career path for Machine Learning Engineer

Statistician

A statistician collects, analyzes, interprets, and presents data. This course may be useful for statisticians who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate large amounts of data.

See salaries and explore the career path for Statistician

Operations Research Analyst

An operations research analyst uses mathematical and analytical methods to solve complex business problems. This course may be useful for operations research analysts who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate large amounts of data.

See salaries and explore the career path for Operations Research Analyst

Quantitative Analyst

A quantitative analyst uses mathematical and statistical models to analyze and interpret financial data. This course may be useful for quantitative analysts who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate large amounts of financial data.

See salaries and explore the career path for Quantitative Analyst

Computer Scientist

A computer scientist researches and develops new computing technologies. This course may be useful for computer scientists who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate large amounts of data.

See salaries and explore the career path for Computer Scientist

Software Developer

A software developer designs, builds, and maintains software systems. This course may be useful for software developers who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate batch data.

See salaries and explore the career path for Software Developer

Actuary

An actuary is a business professional who assesses financial risks in the insurance and finance industries. This course may be useful for actuaries who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate large amounts of data.

See salaries and explore the career path for Actuary

Database Administrator

A database administrator is responsible for the installation, configuration, maintenance, and performance of database management systems. This course may be useful for database administrators who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate large amounts of data.

See salaries and explore the career path for Database Administrator

Business Analyst

A business analyst is a professional who can identify and solve business problems through the use of data analysis. This course may be useful for business analysts who need to learn how to use Apache Spark on Azure Databricks to transform and aggregate batch data in order to gain insights that can be used to improve business processes.

See salaries and explore the career path for Business Analyst

Data Analyst

Data analysts gather, clean, and analyze data in order to obtain insights that may be useful for a business. This course, Handling Batch Data with Apache Spark on Databricks, may be useful to anyone in this field because it shows how to read and analyze data from a variety of sources, including Azure Cosmos DB.

See salaries and explore the career path for Data Analyst

Software Engineer

Software engineers research, plan, design, develop, test, and deploy computer systems and applications. This course can be useful for software engineers who are working on big data projects and need to learn how to use Apache Spark on Azure Databricks to transform and aggregate batch data.

See salaries and explore the career path for Software Engineer