Handling Fast Data with Apache Spark SQL and Streaming from Pluralsight

Apache Spark is a leader in enabling quick and efficient data processing. This course will teach you how to use Spark's SQL, Streaming, and even the newer Structured Streaming APIs to create applications able to handle data as it arrives.

Analyzing data used to be something you did once a night. Now you need to be able to process data on the fly so you can provide up to the minute insights. But, how do you accomplish in real time what used to take hours without a complicated code base? In this course, Handling Fast Data with Apache Spark SQL and Streaming, you'll learn to use Apache Spark Streaming and SQL libraries as a great way to handle this new world of real time, fast data processing. First, you'll dive into SparkSQL. Next, you'll explore how to catch potential fraud by analyzing streams with Spark Streaming. Finally, you'll discover the newer Structured Streaming API. By the end of this course, you'll have a deeper understanding of these APIs, along with a number of streaming concepts that have driven the API design.

What's inside

Syllabus

Course Overview

Introduction

Querying Data with the DataFrames (Part 1)

Querying Data with the DataFrames (Part 2)

Improving Type Safety with Datasets

Processing Data with the Streaming API

Optimizing, Structured Streaming, and Spark 2.x

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Intended for learners with some experience in data querying and analytics, this course includes advanced topics like optimizing processing and structured streaming

Taught by Justin Pihony, a respected expert in big data analysis

Covers Spark SQL's dataframe API and Spark Streaming's structured streaming API

Examines both the theoretical underpinnings and practical applications of real-time data processing with Apache Spark

May be less accessible for learners without prior experience in data engineering or Apache Spark

Course materials are up-to-date, using Spark 2.x and its latest features

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Handling Fast Data with Apache Spark SQL and Streaming with these activities:

Review Apache Spark Concepts

Show steps

Refresh your understanding of Apache Spark's fundamental concepts before the course starts.

Browse courses on Apache Spark

Show steps

Revisit documentation or tutorials on Apache Spark
Review key concepts such as RDDs and transformations

Gather Resources on Spark Streaming

Show steps

Compile a collection of useful resources on Spark Streaming to support your learning.

Browse courses on Spark Streaming

Show steps

Search for articles, tutorials, and documentation on Spark Streaming
Organize the resources into a structured collection

Practice Data Transformation in SparkSQL

Show steps

Practice data transformation to reinforce your knowledge of SparkSQL's functionality.

Browse courses on SparkSQL

Show steps

Create a DataFrame from a CSV file
Select and rename columns
Filter data based on criteria
Join two DataFrames

Five other activities

Expand to see all activities and additional details

Show all eight activities

Follow Tutorials on Structured Streaming API

Show steps

Expand your knowledge of Structured Streaming API through guided tutorials.

Show steps

Find a tutorial on Structured Streaming API
Work through the tutorial step-by-step
Implement a simple use case of Structured Streaming API

Join a Study Group for Apache Spark

Show steps

Engage with peers in a study group to discuss concepts, share knowledge, and reinforce your understanding of Apache Spark.

Browse courses on Apache Spark

Show steps

Find or start a study group for Apache Spark
Actively participate in discussions and knowledge sharing

Build a Spark Streaming Application to Analyze Data Streams

Show steps

Develop a real-world application of Spark Streaming to improve your understanding of the technology.

Browse courses on Spark Streaming

Show steps

Define the data source and streaming parameters
Process data streams with transformation and filtering
Set up a streaming visualization dashboard

Build a Real-Time Data Analytics Dashboard with Spark

Show steps

Apply your knowledge to develop a practical project that combines Spark for data processing and a visualization tool for presenting insights.

Browse courses on Spark

Show steps

Design the dashboard and data pipelines
Implement data processing and analytics using Spark
Create visualizations and integrate them into the dashboard

Contribute to Spark Community Projects

Show steps

Engage with the Apache Spark community by contributing to open-source projects and expanding your hands-on experience.

Browse courses on Spark

Show steps

Identify and choose a Spark community project to contribute to
Follow the project's guidelines and contribute code or documentation
Participate in discussions and get feedback from the community

Career center

Learners who complete Handling Fast Data with Apache Spark SQL and Streaming will develop knowledge and skills that may be useful to these careers:

Data Architect

Data Architects design and implement data architectures. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Data Architects to design and develop data-intensive systems, and proficiency in it can make you a more competitive candidate for Data Architect positions.

See salaries and explore the career path for Data Architect

Big Data Architect

Big Data Architects design and implement big data systems. This course may be useful as it provides a foundation in Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Big Data Architects to design and develop data-intensive systems, and proficiency in it can make you a more competitive candidate for Big Data Architect positions.

See salaries and explore the career path for Big Data Architect

Data Warehouse Architect

Data Warehouse Architects design and implement data warehouses. This course may be useful as it provides a foundation in Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Data Warehouse Architects to design and develop data-intensive systems, and proficiency in it can make you a more competitive candidate for Data Warehouse Architect positions.

See salaries and explore the career path for Data Warehouse Architect

Data Scientist

Data Scientists use statistical and machine learning techniques to extract insights from data. This course can be useful as it provides a foundation in Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Data Scientists to handle large volumes of data, and proficiency in it can make you a more competitive candidate for Data Scientist positions.

See salaries and explore the career path for Data Scientist

Machine Learning Engineer

Machine Learning Engineers design, build, and deploy machine learning models. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Machine Learning Engineers to handle large volumes of data, and proficiency in it can increase your chances of success in this field.

See salaries and explore the career path for Machine Learning Engineer

Enterprise Architect

Enterprise Architects design and implement enterprise-wide IT systems. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Enterprise Architects to design and develop data-intensive systems, and proficiency in it can make you a more competitive candidate for Enterprise Architect positions.

See salaries and explore the career path for Enterprise Architect

Data Engineer

Data Engineers are responsible for designing and building data pipelines, ensuring that data is clean, reliable, and accessible to analysts and other users. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is widely used in data engineering roles, and proficiency in it can increase your chances of success in this field.

See salaries and explore the career path for Data Engineer

Data Analytics Manager

Data Analytics Managers oversee the collection, analysis, and interpretation of data. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Data Analytics Managers to handle large volumes of data, and proficiency in it can make you a more competitive candidate for Data Analytics Manager positions.

See salaries and explore the career path for Data Analytics Manager

Business Analyst

Business Analysts use data to identify and solve business problems. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Business Analysts to handle large volumes of data, and proficiency in it can make you a more competitive candidate for Business Analyst positions.

See salaries and explore the career path for Business Analyst

Systems Analyst

Systems Analysts design and implement computer systems. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Systems Analysts to design and develop data-intensive systems, and proficiency in it can make you a more competitive candidate for Systems Analyst positions.

See salaries and explore the career path for Systems Analyst

Database Administrator

Database Administrators are responsible for managing and maintaining databases. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Database Administrators to handle large volumes of data, and proficiency in it can make you a more competitive candidate for Database Administrator positions.

See salaries and explore the career path for Database Administrator

Software Architect

Software Architects design and develop software systems. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Software Architects to design and develop data-intensive systems, and proficiency in it can make you a more competitive candidate for Software Architect positions.

See salaries and explore the career path for Software Architect

Software Development Manager

Software Development Managers oversee the development of software applications. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by Software Development Managers to manage the development of data-intensive applications, and proficiency in it can make you a more competitive candidate for Software Development Manager positions.

See salaries and explore the career path for Software Development Manager

Software Engineer

Software Engineers are responsible for analyzing user needs, designing, implementing, and testing software applications. This course can be useful as it provides a foundation in Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by organizations to handle large volumes of data, and proficiency in it can make you a more competitive candidate for Software Engineer positions.

See salaries and explore the career path for Software Engineer

IT Manager

IT Managers are responsible for the planning, implementation, and management of an organization's IT systems. This course can be useful as it provides hands-on experience with Apache Spark, a popular big data processing framework. Apache Spark is increasingly used by IT Managers to manage data-intensive systems, and proficiency in it can make you a more competitive candidate for IT Manager positions.

See salaries and explore the career path for IT Manager