Building Machine Learning Models in Spark 2 from Pluralsight

Training ML models is a compute-intensive operation and is best done in a distributed environment. This course will teach you how Spark can efficiently perform data explorations, cleaning, aggregations, and train ML models all on one platform.

Spark is possibly the most popular engine for big data processing these days. In this course, Building Machine Learning Models in Spark 2, you will learn to build and train Machine Learning (ML) models such as regression, classification, clustering, and recommendation systems on Spark 2.x's distributed processing environment.

This course starts off with an introduction of the 2 ML libraries available in Spark 2; the older spark.mllib library built on top of RDDs and the newer spark.ml library built on top of dataframes. You will get to see the two compared to help you know when to pick one over the other.

You will get to see a classification model built using Decision Trees the old way, and see how you can implement the same model on the newer spark.ml library.

The course covers many features of Spark 2, including going over a brand new feature in Spark 2, the ML pipelines used to chain your data transformations and ML operations.

At the end of this course you will be comfortable using the advanced features that Spark 2 offers for machine learning. You'll learn to use components such as Transformers, Estimators, and Parameters within your ML pipelines to work with distributed training at scale.

What's inside

Syllabus

Course Overview

Machine Learning Packages: spark.mllib vs. spark.ml

Building Classification and Regression Models in Spark ML

Implementing Clustering and Dimensionality Reduction in Spark ML

Building Recommendation Systems in Spark ML

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Appropriate for learners with zero knowledge of Apache Spark

Teaches techniques that are standards in industry

Appropriate for learners in a number of fields

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Building Machine Learning Models in Spark 2 with these activities:

Organize course materials

Show steps

Prepares for efficient learning by organizing course resources.

Show steps

Gather course materials such as notes, assignments, and slides.
Create a system for organizing and storing the materials.
Review the materials regularly to reinforce learning.

Review basic data engineering

Show steps

Provides a foundation for understanding the Spark environment and its applications.

Browse courses on Data Engineering

Show steps

Review data modeling concepts and techniques.
Practice data cleaning and preparation.
Experiment with data visualization tools.

Review basic statistics

Show steps

Machine learning is built on a foundation of statistics. Reviewing foundational concepts will greatly enhance your ability to understand and apply the materials.

Browse courses on Basic Statistics

Show steps

Read overview of statistical concepts
Review specific statistical techniques used in data science
Practice applying these techniques to real world datasets

Seven other activities

Expand to see all activities and additional details

Show all ten activities

Work through Spark ML tutorial

Show steps

This course relies heavily on Spark ML. This tutorial will help you get up to speed with its features and capabilities.

Browse courses on Spark ML

Show steps

Find and read the Apache Spark ML tutorial
Follow the steps to build a simple machine learning model
Optional: Explore the tutorial's additional resources

Follow Spark tutorial series

Show steps

Provides hands-on experience with Spark's features and capabilities.

Show steps

Choose a Spark tutorial series that aligns with your learning goals.
Follow the tutorials step-by-step, experimenting with different examples.
Troubleshoot any errors or challenges encountered.

Share knowledge by mentoring peers

Show steps

Reinforces understanding by explaining concepts to others.

Show steps

Identify a platform or community where you can mentor others.
Answer questions, provide guidance, and share resources.
Reflect on your own understanding while helping others.

Practice building ML models in Spark

Show steps

Building models requires practice. Try to build models of your own beyond the scope of the course.

Browse courses on Machine Learning Models

Show steps

Choose a dataset that interests you
Decide on a machine learning model to build
Build the model in Spark ML
Evaluate the performance of your model

Mentor a junior data scientist or student

Show steps

Mentoring will force you to revisit the materials and solidify your own understanding as you teach it to others.

Show steps

Identify a junior data scientist or student who would benefit from your guidance
Meet with your mentee regularly to discuss their progress and provide support
Help your mentee develop their skills in Spark ML and data science

Build a Spark-based project

Show steps

Applies knowledge to a real-world problem, fostering deeper understanding.

Show steps

Identify a problem or opportunity that can be addressed using Spark.
Design and implement a Spark-based solution.
Evaluate the results and iterate on the solution to improve performance.

Participate in a Kaggle competition using Spark ML

Show steps

Kaggle competitions pit data scientists against one another to build the best models. This promotes deeper learning, collaboration, and industry recognition.

Browse courses on Kaggle Competition

Show steps

Find a Kaggle competition that interests you
Download the data provided by the competition
Build a machine learning model using Spark ML
Submit your model to the competition and track your progress
Collaborate with others to improve your model

Career center

Learners who complete Building Machine Learning Models in Spark 2 will develop knowledge and skills that may be useful to these careers:

Data Scientist

Data Scientists use machine learning to build models that can make predictions based on historical data. This course provides a solid foundation in the use of Spark to implement machine learning models. Data Scientists may also work with big data, which is often associated with Spark. This course's emphasis on building models in a distributed environment, as is often the case with big data, is also highly applicable to the role.

See salaries and explore the career path for Data Scientist

Big Data Analyst

Big Data Analysts are responsible for analyzing large datasets and identifying trends and patterns. This course provides a comprehensive overview of how to use Spark to process and analyze big data. Big Data Analysts may also work with machine learning models, which are often built using Spark. This course's coverage of machine learning in Spark would be highly applicable to a Big Data Analyst role.

See salaries and explore the career path for Big Data Analyst

Machine Learning Engineer

Machine Learning Engineers are responsible for designing, developing, and deploying machine learning models. This course would be a valuable addition to any Machine Learning Engineer's skillset. Spark is widely used in the industry for building and deploying machine learning models, and this course provides a thorough understanding of how to use Spark for these purposes.

See salaries and explore the career path for Machine Learning Engineer

Data Analyst

Data Analysts are responsible for collecting, cleaning, and analyzing data. This course provides a strong foundation in the use of Spark to perform data analysis tasks. Data Analysts may also work with machine learning models, which are often built using Spark. This course's coverage of machine learning in Spark would be beneficial to Data Analysts.

See salaries and explore the career path for Data Analyst

Software Engineer

Software Engineers are responsible for designing, developing, and testing software applications. This course provides a valuable overview of how to use Spark to perform data analysis and build machine learning models. Software Engineers may also work with big data, which is often processed using Spark. This course's emphasis on building models in a distributed environment would be highly applicable to a Software Engineer role.

See salaries and explore the career path for Software Engineer

Data Architecture Engineer

Data Architecture Engineers are responsible for designing and managing data architectures. This course provides a comprehensive overview of how to use Spark to process and analyze big data. Data Architecture Engineers may also work with machine learning models, which are often built using Spark. This course's coverage of machine learning in Spark would be helpful to Data Architecture Engineers.

See salaries and explore the career path for Data Architecture Engineer

Statistician

Statisticians are responsible for collecting, analyzing, and interpreting data. This course provides a solid foundation in the use of Spark to perform data analysis tasks. Statisticians may also work with machine learning models, which are often built using Spark. This course's coverage of machine learning in Spark would be helpful to Statisticians.

See salaries and explore the career path for Statistician

Data Engineer

Data Engineers are responsible for designing and managing data pipelines. This course provides a comprehensive overview of how to use Spark to process and analyze big data. Data Engineers may also work with machine learning models, which are often built using Spark. This course's coverage of machine learning in Spark would be helpful to Data Engineers.

See salaries and explore the career path for Data Engineer

Database Administrator

Database Administrators are responsible for managing and maintaining databases. This course provides a valuable overview of how to use Spark to process and analyze data. Database Administrators may also work with machine learning models, which are often built using Spark. This course's coverage of machine learning in Spark would be beneficial to Database Administrators.

See salaries and explore the career path for Database Administrator

Marketing Analyst

Marketing Analysts are responsible for analyzing marketing data and making recommendations. This course may be useful to Marketing Analysts who are interested in learning how to use Spark to process and analyze big data. Spark is widely used in the industry for big data analytics, and this course provides a solid foundation in the use of Spark for these purposes.

See salaries and explore the career path for Marketing Analyst

Product Manager

Product Managers are responsible for defining and managing the development of products. This course may be useful to Product Managers who are interested in learning how to use Spark to process and analyze big data. Spark is widely used in the industry for big data analytics, and this course provides a solid foundation in the use of Spark for these purposes.

See salaries and explore the career path for Product Manager

Financial Analyst

Financial Analysts are responsible for analyzing financial data and making recommendations. This course may be useful to Financial Analysts who are interested in learning how to use Spark to process and analyze big data. Spark is widely used in the industry for big data analytics, and this course provides a solid foundation in the use of Spark for these purposes.

See salaries and explore the career path for Financial Analyst

Project Manager

Project Managers are responsible for planning and managing projects. This course may be useful to Project Managers who are interested in learning how to use Spark to process and analyze big data. Spark is widely used in the industry for big data analytics, and this course provides a solid foundation in the use of Spark for these purposes.

See salaries and explore the career path for Project Manager

Business Analyst

Business Analysts are responsible for analyzing business data and identifying trends. This course may be useful to Business Analysts who are interested in learning how to use Spark to process and analyze big data. Spark is widely used in the industry for big data analytics, and this course provides a solid foundation in the use of Spark for these purposes.

See salaries and explore the career path for Business Analyst