Big Data Analysis with Scala and Spark from Coursera

Manipulating big data distributed over a cluster using functional concepts is rampant in industry, and is arguably one of the first widespread industrial uses of functional ideas. This is evidenced by the popularity of MapReduce and Hadoop, and most recently Apache Spark, a fast, in-memory distributed collections framework written in Scala. In this course, we'll see how the data parallel paradigm can be extended to the distributed case, using Spark throughout. We'll cover Spark's programming model in detail, being careful to understand how and when it differs from familiar programming models, like shared-memory parallel collections or sequential Scala collections. Through hands-on examples in Spark and Scala, we'll learn when important issues related to distribution like latency and network communication should be considered and how they can be addressed effectively for improved performance.

Learning Outcomes. By the end of this course you will be able to:

- read data from persistent storage and load it into Apache Spark,

- manipulate data with Spark and Scala,

- express algorithms for data analysis in a functional style,

- recognize how to avoid shuffles and recomputation in Spark,

Recommended background: You should have at least one year programming experience. Proficiency with Java or C# is ideal, but experience with other languages such as C/C++, Python, Javascript or Ruby is also sufficient. You should have some familiarity using the command line. This course is intended to be taken after Parallel Programming: https://www.coursera.org/learn/parprog1.

What's inside

Syllabus

Getting Started + Spark Basics

Get up and running with Scala on your computer. Complete an example assignment to familiarize yourself with our unique way of submitting assignments. In this week, we'll bridge the gap between data parallelism in the shared memory scenario (learned in the Parallel Programming course, prerequisite) and the distributed scenario. We'll look at important concerns that arise in distributed systems, like latency and failure. We'll go on to cover the basics of Spark, a functionally-oriented framework for big data processing in Scala. We'll end the first week by exercising what we learned about Spark by immediately getting our hands dirty analyzing a real-world data set.

Reduction Operations & Distributed Key-Value Pairs

This week, we'll look at a special kind of RDD called pair RDDs. With this specialized kind of RDD in hand, we'll cover essential operations on large data sets, such as reductions and joins.

Partitioning and Shuffling

This week we'll look at some of the performance implications of using operations like joins. Is it possible to get the same result without having to pay for the overhead of moving data over the network? We'll answer this question by delving into how we can partition our data to achieve better data locality, in turn optimizing some of our Spark jobs.

Structured data: SQL, Dataframes, and Datasets

With our newfound understanding of the cost of data movement in a Spark job, and some experience optimizing jobs for data locality last week, this week we'll focus on how we can more easily achieve similar optimizations. Can structured data help us? We'll look at Spark SQL and its powerful optimizer which uses structure to apply impressive optimizations. We'll move on to cover DataFrames and Datasets, which give us a way to mix RDDs with the powerful automatic optimizations behind Spark SQL.

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Taught by Prof. Heather Miller, who are recognized for their work in parallel programming in Scala

Examines big data manipulation using functional concepts, which is standard in industry

Develops familiarity with Apache Spark, a fast, in-memory distributed collections framework written in Scala

Assumes proficiency with Java or C# or some familiarity with C/C++, Python, Javascript or Ruby

Requires knowledge of using the command line

Completion of Parallel Programming course is recommended as a prerequisite

Reviews summary

Spark and scala for big data

Students' reviews for Big Data Analysis with Scala and Spark are mixed. Some learners appreciate the course, calling it great, but others say it's lower quality compared to earlier courses in the specialization

One student thinks this course is great.

"Great course."

"Thanks for everything."

One student thinks this course is of a lower quality than earlier courses in the specialization.

"4th Course of the Specialization " Functional Programming In Scala": clearly down in quality compared to 2 first ones."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Big Data Analysis with Scala and Spark with these activities:

Read 'Learning Spark' by Holden Karau, Andy Konwinski, Patrick Wendell, and Matei Zaharia

Show steps

Read 'Learning Spark' to gain a comprehensive understanding of the Spark framework, its architecture, and best practices for data processing and analysis.

View Learning Spark: Lightning-Fast Big Data Analysis on Amazon

Show steps

Read 'Learning Spark' and take notes on key concepts and techniques.
Complete the exercises and examples provided in the book to practice applying Spark in real-world scenarios.

Participate in a study group to discuss Spark concepts

Show steps

Join a study group or discussion forum to connect with other learners and discuss Spark concepts, share knowledge, and enhance your understanding.

Show steps

Join a study group or online discussion forum focused on Spark.
Actively participate in discussions, asking questions, sharing insights, and collaborating with others.

Build a small Spark application for data analysis

Show steps

Develop a mini project that utilizes Spark to analyze a dataset and gain practical experience in applying Spark concepts and techniques.

Show steps

Choose a dataset and define a specific data analysis task.
Design and implement a Spark application to perform the data analysis task.
Evaluate the results and identify areas for improvement.

Show all three activities

Career center

Learners who complete Big Data Analysis with Scala and Spark will develop knowledge and skills that may be useful to these careers:

Data Analyst

Data Analysts use statistical and mathematical modeling and other data analysis techniques to extract meaningful information from data. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Data Analyst by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Data Analyst

Data Scientist

Data Scientists work with data to create knowledge. This can involve collecting, cleaning, and analyzing data, as well as developing algorithms and models to make predictions or recommendations. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Data Scientist by teaching you the basics of data analysis and how to use Apache Spark, a framework for big data processing in Scala.

See salaries and explore the career path for Data Scientist

Machine Learning Engineer

Machine Learning Engineers design and develop machine learning models, which are used to make predictions or recommendations based on data. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Machine Learning Engineer by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Machine Learning Engineer

Software Engineer

Software Engineers design, develop, test, and maintain software systems. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Software Engineer by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Software Engineer

Statistician

Statisticians collect, analyze, interpret, and present data. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Statistician by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Statistician

Data Engineer

Data Engineers design, develop, test, and maintain data systems. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Data Engineer by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Data Engineer

Database Administrator

Database Administrators manage and maintain databases. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Database Administrator by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Database Administrator

Business Analyst

Business Analysts use data to help businesses make informed decisions. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Business Analyst by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Business Analyst

Operations Research Analyst

Operations Research Analysts use mathematical and statistical models to solve business problems. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming an Operations Research Analyst by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Operations Research Analyst

Financial Analyst

Financial Analysts use data to evaluate investments and make recommendations to clients. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Financial Analyst by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Financial Analyst

Market Researcher

Market Researchers use data to understand consumer behavior and trends. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Market Researcher by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Market Researcher

User Experience Researcher

User Experience Researchers use data to understand how users interact with products and services. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a User Experience Researcher by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for User Experience Researcher

Quantitative Analyst

Quantitative Analysts use data to model and analyze financial markets. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Quantitative Analyst by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Quantitative Analyst

Actuary

Actuaries use data to assess and manage risk. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming an Actuary by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Actuary

Data Visualization Specialist

Data Visualization Specialists use data to create visual representations of data. This course in Big Data Analysis with Scala and Spark can help you build a foundation for becoming a Data Visualization Specialist by teaching you the basics of Apache Spark, a framework for big data processing in Scala, as well as how to express algorithms for data analysis in a functional style.

See salaries and explore the career path for Data Visualization Specialist