Apache Spark with Scala - Hands On with Big Data!

A brief introduction to the course, and then we'll get your development environment for Spark and Scala all set up on your desktop, using IntelliJ and SBT. A quick test application will confirm Spark is working on your system!

Let's review some of the high-level material you've learned about Apache Spark so far.

We'll go over the basic syntax and structure of Scala code with lots of examples. It's backwards from most other languages, but you quickly get used to it.

We'll go over the basic syntax and structure of Scala code with lots of examples. It's backwards from most other languages, but you quickly get used to it. Part 2 of 2, with some hands-on practice at the end.

Scala is a functional programming language, and so functions are central to the language. We'll go over the many ways functions can be declared and used in Scala, and practice what you've learned.

We'll cover the common data structures in Scala such as Map and List, and put them into practice.

The core object of Spark programming is the Resilient Distributed Dataset, or RDD. Once you know how to use RDD's, you know how to use Spark. We'll go over what they are, and what you can do with them.

Now that we understand Scala and have the theory of Spark behind us, let's start with a simple example of using RDD's to count up how many of each rating exists in the MovieLens data set.

How does Spark convert your script into a Directed Acyclic Graph and figure out how to distribute it on a cluster? Understanding how this process works under the hood can be important in writing optimal Spark driver scripts.

RDD's that contain a tuple of two values are key/value RDD's, and you can use them much like you might use a NoSQL data store. We'll use key/value RDD's to figure out the average number of friends by age in some fake social network data.

We'll run the average friends by age example on your desktop, and give you some ideas for further extending this script on your own.

We'll cover how to filter data out of an RDD efficiently, and illustrate this with a new example that finds the minimum temperature by location using real weather data.

We'll run our minimum temperature by location example, and modify it to find maximum temperatures as well. Plus, some ideas for extending this script on your own.

flatmap() on an RDD can return variable amounts of new entries into the resulting RDD. We'll use this as part of a hands-on example that finds how often each word is used inside a real book's text.

We extend the previous lecture's example by using regular expressions to better extract words from our book.

Finally, we sort the final results to see what the most common words in this book really are! And some ideas to extend this script on your own.

Your assignment: write a script that finds the total amount spent per customer using some fabricated e-commerce data, using what you've learned so far.

We'll review my solution to the previous lecture's assignment, and challenge you further to sort your results to find the biggest spenders.

Check your results for finding the biggest spenders in our e-commerce data against my own solution.

Understand SparkSQL and the DataFrame and DataSet API's used for querying structured data in an efficient, scalable manner.

We'll revisit our fabricated social network data, but load it into a DataFrame and analyze it with actual SQL queries!

We'll analyze our social network data another way - this time using SQL-like functions on a DataSet, instead of actual SQL query strings.

Earlier we broke down the average number of friends by age using RDD's - see if you can do it using DataSets instead!

We'll revisit our movie ratings data set, and start off with a simple example to find the most-rated movie.

Broadcast variables can be used to share small amounts of data to all of the machines on your cluster. We'll use them to share a lookup table of movie ID's to movie names, and use that to get movie names in our final results.

We introduce the Marvel superhero social network data set, and write a script to find the most-connected superhero in it. It's not who you might think!

As a more complex example, we'll apply a breadth-first-search (BFS) algorithm to the Marvel dataset to compute the degrees of separation between any two superheroes. In this lecture, we go over how BFS works.

We'll go over our strategy for implementing BFS within a Spark script that can be distributed, and introduce the use of Accumulators to maintain running totals that are synced across a cluster.

Finally, we'll review the code for finding the degrees of separation using breadth-first-search, run it, and see the results!

Back to our movie ratings data - we'll discover movies that are similar to each other just based on user ratings. We'll cover the algorithm, and how to implement it as a Spark script.

We'll run our movie similarties script and see the results.

Your challenge: make the movie similarity results even better! Here are some ideas for you to try out.

In a production environment, you'll use spark-submit to start your driver scripts from a command line, cron job, or the like. We'll cover the details on what you need to do differently in this case.

Spark / Scala scripts that have external dependencies can be bundled up into self-contained packages using the SBT tool. We'll use SBT to package up our movie similarities script as an exercise.

Amazon Web Services (AWS) offers the Elastic MapReduce service (EMR,) which gives us a way to rent time on a Hadoop cluster of our choosing - with Spark pre-installed on it. We'll use EMR to illustrate running a Spark script on a real cluster, so let's go over what EMR is and how it works first.

Let's compute movie similarities on a real cluster in the cloud, using one million user ratings!

Explicitly partitioning your Datasets and RDD's can be an important optimization; we'll go over when and how to do this.

Other tips and tricks for taking your script to a real cluster and getting it to run as you expect.

How to troubleshoot Spark jobs on a cluster using the Spark UI and logs, and more on managing dependencies of your script and data.

MLLib offers several distributed machine learning algorithms that you can run on a Spark cluster. We'll cover what MLLib can do and how it fits in.

We'll use MLLib's Alternating Least Squares recommender algorithm to produce movie recommendations using our MovieLens ratings data. The results are... unexpected!

A brief overview of what linear regression is and how it works, followed by a hands-on example of finding a regression and applying it to fabricated page speed vs. revenue data.

We'll run our Spark ML example of linear regression, using DataFrames.

Spark Streaming allows you create Spark driver scripts that run indefinitely, continually processing data as it streams in! We'll cover how it works and what it can do, using the original DStream micro-batch API.

Structured Streaming is a newer DataFrame-based API in Spark for writing continuous applications.

We cover Spark's GraphX library and how it works.

We'll revisit our "superhero degrees of separation" example, and see how its breadth-first-search algorithm could be implemented using Pregel and GraphX.

We'll use GraphX and Pregel to recreate our earlier results analyzing the superhero social network data - but with a lot less code!

You made it to the end! Here are some book recommendations if you want to learn more, as well as some career advice on landing a job in "big data".