Big Data, Hadoop, and Spark Basics from edX

Organizations need skilled, forward-thinking Big Data practitioners who can apply their business and technical skills to unstructured data such as tweets, posts, pictures, audio files, videos, sensor data, and satellite imagery, and more, to identify behaviors and preferences of prospects, clients, competitors, and others. ****

This course introduces you to Big Data concepts and practices. You will understand the characteristics, features, benefits, limitations of Big Data and explore some of the Big Data processing tools. You'll explore how Hadoop, Hive, and Spark can help organizations overcome Big Data challenges and reap the rewards of its acquisition.

Hadoop, an open-source framework, enables distributed processing of large data sets across clusters of computers using simple programming models. Each computer, or node, offers local computation and storage, allowing datasets to be processed faster and more efficiently. Hive, a data warehouse software, provides an SQL-like interface to efficiently query and manipulate large data sets in various databases and file systems that integrate with Hadoop.

Open-source Apache Spark is a processing engine built around speed, ease of use, and analytics that provides users with newer ways to store and use big data.

You will discover how to leverage Spark to deliver reliable insights. The course provides an overview of the platform, going into the different components that make up Apache Spark. In this course, you will also learn how Resilient Distributed Datasets, known as RDDs, enable parallel processing across the nodes of a Spark cluster.

You'll gain practical skills when you learn how to analyze data in Spark using PySpark and Spark SQL and how to create a streaming analytics application using Spark Streaming, and more.

What's inside

Learning objectives

Describe big data, its impact, processing methods and tools, and use cases.
Describe spark programming basics, including parallel programming basics, for dataframes, data sets, and sparksql.
Apply apache spark development and runtime environment options.

"after completing this course, a learner will be able to..."
Describe hadoop architecture, ecosystem, practices, and applications, including distributed file system (hdfs), hbase, spark, and mapreduce.
Describe how spark uses rdds, creates data sets, and uses catalyst and tungsten to optimize sparksql.

Describe big data, its impact, processing methods and tools, and use cases.
Describe spark programming basics, including parallel programming basics, for dataframes, data sets, and sparksql.
Apply apache spark development and runtime environment options.
"after completing this course, a learner will be able to..."
Describe hadoop architecture, ecosystem, practices, and applications, including distributed file system (hdfs), hbase, spark, and mapreduce.
Describe how spark uses rdds, creates data sets, and uses catalyst and tungsten to optimize sparksql.

Syllabus

Module 1 – What is Big Data?

___Introduction to Big Data_ *

o What is Big Data?

o Impact of Big Data

o Parallel Processing, Scaling, and Data Parallelism

o Tools of Big Data

o Beyond the Hype

o Big Data Use Cases

o Viewpoints about Big Data

Module 2 – Introduction to the Hadoop Ecosystem

___Introduction to the Hadoop Ecosystem_ *

o What is Hadoop

o An introduction to MapReduce

o The Hadoop Ecosystem/Common components: Introducing HDFS, Hive, HBase, and Spark, other modules

o Working with HDFS

o Working with HBase

o Lab: MapReduce

Module 3 – Introduction to Apache Spark

___Introduction to Apache Spark_ *

o Why use Apache Spark?

o Functional Programming Basics

o Parallel Programming using Resilient Distributed Datasets

o Scale-out / Data Parallelism in Apache Spark

o DataFrames and SparkSQL

o Lab: Practical examples with PySpark

Module 4 – DataFrames and SparkSQL

___DataFrames and SparkSQL_ *

o Introduction to Data-Frames & SparkSQL

o RDDs in Parallel Programming and Spark

o Data-frames and Datasets

o Catalyst and Tungsten

o ETL with Data-frames

o Lab: ETL with Data-frames

o Real-world usage of SparkSQL

o Lab: SparkSQL

Module 5 – Development and Runtime Environment options

___Development and Runtime Environment options_ *

o Apache Spark architecture

o Overview of Apache Spark Cluster Modes

o How to Run an Apache Spark Application

o Using Apache Spark on IBM Cloud

o Lab: Scale-out on IBM Spark Environment in Watson Studio

o Setting Apache Spark Configuration

o Running Spark on Kubernetes

o Lab: Spark on Kube

Module 6 – Monitoring & Tuning

___Monitoring and tuning Apache Spark_ *

o The Apache Spark User Interface

o Monitoring Jobs

o Debugging of parallel jobs

o Understanding Memory resources

o Understanding Processor resources

o Lab: Monitoring and Performance tuning

Module 7 – Final Quiz ****

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Suitable for beginners with interests in data science, data engineering, or big data analysis

Introduces learners to the fundamentals of big data concepts and tools

Provides hands-on practice with Spark, PySpark, and Spark SQL

Teaches data engineering concepts such as parallel processing, scalability, and distributed file systems

Covers Apache Spark, a widely used big data analytics platform

Assumes prior programming experience

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Big Data, Hadoop, and Spark Basics with these activities:

Review course materials

Show steps

Reviewing the course materials will help you become familiar with the key concepts and topics covered in the course.

Show steps

Read the course syllabus
Review the course schedule
Download and review the course readings

Compile and review course resources

Show steps

Improve knowledge recall by organizing and reviewing course materials, notes, homework, and assignments.

Show steps

Gather and download all the course materials
Create a filing and storage system
Label each document
Place the files in their respective folders or storage locations
Set reminders to review the materials periodically

Practice coding exercises

Show steps

Completing coding exercises will help you develop your skills in using Hadoop, Hive, and Spark.

Browse courses on Hadoop

Show steps

Solve coding exercises from the course materials
Complete coding challenges online

Five other activities

Expand to see all activities and additional details

Show all eight activities

Attend a Big Data conference

Show steps

This will give you the opportunity to connect with experts in the field of Big Data.

Browse courses on Big Data

Show steps

Identify a Big Data conference that you would like to attend
Register for the conference
Attend the conference and participate in the sessions

Build a data analysis project

Show steps

This will help you apply your knowledge of Big Data and Spark to a real-world problem.

Browse courses on Big Data

Show steps

Identify a problem that you can solve using Big Data and Spark
Collect and clean the data
Analyze the data using Spark
Visualize the results

Start a data science project

Show steps

This will allow you to apply your knowledge of Big Data and Spark to a real-world problem.

Browse courses on Big Data

Show steps

Identify a data science project that you would like to work on
Gather the data for your project
Clean the data
Analyze the data
Visualize the results

Write a blog post about Big Data

Show steps

This will help you to consolidate your knowledge of Big Data and improve your communication skills.

Browse courses on Big Data

Show steps

Choose a topic that you would like to write about
Research the topic
Write the blog post
Publish the blog post

Mentor junior data scientists

Show steps

This will help you to reinforce your knowledge of Big Data and develop your leadership skills.

Browse courses on Big Data

Show steps

Identify a junior data scientist who you can mentor
Set up a regular meeting time
Provide guidance and support to the junior data scientist
Help the junior data scientist to develop their skills

Career center

Learners who complete Big Data, Hadoop, and Spark Basics will develop knowledge and skills that may be useful to these careers:

Hadoop Developer

Hadoop Developers specialize in designing, developing, and maintaining Hadoop-based data processing systems. This course is highly relevant to this role, as it provides a thorough introduction to Hadoop, including its architecture, components, and applications. By completing this course, you'll gain proficiency in working with Hadoop, a valuable asset for Hadoop Developers.

See salaries and explore the career path for Hadoop Developer

Data Scientist

As a Data Scientist, you'll be involved in extracting, analyzing, and interpreting large datasets to uncover valuable insights for businesses. This course can help you develop the necessary skills, as it covers the fundamentals of Big Data, Hadoop, and Spark, technologies heavily used by Data Scientists. Moreover, the course delves into data processing, analysis, and optimization, providing you with a solid foundation for a successful career in Data Science.

See salaries and explore the career path for Data Scientist

Data Analyst

Data Analysts collect, analyze, and interpret data to provide insights that drive decision-making within organizations. This course can be a valuable asset, as it provides a comprehensive overview of Big Data technologies and techniques. By becoming proficient in Hadoop, Spark, and other tools, you'll be well-equipped to handle large and complex datasets, a critical skill for Data Analysts.

See salaries and explore the career path for Data Analyst

Big Data Analyst

Big Data Analysts are responsible for analyzing large datasets to identify trends, patterns, and insights that can inform business decisions. This course aligns well with this role, as it provides a comprehensive understanding of Big Data concepts and technologies, including Hadoop and Spark. By mastering these tools and techniques, you'll be well-equipped to extract meaningful insights from complex data, a crucial skill for Big Data Analysts.

See salaries and explore the career path for Big Data Analyst

Data Architect

The job of a Data Architect involves conceptualizing, designing, and creating data solutions, utilizing large datasets to enhance organizations' efficiency. This course can help you prepare for this role by providing a solid grounding in Big Data concepts, Hadoop, and Spark, which are essential tools for Data Architects. Moreover, the course covers topics like data processing, analysis, and optimization, empowering you with the skills to design and implement effective data management solutions.

See salaries and explore the career path for Data Architect

Data Engineer

Data Engineers are responsible for designing, constructing, maintaining, and improving the infrastructure that stores and processes data. If you aspire to become one, this course will be invaluable as it provides a comprehensive overview of Big Data technologies such as Hadoop and Spark. You'll also learn about data processing, analysis, and optimization, key skills for Data Engineers. By completing this course, you'll gain a competitive edge in the job market.

See salaries and explore the career path for Data Engineer

IT Architect

IT Architects design, implement, and maintain the technology infrastructure within organizations. This course can enhance your ability to fulfill this role by providing a solid understanding of Big Data concepts, including Hadoop and Spark. These technologies are revolutionizing enterprise IT architectures, and by mastering them, you'll gain a competitive edge in the job market.

See salaries and explore the career path for IT Architect

Big Data Consultant

Big Data Consultants advise organizations on how to leverage Big Data to achieve their business goals. This course can be useful for this role as it provides a comprehensive understanding of Big Data concepts, including Hadoop and Spark, which are key technologies for Big Data initiatives. By mastering these tools and techniques, you'll be well-equipped to provide valuable insights and guidance to organizations.

See salaries and explore the career path for Big Data Consultant

Database Administrator

Database Administrators are responsible for managing and maintaining database systems. This course can be beneficial in this role, as it covers the fundamentals of Big Data technologies, including Hadoop and Spark. By gaining proficiency in these tools, you'll be well-equipped to manage and maintain Big Data systems, a valuable skill for Database Administrators.

See salaries and explore the career path for Database Administrator

Software Engineer

In their role, Software Engineers design, develop, and maintain software applications. This course can be beneficial to those aspiring to become Software Engineers as it provides a foundation in Big Data technologies like Hadoop and Spark, which are increasingly used for data-intensive applications. Moreover, the course covers data processing, analysis, and optimization techniques, equipping you with valuable skills for a career in software engineering.

See salaries and explore the career path for Software Engineer

Cloud Engineer

Cloud Engineers design, build, and maintain cloud-based infrastructure and applications. This course can be beneficial as it provides a foundation in Big Data technologies such as Hadoop and Spark, which are increasingly deployed in cloud environments. By mastering these tools and techniques, you'll gain valuable skills for a career in Cloud Engineering.

See salaries and explore the career path for Cloud Engineer

Data Warehouse Architect

Data Warehouse Architects design, build, and maintain data warehouses, which are critical for storing and managing large amounts of data. This course can be useful as it provides a foundation in Big Data technologies such as Hadoop and Spark, which are increasingly used in data warehousing. By mastering these tools and techniques, you'll gain valuable skills for a career in Data Warehouse Architecture.

See salaries and explore the career path for Data Warehouse Architect

Machine Learning Engineer

Machine Learning Engineers design, develop, and deploy machine learning models. This course can be useful for this role as it provides a foundation in Big Data technologies such as Hadoop and Spark, which are increasingly used for training and deploying machine learning models. By mastering these tools and techniques, you'll gain valuable skills for a career in Machine Learning Engineering.

See salaries and explore the career path for Machine Learning Engineer

Data Integration Architect

Data Integration Architects design, develop, and implement data integration solutions. This course can be useful for this role as it provides a comprehensive understanding of Big Data concepts, including Hadoop and Spark, which are key technologies for data integration. By gaining proficiency in these tools, you'll be well-equipped to design and implement effective data integration solutions.

See salaries and explore the career path for Data Integration Architect

Research Scientist

Research Scientists conduct research and develop new technologies. This course can be useful as it provides a foundation in Big Data concepts, including Hadoop and Spark, which are increasingly used in scientific research. By mastering these tools and techniques, you'll gain valuable skills for a career in research.

See salaries and explore the career path for Research Scientist