Master Big Data - Apache Spark/Hadoop/Sqoop/Hive/Flume/Mongo from Udemy

In this course, you will start by learning what is hadoop distributed file system and most common hadoop commands required to work with Hadoop File system.

Then you will be introduced to Sqoop Import

Understand lifecycle of sqoop command.
Use sqoop import command to migrate data from Mysql to HDFS.
Use sqoop import command to migrate data from Mysql to Hive.
Use various file formats, compressions, file delimeter,where clause and queries while importing the data.
Understand split-by and boundary queries.
Use incremental mode to migrate the data from Mysql to HDFS.

Further, you will learn Sqoop Export to migrate data.

What is sqoop export
Using sqoop export, migrate data from HDFS to Mysql.
Using sqoop export, migrate data from Hive to Mysql.

Further, you will learn about Apache Flume

Understand Flume Architecture.
Using flume, Ingest data from Twitter and save to HDFS.
Using flume, Ingest data from netcat and save to HDFS.
Using flume, Ingest data from exec and show on console.
Describe flume interceptors and see examples of using interceptors.
Flume multiple agents
Flume Consolidation.

In the next section, we will learn about Apache Hive

Hive Intro
External & Managed Tables
Working with Different Files - Parquet,Avro
Compressions
Hive Analysis
Hive String Functions
Hive Date Functions
Partitioning
Bucketing

You will learn about Apache Spark

Spark Intro
Cluster Overview
RDD
DAG/Stages/Tasks
Actions & Transformations
Transformation & Action Examples
Spark Data frames
Spark Data frames - working with diff File Formats & Compression
Dataframes API's
Spark SQL
Dataframe Examples
Spark with Cassandra Integration
Running Spark on Intellij IDE
Running Spark on EMR

What's inside

Learning objective

Hadoop distributed file system and commands. lifecycle of sqoop command. sqoop import command to migrate data from mysql to hdfs. sqoop import command to migrate data from mysql to hive. working with various file formats, compressions, file delimeter,where clause and queries while importing the data. understand split-by and boundary queries. use incremental mode to migrate the data from mysql to hdfs. using sqoop export, migrate data from hdfs to mysql. using sqoop export, migrate data from hive to mysql. understand flume architecture. using flume, ingest data from twitter and save to hdfs. using flume, ingest data from netcat and save to hdfs. using flume, ingest data from exec and show on console. flume interceptors.

Syllabus

Hadoop distributed file system and Hadoop Commands

Meet your Instructor

Course Intro

Big Data Intro

Understanding Big Data Ecosystem

Google Cloud Cluster Setup

Google Cloud Account Setup

Dataproc Cluster Setup - Part1

DataProc Cluster Setup - Part2

Upload Files on Google Cloud

Sqoop Setup

Environment Update

Hadoop & Yarn

HDFS and Hadoop Commands

Yarn Cluster Overview

Sqoop lifecycle, migrate data from HDFS/Hive to Mysql, working with various file formats,compressions,file delimeters, conditional imports ,queries, split-by, boundary queries and incremental imports

Sqoop Introduction

Managing Target Directories

Working with Parquet File Format

Working with Avro File Format

Working with Different Compressions

Conditional Imports

Split-by and Boundary Queries

Field delimeters

Incremental Appends

Sqoop-Hive Cluster Fix

Access Hive on Google Cloud

Sqoop Hive Import

Sqoop List Tables/Database

Sqoop Assignment1

Sqoop Assignment2

Sqoop Import Practice1

Sqoop Import Practice2

Export data from Hdfs/Hive to Mysql

Export from Hdfs to Mysql

Export from Hive to Mysql

Export Avro Compressed to Mysql

Bonus Lecture: Sqoop with Airflow

Apache Flume Architecture, Working with Various SInks, Source and channels, Interceptors

Flume Setup

Flume Introduction & Architecture

Exec Source and Logger Sink

Moving data from Twitter to HDFS

Moving data from NetCat to HDFS

Flume Interceptors

Flume Interceptor Example

Flume Multi-Agent Flow

Flume Consolidation

This section will teach about doing big data analytics with apache hive

Access Hive Shell on Google Cloud

Hive Introduction

Hive Database

Hive Managed Tables

Hive External Tables

Hive Inserts

Hive Analytics

Working with Parquet

Compressing Parquet

Working with Fixed File Format

Alter Command

Hive String Functions

Hive Date Functions

Hive Partitioning

Hive Bucketing

Spark with Yarn & HDFS

What is Apache Spark

Understanding Cluster Manager (Yarn)

Understanding Distributed Storage (HDFS)

Running Spark on Yarn/HDFS

Understanding Deploy Modes

GCS Cluster

Spark on GCS Cluster

Upload Data files for Spark

Spark Internals

Drivers & Executors

RDDs & Dataframes

Transformation & Actions

Wide & Narrow Transformations

Understanding Execution Plan

Different Plans by Driver

Spark RDD : Transformation & Actions

Map/FlatMap Transformation

Filter/Intersection

Union/Distinct Transformation

GroupByKey/ Group people based on Birthday months

ReduceByKey / Total Number of students in each Subject

SortByKey / Sort students based on their rollno

MapPartition / MapPartitionWithIndex

Change number of Partitions

Join / join email address based on customer name

Spark Actions

Spark RDD Practice

Upload Files

Scala Tuples

Filter Error Logs

Frequency of word in Text File

Population of each city

Orders placed by Customers

average rating of movie

Spark Dataframes & Spark SQL

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Develops foundational skills for big data analytics, which are core skills for data engineering

Covers big data tools that are highly relevant to industry, such as Hadoop, Apache Flume, Apache Hive, and Apache Spark

Teaches foundational big data analytics concepts that are useful for personal growth and development

Covers a comprehensive range of data engineering components, including Hadoop, Apache Flume, Apache Hive, and Apache Spark

Instructors are not recognized for their work in the topic that this course teaches

Reviews summary

Big data tutorial for beginners

According to students, this beginner-friendly Big Data tutorial is well-organized, engaging, and covers a wide range of relevant tools such as Apache Spark, Hadoop, Sqoop, Hive, Flume, and Mongo. It's a great option for those looking to start their journey in Big Data.

Course content is well-paced and easy to follow.

"Lectures are straight to the point."

This course is especially good for beginners to Big Data.

"This course is very good for those who wants to learn sqoop for the first time."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Master Big Data - Apache Spark/Hadoop/Sqoop/Hive/Flume/Mongo with these activities:

Review Basic Data Structures and Algorithms

Show steps

Refresh your foundational understanding of data structures and algorithms to strengthen your problem-solving capabilities in Big Data.

Browse courses on Data Structures

Show steps

Review concepts such as arrays, linked lists, stacks, and queues.
Practice implementing basic algorithms such as sorting, searching, and recursion.

Review 'Hadoop: The Definitive Guide' by Tom White

Show steps

Gain a comprehensive understanding of Hadoop's architecture, components, and use cases by reviewing this authoritative book.

View Hadoop: The Definitive Guide: Storage and... on Amazon

Show steps

Read the book's introduction and overview of Hadoop.
Review the chapters on HDFS, Yarn, and MapReduce.
Read the chapters on advanced topics such as security, performance tuning, and Hadoop ecosystem tools.
Take notes and highlight key concepts.

Follow Tutorials on Apache Spark RDD

Show steps

Enhance your understanding of Apache Spark RDDs by following online tutorials and applying the concepts to practice problems.

Browse courses on Apache Spark

Show steps

Find online tutorials that cover Apache Spark RDD.
Follow the tutorials step-by-step and try out the examples.
Practice using RDD transformations and actions on your own datasets.
Join online forums or communities to discuss your progress and ask questions.

Four other activities

Expand to see all activities and additional details

Show all seven activities

Build a Data Ingestion Pipeline with Apache Flume

Show steps

Build a data ingestion pipeline using Apache Flume to gain hands-on experience in collecting and processing real-world data.

Show steps

Set up a Flume agent and configure data sources (e.g., Twitter, Netcat).
Create a data sink (e.g., HDFS) and configure the Flume agent to send data to the sink.
Write a Flume interceptor to pre-process or filter data before sending to the sink.
Monitor and troubleshoot the data ingestion pipeline to ensure smooth data flow.
Present the results of the data ingestion pipeline and discuss its potential applications.

Practice Apache Hive Functions

Show steps

Practice various Apache Hive functions to solidify understanding and reinforce skills in data analysis and manipulation.

Browse courses on Apache Hive

Show steps

Create a Hive session and load a dataset into a Hive table.
Practice using string functions such as UPPER(), LOWER(), SUBSTRING().
Practice using date functions such as DATE_FORMAT(), DATE_ADD(), DATE_SUB().
Practice using aggregation functions such as COUNT(), SUM(), AVG().
Practice using conditional functions such as CASE WHEN().

Volunteer at a Big Data Project

Show steps

Gain practical experience in working on a real-world Big Data project by volunteering with organizations that specialize in this field.

Browse courses on Big Data

Show steps

Research Big Data projects and identify potential organizations to volunteer with.
Contact the organization and express your interest in volunteering.
Participate in the project and contribute your skills.
Network with other volunteers and professionals in the field.

Develop a Presentation on Spark SQL

Show steps

Create a presentation that showcases your understanding of Spark SQL and its capabilities for data analysis.

Browse courses on Spark SQL

Show steps

Gather information on Spark SQL's features and use cases.
Design the presentation slides with clear and concise content.
Practice delivering the presentation and get feedback.
Present the presentation to an audience.

Career center

Learners who complete Master Big Data - Apache Spark/Hadoop/Sqoop/Hive/Flume/Mongo will develop knowledge and skills that may be useful to these careers:

Data Engineer

Data Engineers use Apache Spark to wrangle big data. This course will give a Data Engineer the skills to work with Apache Spark, which is increasingly-popular in big data management.

See salaries and explore the career path for Data Engineer

Operations Research Analyst

Operations Research Analysts may use Apache Spark to analyze big data to help improve operations and processes. This course teaches the basics of Apache Spark, which can help an Operations Research Analyst build a foundation in Apache Spark.

See salaries and explore the career path for Operations Research Analyst

Data Scientist

Data Scientists use Apache Spark to process large datasets. This course teaches the fundamentals of Apache Spark, allowing a Data Scientist to build on these skills and improve their capabilities with Apache Spark.

See salaries and explore the career path for Data Scientist

Risk Analyst

Risk Analysts may use Apache Spark to analyze large sets of data to identify and assess risk. This course teaches the basics of Apache Spark, which can help a Risk Analyst build a foundation in Apache Spark.

See salaries and explore the career path for Risk Analyst

Data Architect

Data Architects may use Apache Spark to design big data systems. This course teaches the basics of Apache Spark, which can help a Data Architect build a foundation for working with Apache Spark.

See salaries and explore the career path for Data Architect

Database Administrator

Database Administrators may use Apache Spark to assist with big data administration tasks. This course teaches the basics of Apache Spark, which can help a Database Administrator expand their big data skillset.

See salaries and explore the career path for Database Administrator

Software Engineer

Software Engineers may work with Apache Spark when working with big data in a development environment. This course teaches the basics and fundamentals of Apache Spark.

See salaries and explore the career path for Software Engineer

Data Analyst

A Data Analyst may use Apache Spark to work with big data. This course teaches the basics of Apache Spark. These skills can help a Data Analyst succeed, particularly as Apache Spark has become more popular in data analysis.

See salaries and explore the career path for Data Analyst

Financial Analyst

Financial Analysts may use Apache Spark to analyze large sets of financial data. This course teaches the fundamentals of Apache Spark, which can help a Financial Analyst get started with Apache Spark.

See salaries and explore the career path for Financial Analyst

Statistician

Statisticians may use Apache Spark to process large datasets for statistical analysis purposes. This course may be useful for Statisticians who wish to learn about Apache Spark and expand their big data skillset.

See salaries and explore the career path for Statistician

Quantitative Analyst

Quantitative Analysts may use Apache Spark to process big data for quantitative analysis purposes. This course may be useful for a Quantitative Analyst who wants to learn the basics of Apache Spark.

See salaries and explore the career path for Quantitative Analyst

Market Research Analyst

Market Research Analysts may utilize Apache Spark to process and analyze big data for market research purposes. This course may be useful for a Market Research Analyst who wants to learn the basics of Apache Spark.

See salaries and explore the career path for Market Research Analyst

Machine Learning Engineer

Machine Learning Engineers may use Apache Spark to process large amounts of data for model creation. While this course focuses on the fundamentals, it may be useful for a Machine Learning Engineer who wants to learn the basics of Apache Spark.

See salaries and explore the career path for Machine Learning Engineer

Actuary

Actuaries may use Apache Spark to analyze big data to assess risk. While this course is focused on the fundamentals of big data processing, it may be useful for Actuaries who wish to learn about Apache Spark.

See salaries and explore the career path for Actuary

Business Intelligence Analyst

Business Intelligence Analysts may use Apache Spark to analyze large sets of data. This course may be useful for Business Intelligence Analysts who wish to expand their big data skillset and work with Apache Spark.

See salaries and explore the career path for Business Intelligence Analyst