HDInsight Deep Dive: Storm, HBase, and Hive from Pluralsight

HDInsight is Microsoft's managed Big Data stack in the cloud. With Azure you can provision clusters running Storm, HBase, and Hive which can process thousands of events per second, store petabytes of data, and give you a SQL-like interface to query it all. In this course, we'll build out a full solution using the stack and take a deep dive into each of the technologies.

Storm is a distributed compute platform which you can plug into Azure Event Hubs and use to power event stream processing. You can scale Storm to read tens of thousands of events per second and build a reliable workflow so that every event is guaranteed to be processed. HBase is a No-SQL database which is easy to get started with and can store tables with billions of rows and millions of columns. It's for real-time data access and it has a REST interface so you can read and write HBase data from a .NET Storm app. Hive is a data warehouse that provides a SQL-like interface over Big Data - HBase tables, and other sources. With Hive you can join across multiple sources and run queries from PowerShell and .NET. In this course, we use all three technologies running on Microsoft Azure to build a race timing solution and dive into performance tuning, reliability, and administration.

What's inside

Syllabus

Architecting a Solution with HDInsight

Storing Race Data in HBase

HBase Deep Dive

Processing Timing Events with Storm

Storm Deep Dive

Querying Race Data with Hive

Hive Deep Dive

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Taught by Elton Stoneman who are recognized for their work in Microsoft Azure

Provides a comprehensive study of HDInsight

Teaches Storm, HBase, and Hive

Takes a creative approach to an otherwise established topic, field, or subject

Provides real-time access to HBase tables through REST interface

Requires proficiency with PowerShell and .NET

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in HDInsight Deep Dive: Storm, HBase, and Hive with these activities:

Hadoop: The Definitive Guide

Show steps

This book provides a comprehensive overview of the Hadoop ecosystem, including HDInsight, and will deepen your understanding of the underlying concepts and technologies.

View Hadoop: The Definitive Guide: Storage and... on Amazon

Show steps

Review chapters related to HDInsight, Storm, HBase, and Hive.
Highlight and annotate key concepts.
Summarize key points and relate them to the course materials.

Review Course Notes

Show steps

This activity will help you review the concepts covered in the course, solidify your understanding, and improve your chances of success.

Browse courses on HDInsight

Show steps

Gather your course materials, including lecture notes, assignments, and quizzes.
Organize your materials into a logical order.
Review the materials, paying special attention to key concepts and ideas.

Glossary of Key Terms

Show steps

This activity will help you reinforce your understanding of important concepts by creating a comprehensive glossary.

Show steps

Identify key terms and definitions from the course materials.
Create a list of terms and their definitions.
Organize the terms into categories or alphabetical order.

Five other activities

Expand to see all activities and additional details

Show all eight activities

HBase Database Management

Show steps

This activity will supplement your understanding of HBase by providing practical experience in managing HBase databases.

Browse courses on HBase

Show steps

Follow tutorials on HBase database management.
Create and manage HBase tables, including schema design, data insertion, and querying.
Configure and optimize HBase performance.

Follow Azure HDInsight Tutorials

Show steps

This activity will help you gain hands-on experience with the technologies covered in the course and reinforce your understanding.

Browse courses on HDInsight

Show steps

Identify Azure HDInsight tutorials that align with the course topics.
Complete the tutorials, following the instructions carefully.
Document your progress and any challenges you encounter.

Join HDInsight Study Group

Show steps

This activity will provide you with an opportunity to connect with other students, share knowledge, and learn from each other's experiences.

Browse courses on HDInsight

Show steps

Find or create an HDInsight study group on platforms like Discord or Slack.
Participate in regular discussions, ask questions, and share insights.
Collaborate on projects or assignments.

Storm Event Processing Practice

Show steps

This activity will help you solidify your understanding of Storm by working through real-world scenarios and practicing event processing techniques.

Browse courses on Storm

Show steps

Find or create datasets of events.
Create Storm topologies to process the events.
Monitor the topologies and analyze the results.

Create a Real-Time Data Analytics Solution

Show steps

This activity will challenge you to apply the concepts and technologies learned in the course to a real-world problem, fostering deeper understanding and practical skills.

Show steps

Identify a business problem that involves real-time data analytics.
Design a solution using HDInsight technologies such as Storm, HBase, and Hive.
Implement and test your solution.
Present your solution and findings.

Career center

Learners who complete HDInsight Deep Dive: Storm, HBase, and Hive will develop knowledge and skills that may be useful to these careers:

Big Data Engineer

Big Data Engineers build and maintain the infrastructure for processing large datasets. This course may be useful for understanding the components of a big data solution implemented on the Microsoft Azure cloud platform. It covers the details of using HBase for data storage, Storm for distributed compute, and Hive for data warehousing and querying.

See salaries and explore the career path for Big Data Engineer

Big Data Analyst

Big Data Analysts analyze large datasets to uncover insights and trends. This course may be useful for getting started with querying large datasets. It covers using Hive, a data warehousing solution, to perform queries over HBase tables and other data sources. This course may be particularly interesting to analysts interested in using Microsoft Azure's cloud computing platform.

See salaries and explore the career path for Big Data Analyst

Data Engineer

Data Engineers design and maintain an organization's data infrastructure. They build out and manage data pipelines to ensure data is flowing consistently and reliably. Data Engineers may also help develop data science models and analytics solutions. This course dives into implementing an end-to-end data solution using HBase, Storm, and Hive on Microsoft Azure. It covers the concepts of data storage, processing, and querying with these big data technologies, and how the full stack can be applied to solve real-world problems.

See salaries and explore the career path for Data Engineer

Cloud Architect

Cloud Architects design and manage the cloud infrastructure of an organization. This course may be useful for gaining an understanding of the Microsoft Azure cloud platform for implementing a big data solution. It covers considerations and best practices for using HBase, Storm, and Hive for storing, processing, and querying large datasets.

See salaries and explore the career path for Cloud Architect

Data Lake Engineer

Data Lake Engineers leverage cloud computing technology to build and manage petabyte-scale repositories of unstructured data. This course may be useful for understanding data storage and processing options for data lakes. It covers specific technologies like HBase for data storage and Storm for distributed compute, and how to interact with these technologies using SQL-like Hive queries.

See salaries and explore the career path for Data Lake Engineer

Technical Architect

Technical Architects design and manage an organization's IT infrastructure. They work with business leaders to understand their needs and develop a technology strategy that supports those needs. Technical Architects may also oversee the implementation and management of big data solutions. This course may be useful for understanding how to implement a big data solution on Microsoft Azure. It covers the concepts of data storage, processing, and querying with HBase, Storm, and Hive, and how these technologies can be used to build a scalable and reliable data infrastructure.

See salaries and explore the career path for Technical Architect

Software Developer

Software Developers design, develop, and maintain software applications. They work with business leaders to understand their needs and develop software solutions that meet those needs. Software Developers may also work with data engineers and other IT professionals to implement big data solutions. This course may be useful for understanding how to develop software applications that use big data technologies. It covers the concepts of data storage, processing, and querying with HBase, Storm, and Hive, and how these technologies can be used to build scalable and reliable software applications.

See salaries and explore the career path for Software Developer

Database Administrator

Database Administrators ensure that an organization's databases are running smoothly and efficiently. They configure, maintain, and optimize databases to ensure that data is stored and retrieved reliably and quickly. Database Administrators may also work with data engineers and other IT professionals to implement big data solutions. This course may be useful for understanding how to implement and manage a big data solution on Microsoft Azure. It covers the concepts of data storage, processing, and querying with HBase, Storm, and Hive, and how these technologies can be used to build a scalable and reliable data infrastructure.

See salaries and explore the career path for Database Administrator

Systems Administrator

Systems Administrators manage and maintain an organization's IT infrastructure. They install, configure, and maintain servers, networks, and other IT equipment. Systems Administrators may also work with data engineers and other IT professionals to implement big data solutions. This course may be useful for understanding how to manage and maintain a big data solution on Microsoft Azure. It covers the concepts of data storage, processing, and querying with HBase, Storm, and Hive, and how these technologies can be used to build a scalable and reliable data infrastructure.

See salaries and explore the career path for Systems Administrator

Enterprise Architect

Enterprise Architects design and manage an organization's IT infrastructure. They work with business leaders to understand their needs and develop a technology strategy that supports those needs. Enterprise Architects may also oversee the implementation and management of big data solutions. This course may be useful for understanding how to implement a big data solution on Microsoft Azure. It covers the concepts of data storage, processing, and querying with HBase, Storm, and Hive, and how these technologies can be used to build a scalable and reliable data infrastructure.

See salaries and explore the career path for Enterprise Architect

Data Scientist

Data Scientists combine their skills in computer science, statistics, and analytics to solve business problems with data. They work closely with Data Engineers to develop machine learning infrastructure and models, and they use big data technologies to perform data analysis. This course may be useful for understanding how to implement big data solutions on Microsoft Azure with HBase, Storm, and Hive. It goes into detail about how to ingest, process, and query data with these technologies, and how to build an end-to-end data solution.

See salaries and explore the career path for Data Scientist

Data Architect

Data Architects ensure that organizations are creating a data strategy, usually based on big data, that supports business objectives. They help build out and maintain the big data technology stack, and may also develop data science and analytics solutions. This course may be useful for understanding data storage and processing solutions on Microsoft Azure. In particular, the course dives into implementing HBase, Storm, and Hive solutions, all of which are technologies commonly used by Data Architects.

See salaries and explore the career path for Data Architect

Information Security Analyst

Information Security Analysts protect an organization's data and IT systems from security threats. They develop and implement security policies and procedures, and they monitor systems for suspicious activity. Information Security Analysts may also work with data engineers and other IT professionals to implement big data solutions. This course may be useful for understanding how to secure a big data solution on Microsoft Azure. It covers the concepts of data storage, processing, and querying with HBase, Storm, and Hive, and how these technologies can be used to build a secure and compliant data infrastructure.

See salaries and explore the career path for Information Security Analyst