Learn By Example : Apache Storm from Udemy

Storm is to real-time stream processing what Hadoop is to batch processing. Using Storm you can build applications which need you to be highly responsive to the latest data and react within seconds and minutes, such as finding the latest trending topics on twitter, or monitoring spikes in payment gateway failures. From simple data transformations to applying machine learning algorithms on the fly, Storm can do it all.

This course has 25 Solved Examples on building Storm Applications.

What's covered?

1) Understanding Spouts and Bolts which are the building blocks of every Storm topology.

2) Running a Storm topology in the local mode and in the remote mode

3) Parallelizing data processing within a topology using different grouping strategies : Shuffle grouping, fields grouping, Direct grouping, All grouping, Custom Grouping

4) Managing reliability and fault-tolerance within Spouts and Bolts

5) Performing complex transformations on the fly using the Trident topology : Map, Filter, Windowing and Partitioning operations

6) Applying ML algorithms on the fly using libraries like Trident-ML and Storm-R.

What's inside

Learning objectives

Build a storm topology for processing data
Manage reliability and fault tolerance of the topology
Control parallelism using different grouping strategies

Perform complex transformations using trident
Apply machine learning algorithms on the fly in storm applications

Build a storm topology for processing data
Manage reliability and fault tolerance of the topology
Control parallelism using different grouping strategies
Perform complex transformations using trident
Apply machine learning algorithms on the fly in storm applications

Syllabus

Start Here

We'll start with an introduction, what the course covers and who it benefits.

Stream Processing with Storm

Understand how stream processing is different from batch processing

Stream processing is great for certain applications, but performance can be an issue at large scale. How do we solve this?

Understand Spouts and Bolts which make up a Storm topology

Understand how a Storm topology allows parellelism across components

A Storm topology runs on a cluster. Understand the different services which run on the cluster

Implementing a Hello World Topology

A Simple Hello World Topology

Ex 1: Implementing a Spout

Ex 1: Implementing a Bolt

Ex 1: Submitting the Topology

Processing Data using Files

Ex 2: Reading Data from a File

Representing Data using Tuples

Ex 3: Accessing data from Tuples

Ex 4: Writing Data to a File

Running a Topology in the Remote Mode

Setting up a Storm Cluster

Ex 5: Submitting a topology to the Storm Cluster

Adding Parallelism to a Storm Topology

Ex 6 : Shuffle Grouping

Ex 7: Fields Grouping

Ex 8: All Grouping

Ex 9: Custom Grouping

Ex 10: Direct Grouping

Section 7: Building a Word Count Topology

Ex 11: Building a Word Count Topology

Remote Procedure Calls Using Storm

Ex 12: A Storm Topology for DRPC calls

Managing Reliability of Topologies

Ex 13: Managing Failures in Spouts

Integrating Storm with Different Sources/Sinks

Ex 14: Implementing a Twitter Spout

Ex 15: Using a HDFS Bolt

Using the Storm Multilang Protocol

Ex 16: Building a Storm Topology using Python

Complex Transformations using Trident

Ex 17: Building a basic Trident Topology

Ex 18: Implementing a Map Function

Ex 19: Implementing a Filter Function

Ex 20: Aggregating data

Ex 21: Understanding States

Ex 22: Windowing operations

Ex 23: Joining data streams

Ex 24: Building a Twitter Hashtag Extractor

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Suitable for learners who are interested in developing a skill in data stream processing, as this is one of the core tenants of this course

Taught by Loony Corn, an enterprise known for its expertise in big data, analytics, and data engineering

Develops skills in Java, which is one of the most popular programming languages in the industry of data science and applied engineering

Covers machine learning algorithms, which are a foundational technology in today's data economy

In-demand skills taught in this course, such as Storm, are used by companies like Uber, Yahoo, Salesforce, and Twitter

This course does not teach a newer version of Storm than 2.0, which is the current version

Reviews summary

Apache storm examples

According to students, Apache Storm examples are useful, but the narrator may be difficult to follow. Students say the examples are too basic and mostly consist of “Hello World” use cases.

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Learn By Example : Apache Storm with these activities:

Practice Parallel Processing

Show steps

Parallelizing data processing is an essential part of building scalable Storm applications. This activity reinforces your understanding and enhances your skills in using different grouping strategies.

Browse courses on Parallel Processing

Show steps

Implement a Storm topology with different grouping strategies
Experiment with shuffle grouping, field grouping, and direct grouping
Measure the performance and compare the results

Join a Study Group for Storm Topology Development

Show steps

Collaborating with peers will provide you with diverse perspectives and enhance your understanding of topology design and optimization.

Show steps

Find or create a study group with other students taking the course.
Meet regularly to discuss topology design, implementation, and troubleshooting.
Share knowledge, ideas, and resources within the group.

Explore Trident-ML and Storm-R

Show steps

Applying machine learning algorithms on the fly is a powerful use case of Storm. This activity familiarizes you with popular libraries like Trident-ML and Storm-R, enabling you to incorporate ML into your Storm applications.

Show steps

Read tutorials and documentation on Trident-ML and Storm-R
Implement a Storm topology that integrates machine learning algorithms
Evaluate the performance and accuracy of your machine learning model

Five other activities

Expand to see all activities and additional details

Show all eight activities

Explore Advanced Storm Features Using Apache Tridents

Show steps

Expanding your knowledge into advanced features like Tridents will empower you to handle complex data transformations and stateful operations.

Browse courses on Storm

Show steps

Follow online tutorials or documentation on Apache Tridents.
Build a Storm topology using Tridents for windowing, aggregations, and state management.
Experiment with different Tridents operations and configurations.

Build a Twitter Hashtag Extractor

Show steps

Real-time processing of social media data is a perfect application for Storm. In this activity, you'll develop a complete Storm topology that extracts hashtags from Twitter data and gain hands-on experience in building real-world applications.

Show steps

Test and deploy your topology
Create a Storm topology for real-time hashtag extraction
Implement Twitter Spout and HDFS Bolt

Build a Storm Topology for Real-Time Fraud Detection

Show steps

Building a real-world application will help you apply the concepts you've learned in the course and gain practical experience in fraud detection.

Browse courses on Fraud Detection

Show steps

Design the topology architecture.
Implement spouts and bolts for data ingestion, processing, and alerting.
Configure and deploy the topology on a Storm cluster.
Test and evaluate the topology's performance.

Mentor a Junior Student in Storm Development

Show steps

Mentoring others will not only reinforce your own knowledge but also foster a sense of community and support within the learning environment.

Show steps

Identify a junior student who could benefit from your guidance.
Offer to provide mentorship and support on Storm development.
Regularly meet to discuss concepts, troubleshoot issues, and provide feedback.
Encourage the student to ask questions and actively participate in the learning process.

Contribute to the Storm Open-Source Community

Show steps

Engaging with the Storm community will provide you with valuable insights, networking opportunities, and the chance to contribute to the project.

Show steps

Join the Storm community forums or mailing lists.
Review open issues and discussions.
Identify areas where you can contribute, such as bug fixes or feature enhancements.
Submit pull requests or participate in discussions to share your contributions.

Career center

Learners who complete Learn By Example : Apache Storm will develop knowledge and skills that may be useful to these careers:

Data Engineer

Data Engineers design, build, and maintain data pipelines. They work with data from a variety of sources, including databases, sensors, and web logs. Data Engineers use their knowledge of data engineering tools and techniques to create data pipelines that can process data efficiently and reliably. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing pipelines.

See salaries and explore the career path for Data Engineer

Software Engineer

Software Engineers design, develop, and test software applications and systems. They work on a variety of projects, from small personal apps to large enterprise systems. Software Engineers use their knowledge of programming languages, data structures, and algorithms to create software that meets the needs of users. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications.

See salaries and explore the career path for Software Engineer

Big Data Engineer

Big Data Engineers design, build, and maintain big data systems. They work with large datasets that are too large to be processed by traditional data processing tools. Big Data Engineers use their knowledge of big data technologies to create systems that can process data efficiently and reliably. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time big data processing systems.

See salaries and explore the career path for Big Data Engineer

Machine Learning Engineer

Machine Learning Engineers design, build, and deploy machine learning models. They work on a variety of projects, from developing self-driving cars to predicting customer behavior. Machine Learning Engineers use their knowledge of machine learning algorithms and techniques to create models that can learn from data and make predictions. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time machine learning applications.

See salaries and explore the career path for Machine Learning Engineer

Data Scientist

Data Scientists use their knowledge of statistics, machine learning, and data mining to extract insights from data. They work on a variety of projects, from predicting customer churn to identifying fraudulent transactions. Data Scientists use their insights to help businesses make better decisions. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications that can be used to power data science projects.

See salaries and explore the career path for Data Scientist

Data Analyst

Data Analysts use their knowledge of statistics and data analysis techniques to extract insights from data. They work on a variety of projects, from identifying trends to forecasting demand. Data Analysts use their insights to help businesses make better decisions. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications that can be used to power data analysis projects.

See salaries and explore the career path for Data Analyst

Cloud Architect

Cloud Architects design, build, and manage cloud computing systems. They work with a variety of cloud computing technologies, including Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Cloud Architects use their knowledge of cloud computing to create systems that are scalable, reliable, and secure. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications that can be deployed to the cloud.

See salaries and explore the career path for Cloud Architect

DevOps Engineer

DevOps Engineers work to bridge the gap between development and operations teams. They work on a variety of projects, from automating software deployments to monitoring production systems. DevOps Engineers use their knowledge of development and operations tools and techniques to create systems that are efficient and reliable. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing pipelines that can be integrated with DevOps systems.

See salaries and explore the career path for DevOps Engineer

Business Analyst

Business Analysts work with businesses to understand their needs and develop solutions to their problems. They use their knowledge of business analysis techniques to gather and analyze data, and to create reports and recommendations. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications that can be used to power business analysis projects.

See salaries and explore the career path for Business Analyst

Software Architect

Software Architects design and build the architecture of software systems. They work on a variety of projects, from designing enterprise systems to developing mobile apps. Software Architects use their knowledge of software architecture principles and patterns to create systems that are scalable, reliable, and secure. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications.

See salaries and explore the career path for Software Architect

Technical Writer

Technical Writers create documentation for software and hardware products. They work on a variety of projects, from writing user manuals to creating training materials. Technical Writers use their knowledge of technical writing principles and techniques to create documentation that is clear, concise, and accurate. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications that can be used to power technical writing projects.

See salaries and explore the career path for Technical Writer

Product Manager

Product Managers work with teams to develop and launch new products. They work on a variety of projects, from defining product requirements to managing product launches. Product Managers use their knowledge of product management principles and techniques to create products that meet the needs of users. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications that can be used to power product development projects.

See salaries and explore the career path for Product Manager

Network Engineer

Network Engineers design and build computer networks. They work on a variety of projects, from designing enterprise networks to developing wireless networks. Network Engineers use their knowledge of networking principles and technologies to create networks that are reliable, secure, and scalable. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications that can be deployed to networks.

See salaries and explore the career path for Network Engineer

Systems Engineer

Systems Engineers design and build computer systems. They work on a variety of projects, from designing enterprise systems to developing embedded systems. Systems Engineers use their knowledge of computer science and engineering principles to create systems that are efficient, reliable, and secure. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing systems.

See salaries and explore the career path for Systems Engineer

Database Administrator

Database Administrators manage and maintain databases. They work on a variety of projects, from designing databases to optimizing database performance. Database Administrators use their knowledge of database management systems to create and maintain databases that are reliable, secure, and scalable. This course can help you build a foundation in Apache Storm, a popular open-source stream processing framework, which can be useful for building real-time data processing applications that can be integrated with databases.

See salaries and explore the career path for Database Administrator