Apache Kafka Series - Learn Apache Kafka for Beginners v3 from Udemy

Welcome to the Apache Kafka Series. Join a community of 130,000+ students learning Kafka.

Apache Kafka has become the leading distributed data streaming enterprise big data technology. Kafka is used in production by over 33% of the Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart and LinkedIn.

To learn Kafka easily, step-by-step, you have come to the right place. No prior Kafka knowledge is required

If you look at the documentation, you can see that Apache Kafka is not easy to learn...

Thanks to my several years of experience in Kafka and Big Data, I wanted to make learning Kafka accessible to everyone.

We'll take a step-by-step approach to learn all the fundamentals of Apache Kafka. At the end of this course, you'll be productive and you'll know the following:

The Apache Kafka Ecosystem Architecture
The Kafka Core Concepts: Topics, Partitions, Brokers, Replicas, Producers, Consumers, and more.
Launch your own Kafka cluster in no time using native Kafka binaries – Windows / MacOS X / Linux
Learn and Practice using the Kafka Command Line Interface (CLI)
Code Producer and Consumers using the Java API
Real world project using Wikimedia as a source of data for a producer and OpenSearch as a sink for our consumer
Overview of Advanced APIs (Kafka Connect, Kafka Streams)
Real World Case Studies and Big Use cases
Overview of Advanced Kafka for Administrators
Advanced Topic Configurations

Note: The hands-on section is based on Java, which is the native Kafka programming language. But, good news. Your learning in Java will be completely applicable to other programming languages, such as Python, C#, Node.js or Scala, and Big Data frameworks such as Spark, NiFi or Akka

The Kafka for Beginners course is the best place to start your Kafka learning journey... Ask other students.

5 stars: "That's one of the most high-quality on-line courses I ever took" – Paul L.
5 stars: "This training was awesome and learned many things about KAFKA though having 0 years of experience in this technology" – Puneet G.
5 stars: "Outstanding on all fronts. I wish all courses were that well presented. I especially appreciate the hands-on sections. I was able to get everything up and running with ease. Highly recommend this course and this instructor. " – David G.

You can take this course risk-free and if you don't like it, you can get a refund anytime in the first 30 days.

Instructor

My name is Stéphane Maarek, I am passionate about Data Streaming & Cloud Computing, and I will be your instructor in this course. I teach about Apache Kafka, the Kafka ecosystem, and Kafka Certifications, focusing on helping my students improve their professional proficiencies. I am also the co-founder of Conduktor: an enterprise Apache Kafka platform & UI to help everyone use Kafka.

I have already taught

With Apache Kafka becoming much more than a buzzword out there, I've decided it's time for students to learn how to be Kafka professionals properly. So, let’s kick start the course. You are in good hands.

This Course Also Comes With:

Lifetime Access to All Future Updates
A responsive instructor in the Q&A Section
Links to interesting articles, and lots of good code to base your next applications onto
Udemy Certificate of Completion Ready for Download

This is the course that could improve your career.

Apache Kafka is a skill in high demand and there are not enough people to fulfill all the open positions. You can boost your income, take on new roles and fun challenges. Many of my students are now the Kafka experts of their companies. You can be the next.

I hope to see you inside the course.

Note: Looking for more advanced Kafka concepts? There are many volumes in the Apache Kafka Series:

Learn Kafka for Beginners v2 (great to start)
Kafka Connect Hands On Learning
Kafka Streams for Data Processing
KSQL on ksqlDB - Hands On.
Kafka Cluster Setup & Administration
Confluent Schema Registry & Kafka REST Proxy
Kafka Security (

What's inside

Learning objectives

Understand apache kafka ecosystem, architecture, core concepts and operations
Master concepts such as topics, partitions, brokers, producers, consumers
Start a personal kafka development environment
Learn major clis: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs

Create your producers and consumers in java to interact with kafka
Program a real world twitter producer & elasticsearch consumer
Extended apis overview (kafka connect, kafka streams), case studies and big data architecture
Practice and understand log compaction

Understand apache kafka ecosystem, architecture, core concepts and operations
Master concepts such as topics, partitions, brokers, producers, consumers
Start a personal kafka development environment
Learn major clis: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs
Create your producers and consumers in java to interact with kafka
Program a real world twitter producer & elasticsearch consumer
Extended apis overview (kafka connect, kafka streams), case studies and big data architecture
Practice and understand log compaction

Syllabus

Kafka Introduction

Course Introduction

Apache Kafka in 5 minutes

Course Objectives

Welcome! - About your instructor

Code Download

====== Kafka Fundamentals ======

Kafka Fundamentals

Kafka Theory

Topics, Partitions and Offsets

Producers and Message Keys

Consumers & Deserialization

Consumer Groups & Consumer Offsets

Brokers and Topics

Topic Replication

Producer Acknowledgements & Topic Durability

Zookeeper

Kafka KRaft - Removing Zookeeper

Theory Roundup

Quiz on Theory

Starting Kafka

Important: Starting Kafka & Lectures Order

FAQ for Setup Problems

Starting Kafka with Conduktor - Multi Platform

Mac OS X - Download and Setup Kafka in PATH

Mac OS X - Start Zookeeper and Kafka

Mac OS X - Using brew

Linux - Download and Setup Kafka in PATH

Linux - Start Zookeeper and Kafka

Windows WSL2 - Download Kafka and PATH Setup

Windows WSL2 - Start Zookeeper & Kafka

Windows WSL2 - How to Fix Problems

Windows WSL2 - Extra Instructions

Windows non-WSL2 - Start Zookeeper and Kafka

Starting Kafka without Zookeeper

Note: try out Kafka KRaft

Mac OS X - Start Kafka in KRaft mode

Linux - Start Kafka in KRaft mode

Windows WSL2 - Start Kafka in KRaft mode

CLI (Command Line Interface) 101

CLI Introduction

WINDOWS WARNING: PLEASE READ

Kafka Topics CLI

Kafka Console Producer CLI

Kafka Console Consumer CLI

Kafka Consumers in Group

Kafka Consumer Groups CLI

Resetting Offsets

Quiz on CLI

Kafka UI - Conduktor Demo

Conduktor - Demo

Kafka Java Programming 101

Kafka SDK List

Creating Kafka Project

Java Producer

Java Producer Callbacks

Java Producer with Keys

Java Consumer

Java Consumer - Graceful Shutdown

Java Consumer inside Consumer Group

Java Consumer Incremental Cooperative Rebalance & Static Group Membership

Java Consumer Incremental Cooperative Rebalance - Practice

Java Consumer Auto Offset Commit Behavior

Programming - Advanced Tutorials

Quiz on Java Programming 101

===== Kafka Real World Project =====

Real World Project Overview

Real World Exercise - Solution

Kafka Wikimedia Producer & Advanced Producer Configurations

IMPORTANT: Start Local Kafka with Conduktor using Docker

Wikimedia Producer Project Setup

Wikimedia Producer Implementation

Wikimedia Producer Run

Wikimedia Producer - Producer Config Intros

Producer Acknowledgements Deep Dive

Producer Retries

Idempotent Producer

Safe Kafka Producer Settings

Wikimedia Producer Safe Producer Implementation

Kafka Message Compression

linger.ms and batch.size Producer settings

Wikimedia Producer High Throughput Implementation

Producer Default Partitioner & Sticky Partitioner

[Advanced] max.block.ms and buffer.memory

Quiz on Producer Configurations

OpenSearch Consumer & Advanced Consumer Configurations

OpenSearch Consumer - Project Overview

OpenSearch Consumer - Project Setup

Setting up OpenSearch on Docker

Setting up OpenSearch on the Cloud

OpenSearch 101

OpenSearch Consumer Implementation - Part 1

OpenSearch Consumer Implementation Part 2

Consumer Delivery Semantics

OpenSearch Consumer Implementation Part 3 - Idempotence

Consumer Offsets Commit Strategies

OpenSearch Consumer Implementation Part 4 - Delivery Semantics

OpenSearch Consumer Implementation Part 5 - Batching Data

Consumer Offset Reset Behavior

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Develops skills such as working knowledge of Kafka and related technologies, which is standard in industry

Taught by Stephane Maarek, a recognized expert in Kafka

Covers fundamentals and advanced topics, providing a solid foundation for beginners and intermediate learners

Offers hands-on projects, giving learners practical experience with Kafka

Prerequisites, such as a basic understanding of Java, may limit accessibility for some learners

Focuses on Java programming, assuming learners have prior knowledge

Reviews summary

Excellent kafka course for beginners

According to students, this course on Apache Kafka for beginners is very well-received due to engaging assignments and advanced configuration. Students especially appreciated getting hands-on experience with examples. They found the instructor's knowledge to be impressive.

Learners appreciated that the course provided more than one setup solution.

"I also enjoyed that more than one setup solution is provided."

Learners enjoyed the instructor's knowledge and teaching style.

"Your knowledge is impressive."

Students loved that the course included many opportunities to get hands-on experience with examples.

"I enjoyed getting our hands dirty with examples."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Apache Kafka Series - Learn Apache Kafka for Beginners v3 with these activities:

Kafka Ecosystem Overview

Show steps

Reviewing this material will help you more fully understand the course.

Show steps

Read the Kafka Documentation
Watch YouTube videos on Kafka
Read the Kafka blog

Review Java programming basics

Show steps

Ensure you have a solid foundation in Java.

Show steps

Go through Java tutorials or documentation.
Practice writing simple Java programs.
Review Java concepts like data types, loops, and methods.

Kafka: The Definitive Guide

Show steps

Gain a comprehensive theoretical understanding of Kafka.

View Kafka: The Definitive Guide: Real-Time Data and... on Amazon

Show steps

Read through the chapters in order.
Take notes on the key concepts.
Complete the practice exercises.

Show all three activities

Career center

Learners who complete Apache Kafka Series - Learn Apache Kafka for Beginners v3 will develop knowledge and skills that may be useful to these careers:

Data Engineer

Data Engineers are in charge of designing, implementing, maintaining, and servicing big data pipelines. This course can help a Data Engineer get up to speed on Kafka, so they can use it to ingest, store, and distribute large amounts of streaming data. Apache Kafka is a massively scalable, high throughput, low-latency platform for handling real-time data feeds. Data Engineers can leverage the skills taught in this course to maximize the value of streaming data in the organization.

See salaries and explore the career path for Data Engineer

Data Architect

Data Architects design and build data architectures to meet the needs of an organization. This includes designing data models, selecting and implementing data storage technologies, and developing data integration and processing solutions. Apache Kafka is a popular technology for building real-time data architectures. By taking this course, Data Architects can gain the skills and knowledge they need to design and implement Kafka-based solutions.

See salaries and explore the career path for Data Architect

Data Scientist

Data Scientists use data to solve business problems and improve decision-making. Apache Kafka is a powerful tool for collecting, storing, and processing large amounts of data. This course can help Data Scientists learn how to use Kafka to build data pipelines and develop real-time data applications.

See salaries and explore the career path for Data Scientist

Software Engineer

Software Engineers design, develop, and maintain software systems. This course can benefit Software Engineers who want to learn about Apache Kafka, a popular platform for building real-time data applications. Kafka is used by many large organizations to process billions of events per day. Software Engineers who take this course will learn how to use Kafka to build scalable, reliable, and high-performance data applications.

See salaries and explore the career path for Software Engineer

Cloud Engineer

Cloud Engineers design, build, and maintain cloud-based systems. This course can benefit Cloud Engineers who want to learn about Apache Kafka, a popular platform for building real-time data applications in the cloud. Kafka is used by many large organizations to process billions of events per day. Cloud Engineers who take this course will learn how to use Kafka to build scalable, reliable, and high-performance data applications in the cloud.

See salaries and explore the career path for Cloud Engineer

Big Data Architect

Big Data Architects design and build big data architectures to meet the needs of an organization. This includes designing data models, selecting and implementing data storage technologies, and developing data integration and processing solutions. Apache Kafka is a popular technology for building real-time big data architectures. By taking this course, Big Data Architects can gain the skills and knowledge they need to design and implement Kafka-based solutions.

See salaries and explore the career path for Big Data Architect

Data Analyst

Data Analysts collect, analyze, and interpret data to help businesses make informed decisions. Apache Kafka is a powerful tool for collecting, storing, and processing large amounts of data. This course can help Data Analysts learn how to use Kafka to build data pipelines and develop real-time data applications.

See salaries and explore the career path for Data Analyst

DevOps Engineer

DevOps Engineers work to bridge the gap between development and operations teams. This includes automating build, test, and deployment processes, as well as monitoring and managing production systems. Apache Kafka is a popular platform for building real-time data applications. This course can help DevOps Engineers learn how to use Kafka to build and manage scalable, reliable, and high-performance data applications.

See salaries and explore the career path for DevOps Engineer

Systems Engineer

Systems Engineers design, implement, and maintain computer systems. This course can benefit Systems Engineers who want to learn about Apache Kafka, a popular platform for building real-time data applications. Kafka is used by many large organizations to process billions of events per day. Systems Engineers who take this course will learn how to use Kafka to build scalable, reliable, and high-performance data applications.

See salaries and explore the career path for Systems Engineer

Database Administrator

Database Administrators manage and maintain databases. This course can benefit Database Administrators who want to learn about Apache Kafka, a popular platform for building real-time data applications. Kafka is used by many large organizations to process billions of events per day. Database Administrators who take this course will learn how to use Kafka to build scalable, reliable, and high-performance data applications.

See salaries and explore the career path for Database Administrator

Information Security Analyst

Information Security Analysts design and implement security measures to protect an organization's data and systems. This course can benefit Information Security Analysts who want to learn about Apache Kafka, a popular platform for building real-time data applications. Kafka is used by many large organizations to process billions of events per day. Information Security Analysts who take this course will learn how to use Kafka to build secure and reliable data applications.

See salaries and explore the career path for Information Security Analyst

Data Warehouse Architect

Data Warehouse Architects design and build data warehouses to meet the needs of an organization. This includes designing data models, selecting and implementing data storage technologies, and developing data integration and processing solutions. Apache Kafka is a popular technology for building real-time data warehouses. By taking this course, Data Warehouse Architects can gain the skills and knowledge they need to design and implement Kafka-based solutions.

See salaries and explore the career path for Data Warehouse Architect

Machine Learning Engineer

Machine Learning Engineers design and build machine learning models to solve business problems. Apache Kafka is a powerful tool for collecting, storing, and processing large amounts of data. This course can help Machine Learning Engineers learn how to use Kafka to build data pipelines and develop real-time machine learning applications.

See salaries and explore the career path for Machine Learning Engineer

Technical Architect

Technical Architects design and build software systems to meet the needs of an organization. This includes designing system architectures, selecting and implementing technologies, and developing software solutions. Apache Kafka is a popular platform for building real-time data applications. This course can help Technical Architects learn how to use Kafka to build scalable, reliable, and high-performance data applications.

See salaries and explore the career path for Technical Architect

Enterprise Architect

Enterprise Architects design and build enterprise-wide software systems. This includes designing system architectures, selecting and implementing technologies, and developing software solutions. Apache Kafka is a popular platform for building real-time enterprise-wide data applications. This course can help Enterprise Architects learn how to use Kafka to build scalable, reliable, and high-performance data applications.

See salaries and explore the career path for Enterprise Architect