May 1, 2024
Updated June 4, 2025
20 minute read
Understanding Data Streaming: A Comprehensive Guide for Aspiring Professionals
Data streaming is the continuous flow of data generated by various sources. Think of it as a ceaseless river of information, rather than a still lake. This data is processed, analyzed, and acted upon in real-time or near real-time, enabling organizations to gain immediate insights and make timely decisions. Unlike traditional batch processing, which collects and processes data in large chunks at scheduled intervals, data streaming deals with data as it arrives, piece by piece.
nub8xl|
Find a path to becoming a Data Streaming. Learn more at:
OpenCourser.com/topic/nub8xl/data
Reading list
We've selected 25 books
that we think will supplement your
learning. Use these to
develop background knowledge, enrich your coursework, and gain a
deeper understanding of the topics covered in
Data Streaming.
Comprehensive guide to Apache Kafka, a core technology in data streaming. It covers Kafka basics, architecture, and how to build reliable data pipelines and stream processing applications. It's considered a must-read for anyone working with Kafka and provides a solid understanding of a key streaming platform. Engineers from Confluent and LinkedIn, responsible for developing Kafka, explain how to deploy production Kafka clusters and build scalable stream processing applications.
Delves into the fundamental concepts and challenges of large-scale data processing, with a strong focus on streaming systems. It provides a theoretical foundation and explores various processing models and architectures. It valuable resource for gaining a deeper understanding of the principles behind stream processing. Familiarity with lambda architecture is helpful, and the book expands on prior knowledge of batch programming.
Focuses specifically on building stream processing applications with Kafka Streams and ksqlDB. It provides practical examples and covers topics like the KStream and Processor APIs, integrating with Kafka Connect, and building event-driven applications. This revised edition includes more of the Kafka platform and full coverage of ksqlDB.
This practical guide shows data engineers how to use Kafka Streams and ksqlDB to build scalable stream processing applications. It covers moving, enriching, and transforming data in real time and great way to familiarize yourself with stream processing concepts using these tools.
Practical guide to building scalable streaming applications using Apache Flink. It explores the fundamental concepts of parallel stream processing and how Flink differs from batch processing. It's valuable for understanding a major stream processing framework beyond Kafka.
Provides a comprehensive introduction to data stream processing, covering the fundamental concepts, architectures, and algorithms. It also includes hands-on examples and exercises to help readers gain practical experience.
While not exclusively about data streaming, this book provides a foundational understanding of data systems, including concepts crucial to streaming architectures like reliability, scalability, and maintainability. It is highly regarded in the industry and serves as excellent background reading for anyone diving into data-intensive topics. valuable reference tool for understanding the trade-offs and fundamental principles behind various data processing systems.
Focuses on stream processing capabilities within Apache Spark, covering both the older Spark Streaming and the newer Structured Streaming APIs. It's useful for those familiar with Spark and looking to apply it to streaming use cases. It helps in understanding how Spark fits into the stream processing landscape.
Comprehensive guide to Apache Pulsar, another distributed messaging and streaming platform. It covers building scalable streaming messaging systems and explores Pulsar's unique benefits and features like Pulsar Functions. It's valuable for understanding alternatives to Kafka in the streaming ecosystem.
Focuses on building real-time event systems using popular streaming platforms like Kafka and Kinesis. It provides practical guidance and examples for working with event streams and is valuable for understanding how to implement streaming solutions with specific technologies.
Authored by key figures in the Apache Flink community, this book likely offers practical insights and guidance on using Flink for stream processing. It would be a valuable resource for developers and engineers working with or planning to use Flink.
Offers a broad overview of the data engineering landscape, including the data engineering lifecycle and best practices for building robust data systems. While not solely focused on streaming, it provides essential context and foundational knowledge for understanding where data streaming fits within a larger data architecture. It helps in evaluating technologies and incorporating data governance and security.
Explores the concept of event-driven architectures and how they can be built using technologies like Kafka. It provides insights into designing and implementing microservices that communicate via event streams, a common pattern in modern data streaming applications.
Covers the theory and practice of real-time data analytics, with a focus on how to build and deploy data stream processing systems. It also includes case studies from a variety of industries.
Explores the application of data mesh principles to streaming data. It discusses designing a streaming data mesh using technologies like Kafka and delves into topics like data domains, data products, and data governance in a streaming context. It's relevant for understanding contemporary architectural patterns in data streaming.
Introduces the concept of streaming databases and how they aim to unify batch and stream processing. It explores the fundamentals of these databases and how they can be used to build real-time solutions, including constructing materialized views from streams.
Covers the theory and practice of streaming data analysis, with a focus on how to develop scalable and efficient machine learning algorithms for streaming data.
Introduces the concept of Data Mesh, a decentralized data architecture. Understanding Data Mesh is valuable for professionals working with data streaming as streaming data often plays a significant role in a data mesh paradigm, particularly in enabling real-time data products.
Serves as a good introductory guide to the concepts and requirements of streaming and real-time data systems. It explores designs for applications that interact with fast-flowing data and introduces key technologies like Spark, Storm, Kafka, and Flink. It's a solid starting point for beginners to grasp the basics of the real-time data pipeline.
Provides an accessible introduction to the core concepts of streaming systems. It is likely helpful for those new to the topic and looking for a conceptual understanding before diving into specific technologies. It appears to be a good starting point for building intuition about streaming data.
Data streaming is often a key component of data integration strategies, especially for real-time data flows. provides a foundational understanding of data integration principles, which is essential for designing effective streaming data pipelines.
Covers the fundamental concepts and algorithms of stream data processing, with a focus on how to develop scalable and efficient stream data processing systems.
Data streaming is increasingly used to feed real-time data to machine learning models. provides a strong foundation in designing and building production-ready ML systems, which is highly relevant when considering how streaming data integrates into an end-to-end ML pipeline.
For more information about how these books relate to this course, visit:
OpenCourser.com/topic/nub8xl/data
Save
