May 1, 2024
Updated November 14, 2026
15 minute read
Navigating the World of Data Serialization
Data serialization is the process of converting complex data structures or objects into a format that can be easily stored, transmitted, and later reconstructed. Think of it like disassembling a complex toy, packaging its parts neatly into a box with instructions, so it can be shipped and then perfectly reassembled elsewhere. This "packaging" allows data to move between different applications, systems, or over networks, regardless of the underlying programming languages or architectures. The reverse process, reassembling the toy from its parts, is known as deserialization.
hetm1i|
Find a path to becoming a Data Serialization. Learn more at:
OpenCourser.com/topic/hetm1i/data
Reading list
We've selected 16 books
that we think will supplement your
learning. Use these to
develop background knowledge, enrich your coursework, and gain a
deeper understanding of the topics covered in
Data Serialization.
Chapter 4 stands as the most cited comparative analysis of data encoding and schema evolution in modern software engineering. Martin Kleppmann dissects JSON, XML, MessagePack, Apache Thrift, Protocol Buffers, and Apache Avro, illustrating byte-level tag numbering, writer/reader schema resolution, and forward/backward compatibility.
Centered on Protocol Buffers (Protobuf v3) as the strongly typed binary serialization substrate for microservice communication, Kasun Indrasiri and Danesh Kuruppu explain wire-type encoding, varint compression, field reservation, and cross-language code generation.
Eliminating serialization and deserialization CPU overhead altogether through standardized zero-copy columnar memory layouts, Apache Arrow has transformed analytical data interchange. Core contributor Matthew Topol details Arrow IPC buffers, FlatBuffers metadata, Parquet integration, and Arrow Flight RPC.
Separating transport framing from pluggable serialization protocols—including BinaryProtocol, CompactProtocol, and JSONProtocol—Apache Thrift pioneered polyglot RPC serialization. Randy Abernethy walks engineers through IDL struct evolution, union types, and custom serialization stacks.
Covering the most ubiquitous human-readable serialization format on the web, Tom Marrs demonstrates how to enforce structural contracts with JSON Schema, transform payloads with jq and JSON-LD, and benchmark streaming parsers in enterprise architectures.
Dives deep into Protocol Buffers wire encoding, varint compression, field tagging, and backward-compatible schema evolution, equipping engineers with practical Buf and compiler plugin workflows for multi-language binary serialization.
Comparing row-oriented serialization (Avro, SequenceFiles, JSON) against columnar encodings (Apache Parquet, Apache ORC) at the storage block level, four Cloudera engineers explain run-length encoding, dictionary compression, bit-packing, and predicate pushdown.
Decoupling event producers from consumers in streaming pipelines requires centralized schema registries and evolutive binary serialization. Gwen Shapira and co-authors detail custom Kafka serializers, Apache Avro and Protobuf integration, and compatibility enforcement.
Devoting an entire chapter to the security vulnerabilities, brittle object graphs, and versioning pitfalls of native language object serialization, Joshua Bloch demonstrates defensive readObject implementations, serialization proxies, and why cross-platform formats should be preferred.
Examining how Rust's Serde framework achieves zero-overhead compile-time data serialization by decoupling Rust data structures from wire formats (JSON, Bincode, CBOR, Postcard, MessagePack) via visitor traits and procedural macros.
Powering X.509 TLS certificates, cellular telecommunications, and aviation protocols, Abstract Syntax Notation One (ASN.1) introduced tag-length-value (TLV) binary encodings long before Protobuf. John Larmouth explains BER, DER, and packed encoding rules (PER).
Demonstrates how high-throughput event streams depend on disciplined serialization contracts, walking through Avro, Protobuf, and JSON Schema Registry integration alongside binary record framing for distributed producers and consumers.
Understanding how compact binary serializers pack integers and strings requires fluency in variable-length integer codes, ZigZag encoding, prefix codes, and entropy coding. David Salomon provides the mathematical foundations underlying compact wire representations.
Exploring binary byte sequences, memoryviews, struct module packing, pickle security hazards, and Pydantic runtime schema validation, Luciano Ramalho equips Python engineers to serialize and validate complex objects safely.
Showing how data layout and serialization structure directly govern payload size and network latency, Colt McAnlis and Aleks Haecky guide practitioners through bit-packing, delta encoding, and restructuring serialized objects for LZ77/Zstandard compression.
Tracing how humanity encoded complex messages into serial streams—from West African talking drums and Morse telegraph codebooks to Claude Shannon's noiseless channel coding theorem—reveals the deep intellectual lineage behind every modern wire format.
For more information about how these books relate to this course, visit:
OpenCourser.com/topic/hetm1i/data