March 29, 2024
Updated April 1, 2025
16 minute read
ETL Developer: Architecting the Flow of Data
At its core, an ETL Developer is a specialized software engineer focused on the critical processes that allow businesses to leverage their data effectively. They design, build, and maintain the systems responsible for Extracting data from various sources, Transforming it into a usable and consistent format, and Loading it into a target system, typically a data warehouse or data lake, for analysis and reporting. Think of them as the architects and plumbers of the data world, ensuring information flows smoothly and accurately from its origin to its destination where it can provide valuable insights.
46meim|
Find a path to becoming a ETL Developer. Learn more at:
OpenCourser.com/career/46meim/etl
Reading list
We've selected some books
that we think will supplement your
learning. Use these to
develop background knowledge, enrich your coursework, and gain a
deeper understanding of the topics covered in
ETL Developer.
Is specifically dedicated to the ETL process within a data warehousing context. It provides practical techniques and best practices for extracting, cleaning, transforming, and loading data. It's an excellent resource for understanding the intricacies of building robust ETL systems and is considered a key text for ETL developers and architects. This book is highly valuable as a reference tool for practitioners.
Is widely considered a foundational text for anyone learning T-SQL. It covers the essential concepts and logic of the language with hands-on exercises, making it ideal for gaining a broad understanding. While titled 'Fundamentals,' it delves into topics crucial for solidifying understanding beyond basic syntax.
Comprehensive guide to the Power Query M language. It's suitable for those who want to build a solid foundation and delve into advanced concepts and data transformations. It serves as an indispensable companion for data analysts, BI professionals, and enthusiasts looking to master Power Query. This book is valuable as both a learning resource and a reference tool.
Written by the father of data warehousing, this book focuses specifically on SCD techniques, providing a thorough examination of different SCD types, their advantages and disadvantages, and implementation considerations.
The updated edition of Gil Raviv's popular book, co-authored with Daniil Maslyuk, provides the latest information on using Power Query in Power BI and Excel. It covers updated features and enhanced techniques for data preparation and transformation. This highly relevant and contemporary resource for anyone working with Power Query.
Provides a solid introduction to the field of data engineering, covering the entire data lifecycle from data generation to consumption. ETL core component of this lifecycle, and the book provides context and foundational knowledge for understanding its role within a larger data ecosystem. It's a great starting point for anyone entering the field.
This forthcoming book, expected in mid-2025, is specifically aimed at users who want to go beyond the Power Query user interface and leverage the M language for advanced data cleaning. It's a highly anticipated resource for those looking to master complex data transformation scenarios using M code.
While not exclusively about ETL, this book provides a deep dive into the fundamental concepts of data systems, including batch and stream processing, which are integral to modern ETL pipelines. It helps in understanding the trade-offs and design choices behind various data processing technologies. is essential for gaining a broader understanding of the landscape in which ETL operates and is highly recommended for architects and senior engineers.
Focuses specifically on the M code within Power Query, aiming to teach users how to write efficient code. It's presented as a beginner's guide to mastering M code for data transformation in Excel and Power BI. It's a recent publication that addresses the importance of writing optimized M code.
Airflow popular platform for orchestrating complex data pipelines, including ETL workflows. provides a practical guide to using Airflow for building, scheduling, and monitoring data pipelines. It's highly relevant for data engineers and developers managing ETL processes in a production environment.
Provides a practical approach to using Power Query in both Power BI and Excel. It covers essential techniques for collecting, combining, and transforming data. It's a valuable resource for users who want to efficiently manage and analyze data, and it includes advanced techniques and focuses on data transformation. A second edition, co-authored with Daniil Maslyuk, was released in late 2024, covering updated features.
Focuses on implementing data engineering concepts, including ETL, using Python. It's a practical guide for those who want to build data pipelines with a popular programming language. It covers extracting, transforming, and loading data using Python libraries and tools. This book is valuable for hands-on learners and professionals using Python for ETL tasks.
This book, often considered a companion to 'T-SQL Querying' by the same lead author, focuses on the programming aspects of T-SQL, including stored procedures, triggers, and dynamic SQL. It's essential for those moving beyond basic querying to build more complex database solutions. While an older edition, the core programming concepts remain valuable.
Apache Spark powerful engine for big data processing, commonly used in ETL workflows for large datasets. This book, written by one of Spark's creators, offers a comprehensive guide to using Spark for various data processing tasks. It's particularly relevant for those dealing with big data ETL challenges.
Authored by one of the co-authors of 'M Is for (Data) Monkey,' this book focuses on leveraging Power Query to streamline data tasks in Excel and Power BI. It's a practical guide for improving data workflows and is suitable for users who want to enhance their efficiency with Power Query.
By Brijendra Singh provides a comprehensive guide to Informatica PowerCenter 9.x, covering its core concepts, advanced features, and real-world use cases. It valuable resource for data integration professionals seeking to master Informatica PowerCenter.
Focuses specifically on the creation and optimization of stored procedures in SQL Server. Given the course topic's emphasis on stored procedures, this book provides targeted and in-depth coverage, making it a highly relevant resource for mastering this specific aspect of T-SQL programming.
Provides a deep understanding of how SQL Server works internally, which is crucial for writing high-performance T-SQL. While not solely focused on T-SQL syntax, its coverage of the database engine significantly aids in understanding query execution and optimization. It valuable resource for experienced developers and DBAs.
Following a cookbook format, this book offers a series of worked examples and easy-to-follow steps for performing various data transformation tasks in Power Query. It's a practical guide that is useful for anyone learning Power Query and wanting to practice creating queries. It covers both Power Query in Power BI Desktop and Dataflows and includes recently-added features.
Provides a comprehensive overview of dimensional data warehousing and includes a chapter on SCDs, covering the theoretical foundations and practical considerations for managing changing data in a data warehouse.
Takes a professional approach to using Power BI Desktop, which involves extensive use of Power Query for data import and transformation. It's suitable for users who want to develop a deeper understanding of Power BI's capabilities, including advanced Power Query techniques.
Data quality critical aspect of the ETL process, particularly the 'Transform' phase. comprehensive guide to understanding and improving data quality. It provides methodologies and techniques for identifying, measuring, and addressing data quality issues.
Focuses on writing efficient and scalable T-SQL code in SQL Server 2019 and beyond. It addresses contemporary topics related to performance and best practices, making it highly relevant for professionals working with recent versions of SQL Server.
Focuses on the fundamentals of SQL performance from a developer's perspective. While not T-SQL specific, the concepts covered regarding indexing and query execution are universally applicable and essential for anyone writing T-SQL who wants to ensure their queries perform well.
SQL fundamental language for data manipulation in ETL processes. highlights common mistakes and inefficient practices in SQL programming, which can directly impact the performance and maintainability of ETL code. It's a valuable resource for anyone writing SQL for ETL.
Offers a more in-depth look at applying Power BI, which includes significant coverage of data preparation with Power Query. It's suitable for users who want to move beyond the basics and understand how to apply Power Query in real-world business scenarios.
Another foundational book from W.H. Inmon, focusing on the enterprise data warehouse. It provides a different perspective on data warehousing compared to the Kimball approach, which can be beneficial for a well-rounded understanding. It discusses the architecture and design of data warehouses at an enterprise level, relevant to the scope and complexity of ETL.
While primarily focused on Power BI and Power Pivot, this book provides essential context for how Power Query fits into the overall data analysis process. Authored by renowned experts in the field, it's a valuable resource for understanding the role of data preparation using Power Query before modeling and analysis.
Effective ETL relies heavily on good data governance and master data management. provides a thorough understanding of these crucial concepts. It's valuable for professionals who need to understand the broader data landscape and how to ensure data quality and consistency within their ETL processes.
Provides a business-oriented perspective on data modeling and includes a chapter on SCDs, explaining how to model changing data in a way that meets business requirements.
Authored by a well-known expert in the Microsoft BI field, this book provides a guide to Power Query for users of both Power BI and Excel. While an older publication, its coverage of the underlying M language remains relevant. It's a good resource for understanding how to use Power Query to solve data-loading problems and go beyond the graphical interface.
Explores advanced SQL concepts related to set-based thinking and the use of various types of tables. Celko's unique perspective challenges readers to think about SQL problems in different ways, which can significantly enhance a T-SQL developer's ability to write efficient and elegant code.
Good data modeling is crucial for effective Power BI reports, and Power Query is the foundational step. This book, while focused on data modeling, reinforces the importance of well-prepared data using Power Query. It's valuable for users who want to understand how their Power Query efforts impact the subsequent data modeling phase.
Introduces agile methodologies to data warehouse design, which can impact how ETL processes are developed and managed. It emphasizes collaborative approaches and iterative development, offering a modern perspective on data warehousing projects.
Offers a collection of T-SQL solutions to common database tasks, serving as a practical reference for developers. It covers a wide range of recipes, including those for stored procedures, which are directly relevant to the course topics.
Offers practical tips and techniques for writing better SQL code. The advice provided is highly relevant to T-SQL development and can help practitioners improve the efficiency and maintainability of their queries and scripts. It's a useful reference for refining T-SQL practices.
Modern ETL often involves extracting data from and loading data into nonrelational databases. provides an overview of various NoSQL databases and their characteristics. Understanding these databases is increasingly important for data engineers working with diverse data sources and destinations.
Provides a business-oriented introduction to data science and includes a section on SCDs, explaining how to use SCDs for data analysis and decision-making.
Presents a series of SQL puzzles and their solutions, challenging readers to think creatively about how to solve problems using SQL. While not T-SQL specific, the puzzles often highlight nuances of the language and can significantly deepen problem-solving skills applicable to T-SQL.
With the shift to the cloud, understanding cloud data warehousing is essential for modern ETL. provides an introduction to cloud data warehousing concepts and platforms. It helps in understanding how ETL processes are implemented and managed in cloud environments.
Is designed for absolute beginners to Power Query. It provides step-by-step guidance on how to use the tool for data transformation. It's a good starting point for high school or undergraduate students who are new to data manipulation and Power Query.
While not exclusively T-SQL, this book widely recommended resource for learning the fundamentals of SQL query writing. Its hands-on approach and clear explanations make it an excellent starting point for beginners to gain a broad understanding of data manipulation, which is directly applicable to T-SQL.
Data lakes are increasingly used alongside or instead of traditional data warehouses, impacting ETL strategies. introduces the concept of data lakes and their role in a data architecture. It's helpful for understanding how ETL or ELT processes differ in a data lake environment.
Provides a practical introduction to Power BI, including the use of Power Query for data preparation. It's suitable for beginners who want to learn self-service data analytics. It helps establish a foundational understanding of Power Query within the broader Power BI workflow.
Provides a comprehensive overview of machine learning for data science and includes a section on SCDs, explaining how to use SCDs for machine learning tasks.
Provides a comprehensive overview of big data analytics and includes a section on SCDs, explaining how to handle changing data in big data environments.
Focuses on data quality in data warehousing and includes a section on SCDs, discussing how to ensure the accuracy and consistency of changing data over time.
Is designed to help individuals prepare for a Microsoft certification exam focused on querying data with T-SQL. It covers the objectives of the exam and can be a useful resource for structured learning and validating T-SQL knowledge.
Is geared towards developers working with SQL Server and covers a wide range of T-SQL topics relevant to application development. It's a practical guide that helps solidify understanding through hands-on examples.
For more information about how these books relate to this course, visit:
OpenCourser.com/career/46meim/etl
Save
