Azure Data Factory For Data Engineers - Project on Covid19 from Udemy

Major updates to the course since the launch

October 2023 - Updates related to UI changes to Storage Browser & Azure Data Factory. Renaming of Azure Active Directory to Microsoft Entra ID & Default settings changes to Devops Organisation

January 2023 - Updates to section 3 (Environment Set-up) to reflect the change to the User Interface. Re-recorded 5 lessons.

November 2022 - Addition of sections 15 & 16 focusing on Continuous Integration & Continuous Delivery (CI/CD)

Welcome.

I am looking forward to helping you with learning one of the in-demand data engineering tools in the cloud, Azure Data Factory (ADF). This course has been taught with implementing a data engineering solution using Azure Data Factory (ADF) for a real world problem of reporting Covid-19 trends and prediction of the spread of this virus.

This is like no other course in Udemy for Azure Data Factory or Data Engineering Technologies. Once you have completed the course including all the assignments, I strongly believe that you will be in a position to start a real world data engineering project on your own and also proficient on Azure Data Factory (ADF).

I have also included lessons on the storage solutions such as Azure Data Lake Storage, Azure Blob Storage, Azure SQL Database etc. Also, there are lessons on Azure HDInsight and Azure Databricks. I have even included lessons on building reports using Power BI on the data processed by the Azure Data Factory data pipelines. I have considered the machine learning models to be out of scope. You can use this data to build your own models and predict the spread.

The course follows a logical progression of real world project implementation with technical concepts being explained and the data pipelines in Azure Data Factory (ADF) being built at the same time. Even-though this course is not specifically designed to teach you the skills required for passing the Azure Data Engineer Associate Certification exam DP203, it can greatly help you get most of the necessary skills required for the exam.

I value your time as much as I do mine. So, I have designed this course to be fast-paced and to the point. Also, the course has been taught with simple English and no jargons. I start the course from basis and by the end of the course you will be proficient in the technologies used.

Currently the course teaches you the following

Azure Data Factory

Building a solution architecture for a data engineering solution using Azure Data Engineering technologies such as Azure Data Factory (ADF), Azure Data Lake Gen2, Azure Blob Storage, Azure SQL Database, Azure Databricks, Azure HDInsight and Microsoft PowerBI.
Integrating data from HTTP clients, Azure Blob Storage and Azure Data Lake Gen2 using Azure Data Factory.
Branching and Chaining activities in Azure Data Factory (ADF) Pipelines using control flow activities such as Get Metadata. If Condition, ForEach, Delete, Validation etc.
Using Parameters and Variables in Pipelines, Datasets and LinkedServices to create a metadata driven pipelines in Azure Data Factory (ADF)
Debugging the data pipelines and resolving issues.
Scheduling pipelines using triggers such as Event Trigger, Schedule Trigger and Tumbling Window Trigger in Azure Data Factory (ADF)
Creating Mapping Data Flows to create transformation logic. The course covers all of the transformation steps such as Source, Filter, Select, Pivot, Lookup, Conditional Split, Derived Column, Aggregate, Join and Sink transformation.
Debugging data flows, investigating issues, fixing failures etc
Implementing Azure Data Factory pipelines to invoke Mapping Data Flows and executing them.
Creating ADF pipelines to execute HDInsight activities and carry out data transformations.
Creating ADF pipelines to execute Databricks Notebook activities to carry out transformations.
Creating dependency between pipelines to orchestrate the data flow
Creating dependency between triggers to orchestrate the data flow
Monitoring data pipelines, creating alerts, reporting of metrics from the Azure Data Factory Monitor.
Monitoring of Data Factory pipelines using Azure Monitor and setting diagnostic setting to be forwarded to Azure Storage Account or Log Analytics Workspace.
Creating Log Analytics workspace, creating workbooks and charts from log analytics on the Azure Data Factory pipelines
Implementing the Azure Data Factory Analytics monitoring tool and how to extend the capability further.

Azure Storage Solutions

Creating Azure Storage Account, Creating containers, Uploading data, Access Control (IAM), Using Azure Storage explorer to interact with the storage account
Creating Azure Data Lake Gen2, Creating containers, Uploading data, Access Control (IAM), Using Azure Storage explorer to interact with the storage account
Creating Azure SQL Database, Pricing Tiers, Creating Admin User, Creating Tables, Loading Data and Querying the database.

Azure HDInsight & Databricks

Creating HDInsight Clusters, Interacting with the UI, Using Ambari, Creating Hive tables, Invoking HDInsight activities from Azure Data Factory
Creating Azure Databricks Workspace, Creating Databricks clusters, Mounting storage accounts, Creating Databricks notebooks, performing transformations using Databricks notebooks, Invoking Databricks notebooks from Azure Data Factory.

Azure Devops (CI/CD)

Creating Azure Devops Environment and configuring Azure Devops Git Repository
CI/ CD process for releasing Azure Data Factory artefacts to higher environments
Creating build and release pipelines in Azure Devops to release code to higher environments (Test/ Prod)
Configuring/ Parameterise CI/CD pipelines to release ADF pipelines that access Azure Data Lake Storage.

What's inside

Learning objectives

You will learn how to build a real-world data pipeline in azure data factory (adf).
You will acquire good data engineering skills in azure using azure data factory (adf), azure data lake storage gen2, azure sql database and azure monitor
You will learn how to ingest data from sources such as http and azure blob storage into azure data lake gen2 using azure data factory (adf)
You will learn how to transform data using data flows in azure data factory (adf) and load into azure data lake storage gen2
You will learn how to transform data using databricks notebook activity in azure data factory (adf) and load into azure data lake storage gen2
You will learn how to transform data using azure hdinsight activity in azure data factory (adf) and load into azure data lake storage gen2
You will learn how to load transformed data from azure data lake storage gen2 to azure sql database using azure data factory (adf)

You will learn extensively about triggers in azure data factory (adf) and how to use them to schedule the data pipelines.
You will learn how to monitor pipelines using azure data factory (adf), azure monitor and log analytics with a real-world project.
You will learn how to build production ready pipelines and good practices and naming standards
You will learn the topics required on azure data factory to pass the azure data engineer associate certification exam dp203
You will learn about how to create ci/cd pipelines in azure devops to release adf pipelines to higher environments (testing/ production)
Show more
Show less

You will learn how to build a real-world data pipeline in azure data factory (adf).
You will acquire good data engineering skills in azure using azure data factory (adf), azure data lake storage gen2, azure sql database and azure monitor
You will learn how to ingest data from sources such as http and azure blob storage into azure data lake gen2 using azure data factory (adf)
You will learn how to transform data using data flows in azure data factory (adf) and load into azure data lake storage gen2
You will learn how to transform data using databricks notebook activity in azure data factory (adf) and load into azure data lake storage gen2
You will learn how to transform data using azure hdinsight activity in azure data factory (adf) and load into azure data lake storage gen2
You will learn how to load transformed data from azure data lake storage gen2 to azure sql database using azure data factory (adf)
You will learn extensively about triggers in azure data factory (adf) and how to use them to schedule the data pipelines.
You will learn how to monitor pipelines using azure data factory (adf), azure monitor and log analytics with a real-world project.
You will learn how to build production ready pipelines and good practices and naming standards
You will learn the topics required on azure data factory to pass the azure data engineer associate certification exam dp203
You will learn about how to create ci/cd pipelines in azure devops to release adf pipelines to higher environments (testing/ production)
Show more
Show less

Syllabus

Introduction

Course Introduction

Course Structure

Course Slides Download

Traffic lights

Read about what's good

what should give you pause

and possible dealbreakers

Uses a real-world project, reporting Covid-19 trends, to teach Azure Data Factory, which provides practical experience for learners

Covers Azure Data Lake Storage, Azure Blob Storage, Azure SQL Database, Azure HDInsight, and Azure Databricks, offering a broad understanding of the Azure ecosystem

Includes continuous integration and continuous delivery (CI/CD) using Azure DevOps, which is essential for production-ready data pipelines

Teaches how to build metadata-driven pipelines using parameters and variables, which is a key skill for creating scalable and maintainable data solutions

Focuses on using Azure Data Factory to address a real-world problem, which can help learners build a portfolio of projects

Includes updates related to UI changes and renaming of Azure Active Directory, which helps learners stay current with the Azure platform

Reviews summary

Azure data factory real-world project

According to learners, this course offers a hands-on, project-based approach to mastering Azure Data Factory and related services. Students particularly praise the course's focus on building a real-world data pipeline using Covid-19 data, finding it highly practical and directly applicable to their work. The instructor is widely commended for providing clear explanations and being responsive to questions. While some note that the course moves at a fast pace, the frequent updates to keep content current with Azure's evolving UI and new features like CI/CD integration are seen as a significant strength, ensuring the material remains relevant and up-to-date.

Moves quickly, requiring focus or prior knowledge.

"So, I have designed this course to be fast-paced and to the point."

"Some parts were a bit fast for a complete beginner, needed to pause and rewatch."

"It moves quickly, but if you keep up with the hands-on, it works."

Builds a solid understanding of core concepts.

"The course provides a solid foundation on Azure Data Factory and other Azure services."

"Really helped solidify my understanding of the ADF core principles."

"Starts from basis and by the end of the course you will be proficient in the technologies used."

Course actively updated to reflect Azure changes.

"Major updates to the course since the launch... October 2023 - Updates related to UI changes..."

"January 2023 - Updates to section 3 (Environment Set-up) to reflect the change to the User Interface..."

"November 2022 - Addition of sections 15 & 16 focusing on Continuous Integration & Continuous Delivery (CI/CD)"

"It's great to see the instructor keeps the course updated as Azure changes."

Provides skills directly applicable to real-world jobs.

"I really appreciate how practical this course is. I can use these skills immediately."

"It is a great course for data engineers, helping you build real-world solutions."

"The course covers relevant topics for anyone working with Azure data services."

"Highly relevant for anyone wanting to apply ADF in a professional setting."

Highly praised for clarity, knowledge, and support.

"The instructor is very knowledgeable and explains concepts clearly."

"Instructor clarifies all doubts with patience and the course structure is also good."

"Amazing instructor! He makes complex topics easy to understand."

"The instructor's guidance throughout the project was invaluable."

Focuses on building a full data pipeline project.

"This is a very good course and it really helped me build my confidence with Azure data Factory as we build a real world project..."

"The approach of doing a project from scratch until end on Covid19 data is the best method to learn ADF."

"It is amazing to see how a single project can be built using Azure Data factory."

"The hands-on project approach made learning ADF much more engaging and practical."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Azure Data Factory For Data Engineers - Project on Covid19 with these activities:

Review Azure Fundamentals

Show steps

Reinforce your understanding of core Azure concepts before diving into Data Factory. This will help you better grasp the underlying infrastructure and services ADF interacts with.

Browse courses on Azure Fundamentals

Show steps

Review the official Azure Fundamentals documentation.
Complete a practice quiz on Azure services.
Familiarize yourself with Azure pricing models.

Read 'Pro Azure Data Lake'

Show steps

Gain a deeper understanding of Azure Data Lake Storage Gen2, a key component of the course project. This will help you optimize your data storage and processing strategies.

View Putting Knowledge to Work on Amazon

Show steps

Focus on chapters related to data ingestion and storage.
Compare the book's recommendations to the course's approach.
Experiment with different data lake configurations.

Read 'Azure Data Factory Cookbook'

Show steps

Supplement your learning with a cookbook that provides practical solutions to common ADF challenges. This will give you a broader perspective on how to apply ADF in different scenarios.

View Putting Knowledge to Work on Amazon

Show steps

Select chapters relevant to the course project.
Implement one or two recipes from the book.
Compare the book's approach to the course's approach.

Four other activities

Expand to see all activities and additional details

Show all seven activities

Implement different data flow transformations

Show steps

Practice implementing various data flow transformations in ADF to become proficient in data manipulation. This will improve your ability to design efficient and effective data pipelines.

Show steps

Choose a set of data flow transformations to practice.
Create a sample dataset to use for testing.
Implement each transformation and test its functionality.
Document your findings and best practices.

Build a Data Quality Dashboard

Show steps

Enhance the data pipeline by adding data quality checks and building a dashboard to monitor data quality metrics. This will give you experience with implementing data governance practices.

Show steps

Define data quality metrics for the Covid-19 data.
Implement data quality checks in ADF data flows.
Store data quality metrics in Azure SQL Database.
Create a Power BI dashboard to visualize data quality.

Extend the Covid-19 Data Pipeline

Show steps

Solidify your understanding by expanding the course project with additional data sources or transformations. This will challenge you to apply what you've learned in a more complex scenario.

Show steps

Identify a new data source related to Covid-19.
Ingest the new data into Azure Data Lake Storage.
Create a new data flow to transform the data.
Integrate the new data into the existing Power BI reports.

Contribute to an open-source ADF project

Show steps

Contribute to an open-source project related to Azure Data Factory to gain real-world experience and collaborate with other developers. This will enhance your skills and broaden your network.

Show steps

Find an open-source project related to ADF on GitHub.
Review the project's documentation and contribution guidelines.
Identify a bug or feature to work on.
Submit a pull request with your changes.

Career center

Learners who complete Azure Data Factory For Data Engineers - Project on Covid19 will develop knowledge and skills that may be useful to these careers:

Data Engineer

A data engineer builds and maintains the infrastructure for data processing and storage, and this course directly aligns with those responsibilities. This role involves designing data pipelines, optimizing data flow, and ensuring data quality. This course, with its focus on Azure Data Factory, provides practical experience in building data pipelines, integrating different data sources, and transforming data using various techniques. The course covers key aspects such as data ingestion, transformation with Data Flows, and orchestration with triggers, all of which are everyday tasks for a data engineer. The hands-on experience gained in this course will help you build a strong foundation in cloud-based data engineering.

See salaries and explore the career path for Data Engineer

Cloud Data Architect

A cloud data architect designs and oversees the implementation of cloud-based data solutions, and this course provides a valuable toolkit for such a role. It requires a deep understanding of cloud data technologies, data modeling, and data integration strategies. This course directly equips you with the skills to use Azure Data Factory for real-world data processing, encompassing storage solutions like Azure Data Lake Storage and SQL Database, along with orchestration of pipelines, which are key elements of a cloud data architecture. The course also covers data transformation using Azure Data Factory's Data Flows, and monitoring methods, important for maintaining a robust system. With the practical experience gained, you'll be well-prepared to design efficient and scalable cloud data solutions.

See salaries and explore the career path for Cloud Data Architect

Data Integration Specialist

A data integration specialist is responsible for designing, developing, and maintaining the flow of data between different systems. This course provides a strong foundation for such a role, particularly focusing on cloud-based data integration using Azure Data Factory. The course covers data ingestion from various sources, data transformation techniques, and pipeline orchestration, all crucial skills for integrating disparate data systems. The course emphasizes practical aspects and helps in debugging pipeline issues, which helps you gain the experience to ensure smooth data flow. The specific focus on Azure services and continuous integration and continuous delivery further enhances your ability to effectively integrate data within a cloud environment.

See salaries and explore the career path for Data Integration Specialist

ETL Developer

An ETL developer designs and implements Extract, Transform, and Load processes to move data from various sources into a data warehouse or data lake, and this course provides direct training in the use of Azure Data Factory for that purpose. This role involves creating data pipelines, transforming data to meet business requirements, and ensuring the data is loaded correctly. The course provides hands-on experience with building data pipelines that ingest data from multiple sources, such as HTTP and Azure Blob Storage, and transforms it using Data Flows and other techniques. The course also teaches how to debug data pipelines and implement CI/CD, which is critical to an ETL developer. This course allows you to gain expertise with these activities.

See salaries and explore the career path for ETL Developer

Cloud Solutions Engineer

A cloud solutions engineer designs and implements cloud-based solutions, and this course helps develop valuable skills for this role. This role requires proficiency in cloud services, data engineering principles and automation. The course's focus on Azure Data Factory provides hands-on experience building and managing data pipelines, integrating data from different sources, and automating data workflows. You learn to use Azure Data Lake Storage, Azure SQL Database, and other related services. The practical nature of the course, with its focus on a real-world project also gives you experience solving end-to-end data problems, preparing you for solution-oriented work as a Cloud Solutions Engineer.

See salaries and explore the career path for Cloud Solutions Engineer

Data Pipeline Developer

A data pipeline developer focuses on building and maintaining the systems that move and transform data, and this course is highly relevant to this role. This work specifically involves creating data pipelines using various technologies, and ensuring data flows smoothly and efficiently. This course helps learn to work with Azure Data Factory to build data pipelines for ingesting, transforming, and loading data, as seen in the course's various hands-on projects. You learn how to use control flow activities, data flows, triggers, and other essential features for pipeline development. Through this training you gain practical experience and understand how to construct robust data pipelines.

See salaries and explore the career path for Data Pipeline Developer

Azure Data Engineer

An Azure data engineer specializes in building and managing data solutions on the Azure cloud platform, and this course provides an excellent foundation. This role requires expertise in Azure services, data processing, and data pipeline implementation. This course, focused on using Azure Data Factory, provides hands-on experience with data ingestion, transformation, and orchestration within the Azure ecosystem. You also learn how to work with various storage solutions, integrate data from multiple sources, and use other services. The course highlights the practical application of Azure Data Engineer skills, which provides a very helpful background for an Azure Data Engineer.

See salaries and explore the career path for Azure Data Engineer

Business Intelligence Developer

A Business Intelligence developer is responsible for creating data analysis reports and dashboards to support business decisions. This course helps build the data engineering skills needed in this role. While Business Intelligence often focuses on the analysis and presentation of data, such as reports built with Power BI as seen in this course, it is also necessary to understand how the underlying data is moved, managed and transformed. Thus, having expertise with data pipelines, data sources, and transformation logic, as taught in this course, is essential. The course also shows how to use Azure Data Factory and related tools, which are useful for building data pipelines that feed into business intelligence applications.

See salaries and explore the career path for Business Intelligence Developer

Cloud Data Specialist

A cloud data specialist focuses on managing and optimizing data within cloud environments, and this course provides valuable experience in that domain. This role often involves working with cloud data storage, data processing, and data integration solutions, and you would use those technologies as part of this course. The course covers key Azure services such as Data Factory, Data Lake Storage, and SQL Databases while demonstrating the end-to-end data pipelines. The experience in data ingestion, transformation, and pipeline orchestration, all of which are taught in this course, would be directly applicable for a cloud data specialist working within the Azure ecosystem. The course helps prepare you to effectively manage and optimize data within a cloud environment.

See salaries and explore the career path for Cloud Data Specialist

Database Administrator

A database administrator manages the performance, security, and integrity of databases, and this course may be helpful for those who need an understanding of cloud data principles. While database administrators typically manage specific databases, this course helps one understand how such databases are part of the broader data ecosystem. This course covers Azure SQL Database, along with other services for data storage and processing. The concepts of data pipelines, data transformation, and data integration learned in this course are relevant to those who need to understand how databases work with broader data systems. The course helps you improve your overall understanding of data systems that interact with databases.

See salaries and explore the career path for Database Administrator

Data Analyst

Although a data analyst primarily focuses on analyzing data, this course may be useful in providing context for the broader data landscape. This role involves interpreting data, identifying trends, and communicating insights, and a greater understanding of the overall data pipeline can be helpful. While this course is not directly focused on data analysis itself, it teaches the underlying data engineering processes that provide the data for analysis. The course’s focus on data ingestion, transformation and storage, especially as it relates to Azure, allows a data analyst to better appreciate the journey of data. The course helps you understand data systems and their processes.

See salaries and explore the career path for Data Analyst

Solutions Architect

A solutions architect designs and oversees the implementation of technology solutions, and this course may be useful in understanding the data-related aspects of projects. This role requires broad technical knowledge, including understanding data systems and data integration. This course helps build your understanding about data solutions, and how to leverage Azure Data Factory. While this course does not cover the breadth of skills a solutions architect uses, it provides practical experience with data pipelines and cloud-based data engineering, which is relevant. The course helps extend your knowledge of how data engineering works within the larger context of technology solutions.

See salaries and explore the career path for Solutions Architect

Software Developer

A software developer designs, codes, and tests software applications, and this course may be useful in understanding data integration in software systems. While this course does not focus on software development, the core principles and concepts behind data pipelines, integration, and cloud infrastructure covered in this course are relevant to building modern software applications. The course teaches how data is moved and managed in a cloud environment using services such as Azure Data Factory. The course may help you understand how to build software systems involving data pipelines, which are increasingly common in modern software.

See salaries and explore the career path for Software Developer

IT Project Manager

An IT project manager oversees the planning, execution, and completion of technology projects, and this course may be useful in better understanding the details of data-related projects. While this course is not focused on project management techniques, it provides hands-on experience in using data engineering technologies in Azure. The course helps develop a technical understanding of Azure Data Factory, and related services. The course provides familiarity with technical terminology and processes, which may be helpful for those managing IT projects that involve data infrastructure.

See salaries and explore the career path for IT Project Manager

Machine Learning Engineer

A machine learning engineer develops and deploys machine learning models, and this course may be useful for them to understand how data is prepared for modeling. Though this course does not cover machine learning models themselves, it teaches how data is prepared, stored, and processed via use of Azure Data Factory, Azure Storage, and related services. The hands-on experience with data pipelines, data transformation, and orchestration gained from the course allows you to better understand the data preparation stage of a machine learning workflow. Thus, this course may be helpful in building a foundation for working with data in AI and machine learning applications.

See salaries and explore the career path for Machine Learning Engineer

Azure Data Factory For Data Engineers - Project on Covid19

What's inside

Learning objectives

Syllabus

Traffic lights

Save this course

Reviews summary

Azure data factory real-world project

Activities

Career center

Reading list

Share

Similar courses