Introduction to the Azure Data Lake and U-SQL from Pluralsight

The Big Data revolution has exposed the limitations of traditional data processing models like cubes and ETL. Learn a different way of doing things with the Azure Data Lake, using the U-SQL language to query raw data files and create databases.

Building good reporting structures can be difficult, especially when those pesky users keep asking for new reports. Throw Big Data into the mix and things become a lot more complicated. What if you didn’t need to build any data models at all, or you could build models that could be quickly put up and torn down? In this course, Introduction to the Azure Data Lake and U-SQL, you'll be introduced to Azure Data Lake and the U-SQL language, and learn how to abandon ETL. First, you'll delve into querying by using the powerful U-SQL language, built straight into the Azure Data Lake. Next, you'll discover how to throw your files into the Data Lake and query them directly without needing to load them into a database. Finally, you'll learn about how Azure Data Lakes offers the best of both worlds, with support for unstructured files and structured databases. By the end of this course, you’ll not only know what a Data Lake is, you’ll know how to populate it, query it, and develop for it using Visual Studio. Software required: Visual Studio 2017 Community Edition and Azure subscription (optional).

What's inside

Syllabus

Course Overview

Abandoning ETL with an Azure Data Lake and U-SQL

Tool Time – Saving Money with Visual Studio

Cutting Development Timescales

Building Dams - Structuring the Data Lake

Converting and Manipulating Data

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Explores emerging database systems, which are highly relevant to industry

Develops databases and data structures, which are core skills for data science

Taught by Mike McQuillan, who is recognized for their work in data science

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Introduction to the Azure Data Lake and U-SQL with these activities:

Explore Azure Data Lake tutorials

Show steps

Follow online tutorials provided by Microsoft to get hands-on experience with Azure Data Lake before diving into the course content.

Browse courses on Azure Data Lake

Show steps

Find a suitable tutorial on the Microsoft website
Follow the steps in the tutorial to create a Data Lake
Upload some sample data to your Data Lake

Learn relational database fundamentals

Show steps

Start by exploring relational databases to better grasp data modeling concepts.

Browse courses on Databases

Show steps

Take a beginner-friendly course on relational databases.
Complete a tutorial on data modeling.
Read articles and blog posts about database design principles.

Read "Cloud Computing: Concepts, Technology and Architecture"

Show steps

Review and understand cloud computing concepts to understand how Azure Data Lake fits into the broader cloud ecosystem.

View Cloud Computing: Concepts, Technology &... on Amazon

Show steps

Read Chapters 1 and 2
Complete the exercises at the end of each chapter

17 other activities

Expand to see all activities and additional details

Show all 20 activities

Practice using SQL

Show steps

Review basic SQL syntax and practice writing queries to refresh your skills before starting the course.

Browse courses on SQL

Show steps

Create a practice database
Write queries to select, insert, update, and delete data
Run your queries and verify the results

Review Basic Databases

Show steps

Refresh your knowledge of basic database concepts, such as data modeling, query languages, and database administration, to ensure a solid foundation for learning about Azure Data Lake and U-SQL.

Browse courses on Relational Databases

Show steps

Review database terminology and concepts
Go through examples of data modeling using ER diagrams
Practice writing basic SQL queries

Join a study group focused on big data with Azure

Show steps

Collaborate with peers to enhance your understanding of big data with Azure.

Browse courses on Big Data

Show steps

Find or create a study group.
Set regular meeting times.
Discuss course topics and work on assignments together.
Share resources and knowledge with each other.

Complete Azure Data Lake Tutorial (Beginner)

Show steps

Follow a guided tutorial to gain practical experience with Azure Data Lake.

Browse courses on Azure Data Lake

Show steps

Go to Microsoft Learn
Search for "Azure Data Lake Tutorial"
Complete the "Create an Azure Data Lake Gen2 storage account" module

Practice U-SQL queries

Show steps

Reinforce your understanding of U-SQL syntax and query construction by practicing writing and executing queries.

Browse courses on U-SQL

Show steps

Review U-SQL syntax and documentation
Create a sample data set in Azure Data Lake
Write U-SQL queries to extract, filter, and aggregate data

Attend a workshop on big data analytics with Azure

Show steps

Attend a workshop to learn about big data analytics using Azure Data Lake and U-SQL.

Browse courses on Big Data Analytics

Show steps

Research and find a reputable workshop.
Register for the workshop.
Attend the workshop and actively participate.
Follow up with the workshop organizers after the event.

Build a simple Data Lake and U-SQL query

Show steps

Hands-on practice will help you grasp the core concepts of the course and solidify your understanding.

Browse courses on Azure Data Lake

Show steps

Create an Azure Data Lake account
Upload sample data to your Data Lake
Write a U-SQL query to retrieve data from your Data Lake
Run your query and analyze the results

Join a Study Group

Show steps

Collaborate with peers to discuss concepts, share knowledge, and work through problems together.

Browse courses on Azure Data Lake

Show steps

Find or create a study group
Meet regularly to discuss course material and work on projects

Practice U-SQL Queries

Show steps

Reinforce your understanding of U-SQL through repetitive exercises.

Browse courses on U-SQL

Show steps

Find U-SQL practice problems online
Solve 10 practice problems
Review your answers

Follow U-SQL Tutorials

Show steps

Complete tutorials on the U-SQL language to gain hands-on experience with querying raw data files and creating databases in Azure Data Lake.

Browse courses on U-SQL

Show steps

Find tutorials on the Microsoft Learn platform
Follow the step-by-step instructions
Experiment with different U-SQL queries

Practice U-SQL Queries

Show steps

Reinforce your understanding of U-SQL by practicing writing queries on sample data sets, experimenting with different operators and functions.

Browse courses on U-SQL

Show steps

Create a practice data set
Write queries to extract specific data
Test your queries with different data sets

Build a sample data lake using Azure Data Lake

Show steps

Put your knowledge into practice by building your own data lake using Azure Data Lake.

Browse courses on Azure Data Lake

Show steps

Create an Azure Data Lake account.
Ingest data into your data lake.
Query your data using U-SQL.
Create a data visualization based on your query results.

Practice writing U-SQL queries

Show steps

Solidify your understanding of U-SQL by practicing writing queries.

Browse courses on U-SQL

Show steps

Find a dataset to practice with.
Write U-SQL queries to retrieve, filter, and aggregate data.
Test your queries and troubleshoot any errors.
Share your queries with others for feedback.

Develop a data pipeline using Azure Data Lake and U-SQL

Show steps

Build a practical understanding of how to design and implement data pipelines using Azure Data Lake and U-SQL.

Show steps

Define the data ingestion, processing, and storage requirements
Create an Azure Data Lake account and store data
Develop U-SQL scripts for data transformation and analysis
Deploy and monitor the data pipeline

Mentor a Junior Data Engineer

Show steps

Solidify your understanding of Azure Data Lake by mentoring someone who is new to the field.

Browse courses on Mentoring

Show steps

Volunteer to mentor at a local university or online platform
Pair with a junior data engineer and provide guidance on Azure Data Lake

Develop a U-SQL Project

Show steps

Apply your U-SQL skills by creating a project that involves querying and manipulating data in Azure Data Lake, demonstrating your ability to work with real-world data.

Browse courses on U-SQL

Show steps

Identify a problem or dataset
Design a U-SQL solution
Develop and test the U-SQL code
Deploy the solution in Azure Data Lake

Build a Data Lake Pipeline

Show steps

Apply your skills to build a real-world data lake pipeline for a specific use case.

Browse courses on Azure Data Lake

Show steps

Identify a use case and data sources
Design the data lake architecture
Implement the data ingestion and processing pipeline
Test and deploy the pipeline

Career center

Learners who complete Introduction to the Azure Data Lake and U-SQL will develop knowledge and skills that may be useful to these careers:

Data Analyst

Data Analysts use data to make informed decisions and solve problems. This course introduces the Azure Data Lake and U-SQL, which are powerful tools for querying and manipulating data. By learning these tools, you can develop the skills needed to succeed as a Data Analyst. You'll be able to query large datasets, build data models, and create reports that can help your organization make better decisions.

See salaries and explore the career path for Data Analyst

Data Scientist

Data Scientists use data to build predictive models and solve complex problems. This course introduces the Azure Data Lake and U-SQL, which are powerful tools for querying and manipulating data. By learning these tools, you can develop the skills needed to succeed as a Data Scientist. You'll be able to build data pipelines, train machine learning models, and create visualizations that can help your organization make better decisions.

See salaries and explore the career path for Data Scientist

Data Engineer

Data Engineers build and maintain the infrastructure that stores and processes data. This course introduces the Azure Data Lake, which is a cloud-based data storage and processing platform. By learning about the Azure Data Lake, you can develop the skills needed to succeed as a Data Engineer. You'll be able to create and manage data lakes, build data pipelines, and optimize data processing performance.

See salaries and explore the career path for Data Engineer

Big Data Engineer

Big Data Engineers design and build systems for processing and analyzing large datasets. This course introduces the Azure Data Lake, which is a cloud-based data storage and processing platform. By learning about the Azure Data Lake, you can develop the skills needed to succeed as a Big Data Engineer. You'll be able to create and manage data lakes, build data pipelines, and optimize data processing performance.

See salaries and explore the career path for Big Data Engineer

Database Administrator

Database Administrators manage and maintain databases. This course introduces the Azure Data Lake, which is a cloud-based data storage and processing platform. By learning about the Azure Data Lake, you can develop the skills needed to succeed as a Database Administrator. You'll be able to create and manage data lakes, build data pipelines, and optimize data processing performance.

See salaries and explore the career path for Database Administrator

Software Engineer

Software Engineers design, develop, and maintain software applications. This course introduces the U-SQL language, which is a powerful tool for querying and manipulating data. By learning U-SQL, you can develop the skills needed to succeed as a Software Engineer. You'll be able to build data pipelines, create data models, and develop applications that can help your organization make better decisions.

See salaries and explore the career path for Software Engineer

Cloud Architect

Cloud Architects design and implement cloud computing solutions. This course introduces the Azure Data Lake, which is a cloud-based data storage and processing platform. By learning about the Azure Data Lake, you can develop the skills needed to succeed as a Cloud Architect. You'll be able to design and implement data lakes, build data pipelines, and optimize data processing performance.

See salaries and explore the career path for Cloud Architect

Data Architect

Data Architects design and implement data management solutions. This course introduces the Azure Data Lake, which is a cloud-based data storage and processing platform. By learning about the Azure Data Lake, you can develop the skills needed to succeed as a Data Architect. You'll be able to design and implement data lakes, build data pipelines, and optimize data processing performance.

See salaries and explore the career path for Data Architect

Business Analyst

Business Analysts analyze business processes and develop solutions to improve efficiency. This course introduces the Azure Data Lake and U-SQL, which are powerful tools for querying and manipulating data. By learning these tools, you can develop the skills needed to succeed as a Business Analyst. You'll be able to query large datasets, build data models, and create reports that can help your organization make better decisions.

See salaries and explore the career path for Business Analyst

Project Manager

Project Managers plan and execute projects. This course may be useful for Project Managers who are working on data-related projects. By learning about the Azure Data Lake and U-SQL, you can develop the skills needed to manage data-related projects more effectively. You'll be able to understand the technical aspects of data projects, communicate with technical team members, and make informed decisions about data management.

See salaries and explore the career path for Project Manager

Data Visualization Analyst

Data Visualization Analysts create visualizations that communicate data insights. This course may be useful for Data Visualization Analysts who want to learn more about querying and manipulating data. By learning about the Azure Data Lake and U-SQL, you can develop the skills needed to create more effective data visualizations. You'll be able to query large datasets, build data models, and create visualizations that can help your organization make better decisions.

See salaries and explore the career path for Data Visualization Analyst

Machine Learning Engineer

Machine Learning Engineers build and maintain machine learning models. This course may be useful for Machine Learning Engineers who want to learn more about querying and manipulating data. By learning about the Azure Data Lake and U-SQL, you can develop the skills needed to build and maintain more effective machine learning models. You'll be able to query large datasets, build data models, and create visualizations that can help you improve the performance of your machine learning models.

See salaries and explore the career path for Machine Learning Engineer

Data Governance Analyst

Data Governance Analysts develop and implement data governance policies and procedures. This course may be useful for Data Governance Analysts who want to learn more about querying and manipulating data. By learning about the Azure Data Lake and U-SQL, you can develop the skills needed to create and implement more effective data governance policies and procedures. You'll be able to query large datasets, build data models, and create visualizations that can help you understand and manage your organization's data.

See salaries and explore the career path for Data Governance Analyst

Information Architect

Information Architects design and manage information systems. This course may be useful for Information Architects who want to learn more about querying and manipulating data. By learning about the Azure Data Lake and U-SQL, you can develop the skills needed to design and manage more effective information systems. You'll be able to query large datasets, build data models, and create visualizations that can help you understand and manage your organization's data.

See salaries and explore the career path for Information Architect

Knowledge Management Specialist

Knowledge Management Specialists develop and manage knowledge management systems. This course may be useful for Knowledge Management Specialists who want to learn more about querying and manipulating data. By learning about the Azure Data Lake and U-SQL, you can develop the skills needed to create and manage more effective knowledge management systems. You'll be able to query large datasets, build data models, and create visualizations that can help you understand and manage your organization's knowledge.

See salaries and explore the career path for Knowledge Management Specialist