Improving Azure Data Lake Performance from Pluralsight

Running queries in Azure Data Lake? Are your queries costing too much? This course will help you learn how to take control of your Data Lake. Grab those pesky queries by the scruff of the neck and improve Azure Data Lake performance!

OK, so you are using Azure Data Lakes, and you think it's great. You just wish you could improve the performance of your U-SQL queries. Why does that query always read your entire data set? Why does this query take forever to complete? Like anything else in the Big Data world, your Azure Data Lake has to be structured around your data. This course, Improving Azure Data Lake Performance, will show you how to put the right structure in place. Then watch the magic start to happen! First, you'll see how an Azure Data Lake works behind the scenes – how it handles different types of data and how the storage of that data can be optimized. Next, you'll see how it's possible to optimize non-structured data. Finally, you'll be shown how structuring your data opens up a world of possibilities, including horizontal and vertical partitioning. This is where the real power of the Azure Data Lake comes to light! Horizontal partitioning allows you to defer a lot of control to the Data Lake, whereas vertical partitioning allows you – the developer – to take total control of how your data is partitioned and distributed within the Data Lake. When you're finished with this course, you'll understand how you can better optimize your jobs and save some cash. Software required: Visual Studio Community Edition 2017 with the Azure Data Lake and Stream Analytics Tools installed.

What's inside

Syllabus

Course Overview

Why Bother Organizing an Azure Data Lake?

Dividing and Conquering an Azure Data Lake

Kicking the Bucket – Manually Dividing an Azure Data Lake

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Develops skills and knowledge in structuring data in the Azure Data Lake, which is essential for optimizing U-SQL queries

Builds a solid foundation for optimizing Azure Data Lake performance, which may be useful for data analysts and engineers

Strong foundation in essential concepts, which may be valuable for those new to Azure Data Lake and optimizing U-SQL queries

Covers both structured and non-structured data optimization, which may appeal to a broader audience with varying data types

Teaches horizontal and vertical partitioning, which may be valuable for learners interested in advanced Azure Data Lake optimization techniques

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Improving Azure Data Lake Performance with these activities:

Review data storage concepts

Show steps

Review basic data storage concepts to strengthen your understanding of Azure Data Lake.

Browse courses on Data Storage

Show steps

Review different data storage types.
Compare and contrast different data structures.
Consider the pros and cons of different storage options.

Explore Azure Data Lake Tools

Show steps

熟悉 Azure Data Lake 的工具將有助於您開發和調試 U-SQL 查詢。

Browse courses on U-SQL

Show steps

安裝 Azure Data Lake 和 Stream Analytics 工具。
創建一個新的 Azure Data Lake 項目。
查看 Visual Studio 中的各種工具和功能。

Run Queries on Simulated Data

Show steps

Practice running U-SQL queries without needing to actually create a large data lake, which can take time to set up.

Browse courses on Querying Data

Show steps

Create a simulated data lake environment.
Write U-SQL queries to retrieve data from the simulated data lake.
Test the performance of your queries.

Ten other activities

Expand to see all activities and additional details

Show all 13 activities

Azure Data Lake Fundamentals Workshop

Show steps

Warms up students on the principles of organizing and structuring data within an Azure Data Lake to prepare them for later course modules on data optimization.

Browse courses on Azure Data Lake

Show steps

Follow the workshop guide
Complete the hands-on exercises
Review the provided code samples

Peer Review of Queries

Show steps

與其他學生分享您的 U-SQL 查詢並提供反饋將有助於您提高查詢的質量。

Browse courses on Query Optimization

Show steps

查找一個學習小組或線上論壇，您可以在其中分享您的查詢。
提交您的查詢以供審查。
提供對其他學生的查詢的回饋。

Data Query Optimization Exercises

Show steps

Strengthens students' ability to identify and resolve performance bottlenecks in their Azure Data Lake queries, improving their efficiency and cost-effectiveness.

Browse courses on Data Optimization

Show steps

Analyze sample queries
Identify areas for optimization
Implement optimization techniques

Optimize Azure Data Lake queries

Show steps

Practice optimizing Azure Data Lake queries to improve performance.

Browse courses on U-SQL

Show steps

Identify expensive queries.
Review query execution plans.
Identify performance bottlenecks.
Implement optimizations.
Test and validate improvements.

Build a Mini Data Lake

Show steps

建立一個小型數據湖將有助於您了解 Azure Data Lake 的運作方式以及如何優化數據存儲。

Browse courses on Azure Data Lake

Show steps

收集一些數據，例如日誌文件或感測器數據。
將數據上載到 Azure Data Lake。
使用 U-SQL 查詢數據。
最佳化數據湖的效能。

Implement data partitioning

Show steps

Create a data partitioning strategy that suits your Azure Data Lake and improves performance.

Browse courses on Data Partitioning

Show steps

Choose a partitioning scheme.
Implement partitioning in your Azure Data Lake.
Test and validate the performance of your partitioned data.
Re-partition data as needed.

Data Lake Organization Project

Show steps

Provides students with an opportunity to apply the concepts of data partitioning and structuring to a real-world scenario, reinforcing their understanding of data optimization techniques.

Browse courses on Data Organization

Show steps

Design a data organization strategy
Create Azure Data Lake containers and folders
Implement data partitioning techniques

Explore advanced Azure Data Lake techniques

Show steps

Explore advanced Azure Data Lake techniques to enhance your data management skills.

Show steps

Review Microsoft's Azure Data Lake documentation.
Follow online tutorials on advanced Azure Data Lake features.
Experiment with advanced techniques in your own Azure Data Lake.

Develop a Data Lake Optimization Plan

Show steps

建立一個數據湖優化計畫將有助於您識別和解決數據湖效能瓶頸。

Browse courses on Azure Data Lake

Show steps

評估當前數據湖的效能。
識別效能瓶頸。
開發優化策略。
實施優化策略。
監控數據湖的效能。

Contribute to Open Source Data Lake Projects

Show steps

Actively participate in the Data Lake community by contributing to open-source projects, providing valuable insights, and collaborating with others to enhance the ecosystem.

Browse courses on Community Involvement

Show steps

Identify open-source Data Lake projects that align with your interests
Join community forums and discussions to connect with other contributors
Propose and develop improvements, fixes, or new features

Career center

Learners who complete Improving Azure Data Lake Performance will develop knowledge and skills that may be useful to these careers:

Big Data Engineer

Big Data Engineers design and manage the architecture of big data systems. They work closely with other IT professionals to ensure that the big data system meets the needs of the business. This course may be useful to Big Data Engineers who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Big Data Engineers who want to be able to design and manage big data systems that are efficient and reliable.

See salaries and explore the career path for Big Data Engineer

Chief Data Officer

Chief Data Officers are responsible for the overall management of data within an organization. They work closely with other executives to ensure that the organization is using data to its full potential. This course may be useful to Chief Data Officers who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Chief Data Officers who want to be able to make informed decisions about how to use data to improve the performance of their organizations.

See salaries and explore the career path for Chief Data Officer

Data Analytics Manager

Data Analytics Managers lead teams of data analysts and oversee the development and implementation of data analytics solutions. They work closely with other business leaders to ensure that the data analytics solutions meet the needs of the business. This course may be useful to Data Analytics Managers who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Data Analytics Managers who want to be able to lead teams of data analysts who are able to develop and implement efficient and reliable data analytics solutions.

See salaries and explore the career path for Data Analytics Manager

Data Governance Officer

Data Governance Officers are responsible for developing and implementing data governance policies and procedures. They work closely with other business leaders to ensure that the organization is using data in a responsible and ethical manner. This course may be useful to Data Governance Officers who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Data Governance Officers who want to be able to make informed decisions about how to use Azure Data Lake to improve the performance of their organizations.

See salaries and explore the career path for Data Governance Officer

Database Manager

Database Managers are responsible for the overall management of databases within an organization. They work closely with other IT professionals to ensure that the databases are running smoothly and that data is protected. This course may be useful to Database Managers who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Database Managers who want to be able to make informed decisions about how to use Azure Data Lake to improve the performance of their databases.

See salaries and explore the career path for Database Manager

Data Quality Manager

Data Quality Managers are responsible for ensuring the quality of data within an organization. They work closely with other IT professionals to develop and implement data quality standards and procedures. This course may be useful to Data Quality Managers who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Data Quality Managers who want to be able to make informed decisions about how to use Azure Data Lake to improve the performance of their organizations.

See salaries and explore the career path for Data Quality Manager

Cloud Architect

Cloud Architects design and manage the architecture of cloud computing systems. They work closely with other IT professionals to ensure that the cloud computing system meets the needs of the business. This course may be useful to Cloud Architects who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Cloud Architects who want to be able to design and manage cloud computing systems that are efficient and reliable.

See salaries and explore the career path for Cloud Architect

Data Architect

Data Architects design and manage the architecture of data systems. They work closely with other IT professionals to ensure that the data system meets the needs of the business. This course may be useful to Data Architects who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Data Architects who want to be able to design and manage data systems that are efficient and reliable.

See salaries and explore the career path for Data Architect

Database Administrator

Database Administrators are responsible for the installation, configuration, maintenance, and performance monitoring of database management systems. They work closely with other IT professionals to ensure that the database is running smoothly and that data is protected. This course may be useful because it discusses how to optimize the performance of Azure Data Lake. This is an important skill for Database Administrators who want to ensure that their databases are running efficiently.

See salaries and explore the career path for Database Administrator

Project Manager

Project Managers are responsible for planning, executing, and controlling projects. They work closely with stakeholders to ensure that projects are completed on time, within budget, and to the required quality standards. This course may be useful to Project Managers who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Project Managers who want to be able to manage projects that involve the use of Azure Data Lake.

See salaries and explore the career path for Project Manager

Data Engineer

Data Engineers are responsible for designing, building, and maintaining the systems that store and process data. They work closely with data scientists and other data analysts to ensure that the data is accurate, reliable, and accessible. This course may be useful to Data Engineers who want to learn more about how to optimize the performance of Azure Data Lake.

See salaries and explore the career path for Data Engineer

Data Scientist

Data Scientists use their knowledge of statistics, mathematics, and computer science to extract insights from data. They work closely with other data professionals to develop models and algorithms that can be used to improve business decision-making. This course may be useful to Data Scientists who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Data Scientists who want to ensure that their models and algorithms are running efficiently.

See salaries and explore the career path for Data Scientist

Software Engineer

Software Engineers design, develop, and maintain software applications. They work closely with other engineers to ensure that the software is reliable, efficient, and meets the needs of the users. This course may be useful to Software Engineers who want to learn more about how to optimize the performance of Azure Data Lake.

See salaries and explore the career path for Software Engineer

Business Analyst

Business Analysts work with businesses to understand their needs and develop solutions that will help them achieve their goals. They use their knowledge of business processes, data analysis, and technology to develop recommendations that can improve efficiency and profitability. This course may be useful to Business Analysts who want to learn more about how to optimize the performance of Azure Data Lake. This is an important skill for Business Analysts who want to be able to make recommendations that will improve the performance of their businesses.

See salaries and explore the career path for Business Analyst

Systems Administrator

Systems Administrators make sure that all company computer systems are up and running, and help employees troubleshoot problems with their computers, software, or other technologies. This course may be useful because the course discusses the use of Azure Data Lake and Stream Analytics tools. These are essential tools for a Systems Administrator.

See salaries and explore the career path for Systems Administrator