We may earn an affiliate commission when you visit our partners.
Janani Ravi

This course will teach you how you can create, ingest data into, and work with Delta Lakes, an open-source storage layer that brings reliability to data stored in data lakes. Delta Lakes offer ACID transactions, unified batch and stream processing.

Read more

This course will teach you how you can create, ingest data into, and work with Delta Lakes, an open-source storage layer that brings reliability to data stored in data lakes. Delta Lakes offer ACID transactions, unified batch and stream processing.

The Databricks Data Lakehouse architecture is an innovative paradigm that combines the flexibility and low-cost storage offered by data lakes with the features and capabilities of a data warehouse. The lakehouse architecture achieves this by using metadata, indexing, and caching layer on top of data lake storage. This open-source storage layer is Delta Lake. This Delta Lake storage layer lies at the heart of Databricks’ lakehouse architecture.

In this course, Getting Started with Delta Lake on Databricks you will learn how exactly Delta Lakes supports transactions on cloud storage. First, you will learn the basic elements of Delta Lake namely Delta files, Delta tables, DeltaLog, and Delta optimizations.

Next, you will discover how you can get better performance from queries that you run on Delta tables using different optimizations. Here you will explore Delta caching, data skipping, and file layout optimizations such as partitioning, bin-packing, and z-order clustering.

Finally, you will explore how you can ingest data from external sources into Delta tables using batch and streaming ingestion. You will use the COPY INTO command for batch ingestion and the Databricks Auto Loader for stream ingestion.

When you are finished with this course, you will have the skills and ability to create, and ingest data into Delta Lakes and run optimal queries to extract insights.

Enroll now

Here's a deal for you

We found an offer that may be relevant to this course.
Save money when you learn. All coupon codes, vouchers, and discounts are applied automatically unless otherwise noted.

What's inside

Syllabus

Course Overview
Exploring Delta Lake on Databricks
Optimizing Queries on Delta Tables
Ingesting Batch and Streaming Data into Delta Tables
Read more

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Teaches using Databricks, industry-standard in data engineering
Covers batch ingestion, a crucial technique in working with data lakes
Taught by Janani Ravi, an expert in Delta Lake with extensive experience from working at AWS, Google, and Microsoft
Introduces Delta Lake optimizations, an essential element for improving processing on data lakes
Suitable for beginners who have prior knowledge of data lakes
Provides a foundation in Delta Lake concepts, such as Delta files and Delta optimizations, useful for beginners

Save this course

Save Getting Started with Delta Lake on Databricks to your list so you can find it easily later:
Save

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Getting Started with Delta Lake on Databricks with these activities:
Explore Databricks Lakehouse Architecture
Lay a solid foundation for this course by supplementing the course's introduction with an exploration and review of the key concepts behind Databricks Lakehouse Architecture.
Browse courses on Data Lakehouse
Show steps
  • Review online documentation on Databricks Lakehouse Architecture
  • Watch tutorial videos on the architecture and its components
Review previous knowledge of data warehousing concepts
Review foundational concepts of data warehousing to strengthen understanding of Delta Lake's role in the data lakehouse architecture.
Browse courses on Data Warehousing
Show steps
  • Revisit fundamental principles of data warehousing, such as data integration, data transformation, and data analysis.
  • Explore different data modeling techniques used in data warehouses, including star and snowflake schemas.
Practice Acid Transactions on Delta Tables
Practice committing and rolling back data changes in Delta tables, improving your understanding of ACID properties.
Browse courses on ACID Transactions
Show steps
  • Create a Delta table with sample data
  • Perform DML operations (INSERT, UPDATE, DELETE)
  • Commit the data changes
  • Rollback the data changes
  • Verify the data integrity
Six other activities
Expand to see all activities and additional details
Show all nine activities
SQL Optimization Techniques on Delta Tables
Gain proficiency in optimizing SQL queries on Delta tables, a core aspect of the course.
Browse courses on Caching
Show steps
  • Solve practice problems on query optimization techniques
Solve practice problems on Delta Lake optimization
Sharpen your skills in optimizing queries on Delta tables by solving practice problems.
Browse courses on Delta Caching
Show steps
  • Identify techniques for optimizing Delta Lake queries
  • Use practice problems to apply these techniques
  • Evaluate your solutions and identify areas for improvement
Develop a Delta Lake Tool and Software Collection
Assemble a comprehensive collection of tools and software specifically designed for Delta Lake, enabling students to explore and utilize the most relevant technologies.
Browse courses on Big Data Tools
Show steps
  • Research and identify available tools and software that complement Delta Lake.
  • Evaluate and select tools based on their functionality, ease of use, and compatibility with Delta Lake.
  • Create a central repository or documentation that provides access to the tool and software collection.
Attend a Delta Lake Hackathon
Engage in hands-on problem-solving and collaborate with other professionals to advance Delta Lake skills and knowledge.
Browse courses on Big Data Challenges
Show steps
  • Register for a Delta Lake hackathon hosted by Databricks or other organizations.
  • Form a team or join an existing one.
  • Develop innovative solutions using Delta Lake and related technologies.
Design a Data Ingestion Pipeline for Delta Lake
Demonstrate your understanding of data ingestion concepts by creating a comprehensive data ingestion pipeline for Delta Lake.
Browse courses on Data Ingestion
Show steps
  • Design the pipeline architecture and workflow
  • Implement batch ingestion using COPY INTO
  • Implement streaming ingestion using Databricks Auto Loader
Design and Implement a Data Lakehouse Architecture
Apply knowledge of Delta Lake and the lakehouse architecture to design and implement a comprehensive data management solution.
Show steps
  • Gather requirements and define use cases for the data lakehouse.
  • Design the architecture, including data storage, processing, and governance components.
  • Implement the data lakehouse using Delta Lake and other relevant technologies.
  • Evaluate and optimize the performance and scalability of the data lakehouse.

Career center

Learners who complete Getting Started with Delta Lake on Databricks will develop knowledge and skills that may be useful to these careers:
Data Scientist
Data Scientists use data to build models that can predict future outcomes. This course may be useful for Data Scientists who want to learn how to use Delta Lake to improve the performance of their models.
Data Analyst
Data Analysts use data to identify trends and patterns. This course may be useful for Data Analysts who want to learn how to use Delta Lake to analyze large datasets more efficiently.
Data Engineer
A Data Engineer is responsible for designing, building, and maintaining data pipelines. This course may be useful for Data Engineers who want to learn how to use Delta Lake to create reliable and scalable data pipelines.
Big Data Architect
Big Data Architects design and implement data solutions for large-scale data systems. This course may be useful for Big Data Architects who want to learn how to use Delta Lake to build reliable and scalable data solutions.
Cloud Engineer
Cloud Engineers design and implement cloud-based solutions. This course may be useful for Cloud Engineers who want to learn how to use Delta Lake to build reliable and scalable data solutions in the cloud.
Business Intelligence Analyst
Business Intelligence Analysts use data to make better business decisions. This course may be useful for Business Intelligence Analysts who want to learn how to use Delta Lake to analyze large datasets more efficiently.
Database Administrator
Database Administrators maintain and manage databases. This course may be useful for Database Administrators who want to learn how to use Delta Lake to improve the performance and reliability of their databases.
Data Warehouse Engineer
Data Warehouse Engineers design and implement data warehouses. This course may be useful for Data Warehouse Engineers who want to learn how to use Delta Lake to build reliable and scalable data warehouses.
Machine Learning Engineer
Machine Learning Engineers design and implement machine learning models. This course may be useful for Machine Learning Engineers who want to learn how to use Delta Lake to improve the performance and reliability of their machine learning models.
Statistician
Statisticians use data to draw conclusions about the world around us. This course may be useful for Statisticians who want to learn how to use Delta Lake to analyze large datasets more efficiently.
Data Architect
Data Architects design and implement data architectures. This course may be useful for Data Architects who want to learn how to use Delta Lake to build reliable and scalable data architectures.
Software Engineer
Software Engineers design, develop, and maintain software applications. This course may be useful for Software Engineers who want to learn how to use Delta Lake to build reliable and scalable data pipelines.

Reading list

We've selected eight books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Getting Started with Delta Lake on Databricks.
Provides background knowledge on designing and building scalable and reliable data-intensive applications. It covers topics such as data modeling, data storage, and data processing.
Provides a comprehensive overview of data warehousing concepts and best practices. It covers topics such as data modeling, data integration, and data analysis.
Provides a comprehensive overview of deep learning techniques and their application to various domains. It covers topics such as neural networks, convolutional neural networks, and recurrent neural networks.
Provides a comprehensive overview of the Hadoop ecosystem, including HDFS, MapReduce, and YARN. It covers topics such as data storage, data processing, and data analytics.
Provides a hands-on introduction to Delta Lake. It covers topics such as creating and managing Delta tables, querying Delta tables, and using Delta Lake with Spark SQL. It valuable resource for anyone who wants to learn how to use Delta Lake in their projects.
Provides a practical guide to Delta Lake for data engineers. It covers topics such as using Delta Lake for data ingestion, data transformation, and data quality. It valuable resource for anyone who wants to learn how to use Delta Lake to build and manage a scalable and reliable data pipeline.
Provides a guide to using Delta Lake for data warehousing. It covers topics such as data ingestion, data transformation, and data quality. It valuable resource for anyone who wants to learn how to use Delta Lake to build and manage a scalable and reliable data warehouse.

Share

Help others find this course page by sharing it with your friends and followers:

Similar courses

Here are nine courses similar to Getting Started with Delta Lake on Databricks.
Delta Lake with Azure Databricks: Deep Dive
Most relevant
Getting Started with the Databricks Lakehouse Platform
Most relevant
Data Engineering with Databricks
Most relevant
Apache Spark (TM) SQL for Data Analysts
Most relevant
Data Engineering using Databricks on AWS and Azure
Most relevant
Data Management with Databricks: Big Data with Delta Lakes
Most relevant
Data lakes and Lakehouses with Spark and Azure Databricks
Most relevant
Optimizing Apache Spark on Databricks
Most relevant
DP-203: Processing in Azure Using Batch Solutions
Most relevant
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2024 OpenCourser