Databricks Data Science and Engineering: Basic Tools and Infrastructure from Pluralsight

This course will teach you how to make the best use of Databricks assets such as notebooks, clusters, and repos to simplify the development and management of your big data applications.

Building robust and high-performing big data applications requires a well-configured environment. On the Databricks platform, this means setting up various assets such as clusters, notebooks, and repos in order to get the most out of the platform and make development and analysis work as smooth as possible.

In this course, Databricks Data Science and Engineering: Basic Tools and Infrastructure, you'll explore exactly how this can be accomplished.

First, you'll begin by creating and then making use of clusters, tables, files, and notebooks, and explore how all of these can be combined in order to build and run a simple application.

Next, you'll move on to the use of Databricks repos, which allow us to record changes to notebooks and related files in a workspace, and can be linked with an external Git repository. Then, you'll delve into how this linking can be performed, and explore how file additions, modifications, and removals can be performed and viewed on repos.

Finally, you'll move on to jobs which represent the execution of a task on Databricks - how job executions can be configured and scheduled, and how notifications at various stages of a job can be sent.

When you're finished with this course, you'll have skills and knowledge of Databricks resources such as clusters, notebooks, repos, and jobs, as well as their configurations, which will help you create a Databricks environment that is optimized for building and running applications and can help you get the most out of your data.

What's inside

Syllabus

Course Overview

Managing Databricks Workspace Assets

Developing Applications Using Notebooks

Configuring and Managing Job Executions

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Develops skills and knowledge in using Databricks clusters, notebooks, repos, and jobs, which are core skills for data engineers and scientists

Provides step-by-step guidance on creating and managing Databricks assets, building and running applications, and configuring and scheduling jobs

Taught by Kishan Iyer, an expert in Databricks Data Science and Engineering

Focuses on practical applications, making it highly relevant to industry professionals

Does not require extensive prerequisites, making it accessible to learners with varying experience levels

May require basic knowledge of data analysis and coding concepts

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Databricks Data Science and Engineering: Basic Tools and Infrastructure with these activities:

Review Big Data Concepts

Show steps

Refresh your knowledge of big data concepts such as data storage, processing, and analytics to enhance comprehension throughout the course.

Browse courses on Big Data

Show steps

Read articles or watch videos on big data fundamentals.
Review notes or textbooks from previous courses or online resources.
Participate in online discussions or forums to engage with other learners.

Review core programming concepts

Show steps

Refreshing core programming concepts will help you strengthen your foundation and improve your ability to apply these concepts in the context of Databricks.

Browse courses on Object-Oriented Programming

Show steps

Review tutorials and documentation on programming fundamentals
Practice writing code snippets and solving programming problems
Participate in online forums and communities for programming

Practice Writing Notebooks

Show steps

Revisit the general process of writing and developing Notebooks before beginning the course to help recall prior knowledge and increase the likelihood of understanding new concepts.

Browse courses on Notebooks

Show steps

Review online tutorials or documentation on notebook fundamentals.
Create a new notebook in a practice workspace or environment.
Write a simple notebook that includes code cells and markdown cells.
Test the notebook to ensure it runs without errors.
Consider sharing the notebook with a peer for review and feedback.

12 other activities

Expand to see all activities and additional details

Show all 15 activities

Compile Course Resources

Show steps

Organize and review available course materials, such as lecture notes, videos, and assignments, to establish a strong foundation.

Browse courses on Compilation

Show steps

Create a dedicated folder or digital notebook for course materials.
Download and organize lecture notes, slides, and assignments.
Add links to relevant online resources (e.g., articles, videos).

Explore Databricks Repos

Show steps

Familiarize yourself with the capabilities of Databricks Repos to better understand how to utilize them during the course.

Show steps

Follow official Databricks documentation or online tutorials on Repos.
Create a new repo in a practice workspace.
Link the repo to an existing Git repository.
Upload changes to notebooks and track the version history.

Create and Configure Cluster

Show steps

Practice creating and configuring Databricks clusters prior to the course to enhance understanding of cluster management and optimization.

Show steps

Start a Databricks trial account.
Create a new cluster with different configurations (e.g., size, type).
Configure cluster settings (e.g., autoscaling, termination policies).
Monitor cluster performance and make adjustments as needed.

Join a Study Group

Show steps

Connect with fellow learners and form a study group to enhance understanding and foster collaboration throughout the course.

Browse courses on Peer Support

Show steps

Reach out to classmates or online forums to find potential study partners.
Establish communication channels (e.g., messaging app, video conferencing).
Set regular meeting times to discuss course materials and assignments.

Create a sample notebook on Databricks

Show steps

Creating a sample notebook will give you hands-on experience working with Databricks and help you understand how to use notebooks for data exploration and analysis.

Show steps

Create a new notebook in Databricks
Import data into a dataframe
Perform basic data exploration and cleaning
Visualize data using charts and graphs

Participate in Databricks community events and challenges

Show steps

Participating in Databricks community events and challenges will provide you with additional opportunities to practice your skills, engage with other learners, and stay up-to-date on the latest developments in the field.

Show steps

Attend webinars and workshops hosted by Databricks
Participate in hackathons and competitions organized by Databricks
Join online communities and forums related to Databricks
Contribute to open-source projects or initiatives related to Databricks

Build a Simple Data Pipeline

Show steps

Develop a basic data pipeline using Databricks to gain hands-on experience with the platform's capabilities.

Browse courses on Data Pipeline

Show steps

Define the data source and destination.
Create a notebook to perform data extraction, transformation, and loading.
Set up a job to automate the pipeline execution.
Test and monitor the pipeline.

Solve practice problems on Databricks platform

Show steps

Solving practice problems will challenge your understanding of Databricks concepts and help you develop problem-solving skills.

Show steps

Use online resources to find practice problems
Solve problems using Databricks notebooks
Review solutions and learn from your mistakes
Repeat the process for more practice
Explore community forums and Q&A sites for additional problems

Contribute to Open Source Projects

Show steps

Gain practical experience and deepen understanding by contributing to open source projects related to Databricks.

Browse courses on Community Involvement

Show steps

Find open source projects related to Databricks on platforms like GitHub.
Review project documentation and identify areas where you can contribute.
Submit pull requests with your contributions.
Collaborate with other developers and maintainers.

Build a mini project using Databricks

Show steps

Building a mini project will allow you to apply your Databricks skills in a practical setting and gain experience in developing real-world applications.

Show steps

Identify a problem or use case
Design and plan your project
Develop and implement your solution using Databricks
Test and evaluate your project
Iterate and improve your project

Mentor or tutor other learners in Databricks

Show steps

Mentoring or tutoring others in Databricks will reinforce your understanding of the concepts and help you develop your communication and leadership skills.

Show steps

Identify opportunities to mentor or tutor others
Prepare materials and resources for mentoring sessions
Meet with mentees regularly to provide guidance and support
Provide constructive feedback and encouragement

Contribute to open-source projects related to Databricks

Show steps

Contributing to open-source projects related to Databricks will allow you to make a meaningful contribution to the community and gain experience working on real-world projects.

Show steps

Identify open-source projects related to Databricks
Review project documentation and codebase
Identify areas where you can contribute
Submit pull requests with your contributions
Collaborate with other contributors and maintainers

Career center

Learners who complete Databricks Data Science and Engineering: Basic Tools and Infrastructure will develop knowledge and skills that may be useful to these careers:

Data Scientist

The job of a Data Scientist requires an understanding of using Databricks' resources such as clusters, notebooks, repos, and jobs. Databricks Data Science and Engineering: Basic Tools and Infrastructure teaches these very things. Someone interested in this field would do well to take this course to enhance their skillset.

See salaries and explore the career path for Data Scientist

Big Data Architect

Designing and implementing data management solutions is the role of a Big Data Architect. Databricks Data Science and Engineering: Basic Tools and Infrastructure teaches the skills needed to implement data management solutions.

See salaries and explore the career path for Big Data Architect

Big Data Engineer

A Big Data Engineer develops and manages big data architectures and systems. In Databricks Data Science and Engineering: Basic Tools and Infrastructure, you will learn how to build and run big data applications, which is relevant to this career.

See salaries and explore the career path for Big Data Engineer

Data Engineer

Designing and developing big data applications is a task that Data Engineers do regularly. This course named Databricks Data Science and Engineering: Basic Tools and Infrastructure studies this very thing. Someone interested in this field would benefit from this course, as it can help them with building and running big data applications.

See salaries and explore the career path for Data Engineer

Data Architect

Designing and building big data architectures can be done by a Data Architect. The course Databricks Data Science and Engineering: Basic Tools and Infrastructure teaches about the assets provided by Databricks, including clusters, notebooks, repos, and jobs, which would be valuable knowledge in this field.

See salaries and explore the career path for Data Architect

Machine Learning Engineer

Developing machine learning algorithms means being able to harness big data, which is what the course Databricks Data Science and Engineering: Basic Tools and Infrastructure teaches. Someone interested in becoming a Machine Learning Engineer should consider taking this course to get the most out of their data.

See salaries and explore the career path for Machine Learning Engineer

Data Management Analyst

A Data Management Analyst may find use for the knowledge of Databricks' resources such as clusters, notebooks, repos, and jobs. Understanding how to build and run big data applications is an asset for someone in this field.

See salaries and explore the career path for Data Management Analyst

Quantitative Analyst

A Quantitative Analyst may be tasked with developing and implementing data-driven solutions. This course will teach the learner about building and running big data applications, a helpful skill for this role.

See salaries and explore the career path for Quantitative Analyst

Data Warehouse Architect

Designing and developing data warehouse architectures may involve using big data applications. Databricks Data Science and Engineering: Basic Tools and Infrastructure teaches how to build and run big data applications, a useful skillset for this career.

See salaries and explore the career path for Data Warehouse Architect

Data Integration Engineer

A Data Integration Engineer designs and develops data integration solutions. The course named Databricks Data Science and Engineering: Basic Tools and Infrastructure is a good place to learn more about developing data integration solutions.

See salaries and explore the career path for Data Integration Engineer

Business Analyst

Developing applications that help companies improve decision-making is often the job of a Business Analyst. Databricks Data Science and Engineering: Basic Tools and Infrastructure is a good place to learn more about how to develop applications.

See salaries and explore the career path for Business Analyst

Software Architect

As a Software Architect designs and develops software systems, they may utilize big data applications. Databricks Data Science and Engineering: Basic Tools and Infrastructure can assist the learner in developing these applications.

See salaries and explore the career path for Software Architect

Data Analyst

A Data Analyst may be responsible for designing and developing applications. The course named Databricks Data Science and Engineering: Basic Tools and Infrastructure may have information and skills helpful to someone in this field. This course could be beneficial to a person interested in this career as it teaches learners about the use of Databricks' resources such as clusters, notebooks, repos, and jobs, as well as their configurations. These ideas may be applicable in the development of data analysis applications.

See salaries and explore the career path for Data Analyst

Software Engineer

Working as a Software Engineer may require one to build and run big data applications. To build and run big data applications requires an understanding of using Databricks' resources such as clusters, notebooks, repos, and jobs. This course is an ideal place to learn this.

See salaries and explore the career path for Software Engineer

Database Administrator

Databricks Data Science and Engineering: Basic Tools and Infrastructure teaches about the valuable asset named Databricks' resources such as clusters, notebooks, repos, and jobs, as well as their configurations. A Database Administrator might benefit from the knowledge that this course provides.

See salaries and explore the career path for Database Administrator