Building Data Pipelines with Luigi 3 and Python from Pluralsight

Other developers implement data pipelines by putting together a bunch of hacky scripts, that over time turn into liabilities and maintenance nightmares. Take this course to implement sane and smart data pipelines with Luigi in Python.

Data arrives from various sources and needs further processing. It's very tempting to re-invent the wheel and write your own library to build data pipelines for batch processing. This results in data pipelines that are difficult to maintain. In this course, Building Data Pipelines with Luigi and Python, you’ll learn how to build data pipelines with Luigi and Python. First, you’ll explore how to build your first data pipelines with Luigi. Next, you’ll discover how to configure Luigi pipelines. Finally, you’ll learn how to run Luigi pipelines. When you’re finished with this course, you’ll have the Luigi skills and knowledge for building data pipelines that are easy to maintain.

A data pipeline is a series of data processing steps. Data pipelines consist of three components: a source, a processing step or steps, and a destination.

Prerequisites for this course are fluency within Python and familiarity with linux command line.

Luigi is a package within Python that helps you build complex pipelines of data intense jobs. Luigi handles dependency resolution, workflow management, visualization, handling failures, and command line integration.

Some benefits of Python are: easy to read, learn, and write, open-source, portable, dynamically typed, and provides extensive support libraries.

Data pipelines are primarily used to automate the process of extracting, transforming, and loading data.

What's inside

Syllabus

Course Overview

Getting Started with Luigi

Building Luigi Pipelines

Configuring Luigi Pipelines

Running Luigi Pipelines

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Builds strong foundational skills in data pipelines with Python and Luigi

Taught by recognized instructor Dan Tofan

Covers essential concepts of data pipelines, including sources, processing, and destinations

Emphasizes practical implementation of data pipelines, with a focus on maintenance and scalability

Assumes familiarity with Python and Linux command line

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Building Data Pipelines with Luigi 3 and Python with these activities:

Review Core Python Syntax

Show steps

Familiarize yourself with core Python syntax and concepts to build a strong foundation for the course.

Browse courses on Python Syntax

Show steps

Review documentation on Python data types, variables, and operators.
Practice writing simple Python statements and expressions.

Show all one activities

Career center

Learners who complete Building Data Pipelines with Luigi 3 and Python will develop knowledge and skills that may be useful to these careers:

Data Engineer

Data Engineers build and maintain the infrastructure and tools that make it possible to collect, store, and analyze data. This course will help you build a foundation in the skills and knowledge needed to be a successful Data Engineer. You will learn how to use Luigi to build data pipelines that are easy to maintain. This will give you a competitive edge in the job market and help you succeed in your Data Engineering career.

See salaries and explore the career path for Data Engineer

Data Analyst

Data Analysts use data to help businesses make better decisions. They collect, clean, and analyze data to identify trends and patterns. This course will help you build the skills and knowledge needed to be a successful Data Analyst. You will learn how to use Luigi to build data pipelines that are easy to maintain. This will give you a competitive edge in the job market and help you succeed in your Data Analyst career.

See salaries and explore the career path for Data Analyst

Data Scientist

Data Scientists use data to solve business problems. They develop and deploy machine learning models to predict future outcomes and make recommendations. This course will help you build the skills and knowledge needed to be a successful Data Scientist. You will learn how to use Luigi to build data pipelines that are easy to maintain. This will give you a competitive edge in the job market and help you succeed in your Data Scientist career.

See salaries and explore the career path for Data Scientist

Software Engineer

Software Engineers design, develop, and maintain software systems. They use their technical skills to solve problems and create innovative solutions. This course will help you build the skills and knowledge needed to be a successful Software Engineer. You will learn how to use Luigi to build data pipelines that are easy to maintain. This will give you a competitive edge in the job market and help you succeed in your Software Engineer career.

See salaries and explore the career path for Software Engineer

Cloud Engineer

Cloud Engineers design, build, and maintain cloud computing systems. They use their technical skills to manage and optimize cloud resources. This course will help you build the skills and knowledge needed to be a successful Cloud Engineer. You will learn how to use Luigi to build data pipelines that are easy to maintain. This will give you a competitive edge in the job market and help you succeed in your Cloud Engineer career.

See salaries and explore the career path for Cloud Engineer

DevOps Engineer

DevOps Engineers bridge the gap between development and operations teams. They use their technical skills to automate and streamline the software development lifecycle. This course will help you build the skills and knowledge needed to be a successful DevOps Engineer. You will learn how to use Luigi to build data pipelines that are easy to maintain. This will give you a competitive edge in the job market and help you succeed in your DevOps Engineer career.

See salaries and explore the career path for DevOps Engineer

Data Architect

Data Architects design and manage data systems. They use their technical skills to ensure that data is accessible, reliable, and secure. This course will help you build the skills and knowledge needed to be a successful Data Architect. You will learn how to use Luigi to build data pipelines that are easy to maintain. This will give you a competitive edge in the job market and help you succeed in your Data Architect career.

See salaries and explore the career path for Data Architect

Business Analyst

Business Analysts use data to help businesses make better decisions. They work with stakeholders to understand their needs and develop solutions that meet those needs. This course will help you build the skills and knowledge needed to be a successful Business Analyst. You will learn how to use Luigi to build data pipelines that are easy to maintain. This will give you a competitive edge in the job market and help you succeed in your Business Analyst career.

See salaries and explore the career path for Business Analyst

Project Manager

Project Managers plan, execute, and close projects. They use their skills to manage resources, track progress, and ensure that projects are completed on time and within budget. This course may be useful for Project Managers who want to learn how to use Luigi to build data pipelines. This will help you to manage data-related projects more effectively.

See salaries and explore the career path for Project Manager

Product Manager

Product Managers are responsible for the development and launch of new products. They work with engineers, designers, and marketers to bring products to market. This course may be useful for Product Managers who want to learn how to use Luigi to build data pipelines. This will help you to understand the data needs of your products and make informed decisions about how to collect and use data.

See salaries and explore the career path for Product Manager

Marketing Manager

Marketing Managers are responsible for developing and executing marketing campaigns. They use their skills to attract and retain customers. This course may be useful for Marketing Managers who want to learn how to use Luigi to build data pipelines. This will help you to track the effectiveness of your marketing campaigns and make informed decisions about how to allocate your marketing budget.

See salaries and explore the career path for Marketing Manager

Sales Manager

Sales Managers are responsible for leading and motivating sales teams. They use their skills to close deals and generate revenue. This course may be useful for Sales Managers who want to learn how to use Luigi to build data pipelines. This will help you to track the progress of your sales team and make informed decisions about how to allocate your sales resources.

See salaries and explore the career path for Sales Manager

Customer Success Manager

Customer Success Managers are responsible for ensuring that customers are satisfied with their products and services. They use their skills to build relationships with customers and resolve their issues. This course may be useful for Customer Success Managers who want to learn how to use Luigi to build data pipelines. This will help you to track the satisfaction of your customers and make informed decisions about how to improve your products and services.

See salaries and explore the career path for Customer Success Manager

Data Quality Analyst

Data Quality Analysts are responsible for ensuring that data is accurate, complete, and consistent. They use their skills to identify and correct data errors. This course may be useful for Data Quality Analysts who want to learn how to use Luigi to build data pipelines. This will help you to automate the process of data quality control and improve the quality of your data.

See salaries and explore the career path for Data Quality Analyst

Data Governance Analyst

Data Governance Analysts are responsible for developing and implementing data governance policies and procedures. They use their skills to ensure that data is used in a compliant and ethical manner. This course may be useful for Data Governance Analysts who want to learn how to use Luigi to build data pipelines. This will help you to automate the process of data governance and ensure that your data is used in a compliant and ethical manner.

See salaries and explore the career path for Data Governance Analyst