Building Pipelines for Workflow Orchestration Using Google Composer from Pluralsight

Cloud Composer is a fully managed workflow orchestration service that allows creation, scheduling, and monitoring of pipelines. This course covers the architecture of Composer and creation of pipelines to run arbitrary shell scripts and Python code.

Cloud Composer is a pipeline orchestration service on the GCP. Based on the Apache Airflow API, Composer was launched in May 2018 and is fast-emerging as a popular and versatile service for building and executing system pipelines. In this course, Building Pipelines for Workflow Orchestration Using Google Composer, you'll learn how Composer allows cloud users to quickly create pipelines with complex interconnected tasks. First, you'll discover where Composer fits in the taxonomy of GCP services and how it compares to Dataflow, which is another service for building and executing pipelines on the GCP. Next, you'll explore what a Composer environment is and how pipelines are specified, and run on these environments. Then, you'll develop an understanding of the powerful suite of operators made available for use within Composer pipelines by utilizing Airflow operators for executing shell scripts, executing arbitrary Python code, and implementing complex control flow. Finally, you'll learn how to use Airflow’s GCP-specific operators for sending email, working with BigQuery, and instantiating Dataproc clusters. When you're finished with this course, you'll have the skills and knowledge necessary to build and deploy complex pipelines built on the Apache Airflow API by utilizing Composer.

What's inside

Syllabus

Course Overview

Introducing Google Composer

Creating, Configuring, and Accessing Environments

Managing and Monitoring Workflows

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Develops skills and knowledge in pipelining and workflow orchestration, which are core skills for data engineering

Taught by Vitthal Srinivasan, who is recognized for their work in data engineering

Examines Cloud Composer and Apache Airflow API, which are highly relevant to workflow orchestration

Teaches pipeline building, a skill useful for personal growth and development

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Building Pipelines for Workflow Orchestration Using Google Composer with these activities:

Engage in Discussion Forums and Online Communities

Show steps

Connect with fellow learners and practitioners to exchange knowledge, ask questions, and gain diverse perspectives.

Show steps

Identify relevant discussion forums and online communities
Participate in discussions, ask questions, and share insights

Build a Real-World Pipeline with Composer

Show steps

Solidify your skills by designing and implementing a complete pipeline that addresses a real-world problem.

Browse courses on Workflow Orchestration

Show steps

Identify a suitable problem domain
Design the pipeline architecture
Develop the pipeline code in Composer
Test and deploy the pipeline

Contribute to the Apache Airflow Project

Show steps

Deepen your understanding of Airflow by actively participating in its development and contributing to the open source community.

Browse courses on Apache Airflow

Show steps

Review the Apache Airflow documentation and codebase
Identify areas for potential contributions
Create and submit a pull request

Show all three activities

Career center

Learners who complete Building Pipelines for Workflow Orchestration Using Google Composer will develop knowledge and skills that may be useful to these careers:

Project Manager

Project Managers plan, execute, and close projects to achieve specific goals and objectives. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for project management and tracking.

See salaries and explore the career path for Project Manager

Data Scientist

Data Scientists apply scientific methods and expertise to extract insights from data to solve business problems and opportunities. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for data analysis and modeling.

See salaries and explore the career path for Data Scientist

Machine Learning Engineer

Machine Learning Engineers design, develop, and maintain machine learning models for various applications. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for machine learning training and deployment.

See salaries and explore the career path for Machine Learning Engineer

Business Analyst

Business Analysts identify, analyze, and solve business problems by leveraging data and technology. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for data analysis and modeling.

See salaries and explore the career path for Business Analyst

Data Architect

Data Architects design and manage data architectures for organizations. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for data architecture and management.

See salaries and explore the career path for Data Architect

Data Analyst

Data Analysts collect, analyze, and interpret data to help organizations make informed decisions. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for data analysis and reporting.

See salaries and explore the career path for Data Analyst

Product Manager

Product Managers define, develop, and launch products to meet customer needs and business objectives. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for product development and management.

See salaries and explore the career path for Product Manager

Data Engineer

Data Engineers analyze, transform, and manage large and complex data sets to support data-driven decision-making within an organization. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy complex pipelines for data processing and management.

See salaries and explore the career path for Data Engineer

Software Engineer - Big Data

Software Engineers (Big Data) design, develop, and maintain software systems for processing and managing large data sets. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for big data processing and management.

See salaries and explore the career path for Software Engineer - Big Data

Cloud Architect

Cloud Architects design, develop, and manage cloud computing solutions for organizations. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for cloud-based data processing and management.

See salaries and explore the career path for Cloud Architect

DevOps Engineer

DevOps Engineers combine software development and IT operations to improve the speed and quality of software delivery. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for continuous integration and delivery.

See salaries and explore the career path for DevOps Engineer

Data Governance Specialist

Data Governance Specialists establish and maintain data governance policies and procedures within an organization. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for data governance and compliance.

See salaries and explore the career path for Data Governance Specialist

Data Integration Specialist

Data Integration Specialists design, develop, and maintain data integration solutions for organizations. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for data integration and consolidation.

See salaries and explore the career path for Data Integration Specialist

Software Engineer (Data Analytics)

Software Engineers (Data Analytics) design, develop, and maintain software systems for data analysis and reporting. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for data analysis and modeling.

See salaries and explore the career path for Software Engineer (Data Analytics)

Data Scientist (Machine Learning)

Data Scientists (Machine Learning) apply machine learning techniques to solve business problems and opportunities. Building Pipelines for Workflow Orchestration Using Google Composer may be useful for this career as it provides an understanding of how to build and deploy pipelines for machine learning training and deployment.

See salaries and explore the career path for Data Scientist (Machine Learning)

Reading list

We've selected six books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Building Pipelines for Workflow Orchestration Using Google Composer.

Python for Data Analysis

Save

Provides a solid foundation in Python programming for data analysis, which is essential for writing custom code and operators within Composer pipelines.

Python for Data Analysis

Paperback

Check price

Python for Data Analysis

Kindle Edition

Check price

Data Pipelines with Apache Airflow

Save

Explores the benefits and use cases of deploying Apache Airflow as a managed service in the cloud.

Data Pipelines with Apache Airflow

Paperback

Kimball's Data Warehouse Toolkit Classics, 3 Volume...

Save

Provides a comprehensive guide to designing and implementing data warehouses, which are often used as targets or sources for data pipelines. It offers a solid foundation in data modeling, ETL processes, and performance optimization, complementing the focus on pipeline orchestration in the course.

Kimball's Data Warehouse Toolkit Classics, 3 Volume...

Paperback

$$$$

Advanced Python Programming

Save

Provides guidance on optimizing Python code for performance, which is important for ensuring efficient and scalable pipelines in Composer.

Advanced Python Programming: Build high performance...

Kindle Edition

Python Data Science Handbook

Save

Offers a comprehensive guide to data science using Python, covering topics such as data exploration, machine learning, and visualization. While not specifically focused on pipeline orchestration, it provides a valuable background for understanding the data science context.

Python Data Science Handbook: Essential Tools for...

Paperback

$$$

Secret Colors

Save

Provides a comprehensive guide to designing and building data-intensive applications, covering topics such as data modeling, data storage, and data processing. While not specifically focused on pipeline orchestration, it offers a valuable foundation for understanding the principles and challenges of managing large-scale data.

Building Pipelines for Workflow Orchestration Using Google Composer

Here's a deal for you

What's inside

Syllabus

Good to know

Save this course

Activities

Career center

Reading list

Share

Similar courses