Serverless Data Processing with Dataflow: Develop Pipelines from Coursera

What's inside

Syllabus

Introduction

This module covers the course outline

Beam Concepts Review

Review main concepts of Apache Beam, and how to apply them to write your own data processing pipelines.

Windows, Watermarks Triggers

In this module, you will learn about how to process data in streaming with Dataflow. For that, there are three main concepts that you need to learn: how to group data in windows, the importance of watermark to know when the window is ready to produce results, and how you can control when and how many times the window will emit output.

Sources & Sinks

In this module, you will learn about what makes sources and sinks in Google Cloud Dataflow. The module will go over some examples of Text IO, FileIO, BigQueryIO, PubSub IO, KafKa IO, BigTable IO, Avro IO, and Splittable DoFn. The module will also point out some useful features associated with each IO.

Schemas

This module will introduce schemas, which give developers a way to express structured data in their Beam pipelines.

State and Timers

This module covers State and Timers, two powerful features that you can use in your DoFn to implement stateful transformations.

Best Practices

This module will discuss best practices and review common patterns that maximize performance for your Dataflow pipelines.

Dataflow SQL & DataFrames

This modules introduces two new APIs to represent your business logic in Beam: SQL and Dataframes.

Beam Notebooks

This module will cover Beam notebooks, an interface for Python developers to onboard onto the Beam SDK and develop their pipelines iteratively in a Jupyter notebook environment.

Summary

This module provides a recap of the course

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Suitable for learners with knowledge of Apache Beam

Designed for intermediate learners who want to advance their Dataflow skills

Should be taken after the introductory Dataflow course

Could be part of a comprehensive curriculum on Apache Beam

Reviews summary

Dataflow pipeline development

Learners say Serverless Data Processing with Dataflow: Develop Pipelines is a good place to learn about Google Dataflow, Apache Beam, and data pipelines. Students particularly like the course's hands-on labs and engaging assignments. While some reviewers mention that the course is not trivial, many commend the course's focus on both batch and streaming pipelines. However, there are some complaints that the course is sometimes difficult to understand due to the instructors' speech patterns.

Students recommend this course to others.

"Good for an advance engineers"

"Good place to learn Dataflow and Apache Beam."

"Excellent course focus on Batch and Streaming Pipelines using Google Dataflow"

Students like the course's engaging, hands-on labs.

"Liked the hands-on labs."

"It is a good course but it is not trivial"

"This course gives a great overview of the basic building blocks of Apache Beam as well as offers an opportunity to get your hands dirty and use these building blocks to build real data pipelines."

Students sometimes found the instructors difficult to understand.

"people from India (not native english speakers) often speak illegibly and it is difficult (sometimes impossible) to understand them."

"subtitles don't help"

Many students found the material to be challenging.

"Too hard, insufficient signposting"

"it is difficult (sometimes impossible) to understand them"

"some code examples are in java only"

Career center

Learners who complete Serverless Data Processing with Dataflow: Develop Pipelines will develop knowledge and skills that may be useful to these careers:

Data Engineer

A Data Engineer builds and maintains data pipelines that process structured and unstructured data. As a Data Engineer, you would use a course like this one to gain a deeper understanding of Apache Beam concepts and how to apply them to write your own data processing pipelines. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for Data Engineer

Data Analyst

A Data Analyst cleans, analyzes, and interprets data to identify trends and patterns. As a Data Analyst, you might use a course like this one to gain a deeper understanding of how to process streaming data using windows, watermarks, and triggers. You would also learn about options for sources and sinks in your pipelines, schemas to express your structured data, and how to do stateful transformations using State and Timer APIs.

See salaries and explore the career path for Data Analyst

Data Scientist

A Data Scientist builds and applies mathematical and statistical models to data to extract insights and make predictions. As a Data Scientist, you might use a course like this one to gain a deeper understanding of how to process streaming data using windows, watermarks, and triggers. You would also learn about options for sources and sinks in your pipelines, schemas to express your structured data, and how to do stateful transformations using State and Timer APIs.

See salaries and explore the career path for Data Scientist

Software Engineer

A Software Engineer designs, develops, and maintains software applications. As a Software Engineer, you might use a course like this one to gain a deeper understanding of Apache Beam concepts and how to apply them to write your own data processing pipelines. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for Software Engineer

DevOps Engineer

A DevOps Engineer automates and manages the software development and deployment process. As a DevOps Engineer, you might use a course like this one to gain a deeper understanding of how to build and maintain data pipelines. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for DevOps Engineer

Cloud Architect

A Cloud Architect designs and manages cloud computing solutions. As a Cloud Architect, you might use a course like this one to gain a deeper understanding of how to build and maintain data pipelines in the cloud. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for Cloud Architect

Data Integration Engineer

A Data Integration Engineer designs and builds data pipelines that integrate data from multiple sources. As a Data Integration Engineer, you would use a course like this one to gain a deeper understanding of Apache Beam concepts and how to apply them to write your own data processing pipelines. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for Data Integration Engineer

Big Data Engineer

A Big Data Engineer designs and builds data pipelines that process large volumes of data. As a Big Data Engineer, you would use a course like this one to gain a deeper understanding of Apache Beam concepts and how to apply them to write your own data processing pipelines. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for Big Data Engineer

Machine Learning Engineer

A Machine Learning Engineer designs, develops, and deploys machine learning models. As a Machine Learning Engineer, you might use a course like this one to gain a deeper understanding of how to process streaming data using windows, watermarks, and triggers. You would also learn about options for sources and sinks in your pipelines, schemas to express your structured data, and how to do stateful transformations using State and Timer APIs.

See salaries and explore the career path for Machine Learning Engineer

Business Intelligence Analyst

A Business Intelligence Analyst analyzes data to identify trends and patterns that can help businesses make better decisions. As a Business Intelligence Analyst, you might use a course like this one to gain a deeper understanding of how to process streaming data using windows, watermarks, and triggers. You would also learn about options for sources and sinks in your pipelines, schemas to express your structured data, and how to do stateful transformations using State and Timer APIs.

See salaries and explore the career path for Business Intelligence Analyst

Data Visualization Engineer

A Data Visualization Engineer designs and develops data visualizations that help people understand data. As a Data Visualization Engineer, you might use a course like this one to gain a deeper understanding of how to process streaming data using windows, watermarks, and triggers. You would also learn about options for sources and sinks in your pipelines, schemas to express your structured data, and how to do stateful transformations using State and Timer APIs.

See salaries and explore the career path for Data Visualization Engineer

Database Administrator

A Database Administrator manages and maintains databases. As a Database Administrator, you might use a course like this one to gain a deeper understanding of how to build and maintain data pipelines. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for Database Administrator

IT Manager

An IT Manager plans, organizes, and directs the implementation and maintenance of computer systems and networks. As an IT Manager, you might use a course like this one to gain a deeper understanding of how to build and maintain data pipelines. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for IT Manager

Systems Administrator

A Systems Administrator manages and maintains computer systems and networks. As a Systems Administrator, you might use a course like this one to gain a deeper understanding of how to build and maintain data pipelines. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for Systems Administrator

Network Administrator

A Network Administrator manages and maintains computer networks. As a Network Administrator, you might use a course like this one to gain a deeper understanding of how to build and maintain data pipelines. You would also learn about best practices for maximizing the performance of your pipelines.

See salaries and explore the career path for Network Administrator