We may earn an affiliate commission when you visit our partners.

Serverless Data Processing with Dataflow

Pipelines - 日本語版

Google Cloud Training

Dataflow コースシリーズの 2 回目である今回は、Beam SDK を使用したパイプラインの開発について詳しく説明します。まず、Apache Beam のコンセプトについて復習します。次に、ウィンドウ、ウォーターマーク、トリガーを使用したストリーミングデータの処理について説明します。さらに、パイプラインのソースとシンクのオプション、構造化データを表現するためのスキーマ、State API と Timer API を使用してステートフル変換を行う方法について説明します。続いて、パイプラインのパフォーマンスを最大化するためのベストプラクティスを再確認します。コースの終盤では、Beam でビジネスロジックを表現するための SQL と DataFrame、および Beam ノートブックを使用してパイプラインを反復的に開発する方法を説明します。

Enroll now

Or subscribe to Coursera Plus

And get unlimited access to Coursera

Two deals to help you save

Save money when you learn. Here are two deals and offers that may be relevant to this course.

All coupon codes, vouchers, and discounts are applied automatically unless otherwise noted.

Valid until July 14

Coursera Plus Annual

Work toward what's next for your career with access to 10,000+ programs. Discover emerging skills and save.

Valid until August 30

Google AI App Builder

Learn how to use Gemini API and API Studio with a three-course series from Google DeepMind

What's inside

Syllabus

はじめに

このモジュールでは、コースとその概要を紹介します

Beam のコンセプトの復習

Apache Beam の主なコンセプトと、それを独自のデータ処理パイプラインを作成するために適用する方法を復習します。

Traffic lights

Read about what's good

what should give you pause

and possible dealbreakers

Covers technologies highly relevant to the industry, such as Apache Beam

Taught by Google Cloud Training, who are recognized for their work in cloud computing training

Examines windowing, watermarks, and triggers, which are core concepts for streaming data processing

Develops skills in using both SQL and DataFrames to express business logic in Beam pipelines

Introduces Beam Notebooks, which simplifies pipeline development for Python developers within Jupyter Notebooks

Advises course participants to complete the first course in the Dataflow series before starting this one

Save this course

Create your own learning path. Save this course to your list so you can find it easily later.

Save

Reviews summary

Dataflow pipelines: concepts, practice, and optimization

According to students, this course offers a comprehensive and deep dive into serverless data processing with Dataflow and the Beam SDK. Many highlight the course's strength in making complex concepts like windowing and watermarks clear. The hands-on labs are often cited as invaluable for solidifying theory, and the best practices module is particularly praised for providing actionable advice on optimizing Dataflow jobs. While the course is generally well-structured and covers advanced topics like SQL/DataFrame and Beam Notebooks, some learners found the labs occasionally rushed or desired more immediate hands-on reinforcement. A notable point of contention is the level of assumed prior knowledge, with some finding it too advanced without clear prerequisites in Python or GCP, suggesting it's best for those with existing foundational skills.

Labs are beneficial, but some users desire more depth.

"The hands-on labs were invaluable for solidifying the theory."

"I found the sections on State and Timer API very useful, though they could have had more practical examples."

"I struggled with some of the labs. They felt a bit rushed, and the explanations in the lab instructions weren't always as detailed as the lecture."

Offers valuable insights for pipeline performance.

"The best practices module alone was worth the price of admission. I've been struggling with optimizing my Dataflow jobs, and this course provided clear, actionable advice."

"Learned so much about optimizing Dataflow pipelines! The focus on best practices and advanced topics... was exactly what I needed."

Complex concepts are explained with notable clarity.

"The modules on windowing and watermarks were particularly clear and made complex concepts understandable."

"The instructors explain the core concepts well, especially the IO connectors."

"The explanations of windowing and triggers were particularly helpful."

Provides a thorough and deep understanding of Dataflow.

"This course provided an excellent deep dive into Dataflow pipelines with Beam SDK."

"A very thorough course covering most aspects of Dataflow."

"Solid coverage of Dataflow essentials. I appreciated the structured approach."

Requires existing knowledge of Python and GCP.

"Prior experience with Python and GCP is definitely a plus, as it's not explicitly stated as a hard prerequisite."

"This course was too advanced for me. I was expecting a more beginner-friendly approach to Dataflow, and the prerequisites weren't clear enough."

"I found myself lost in the technical jargon and complex examples. This course is not suitable if you're new to cloud data processing or Python."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Serverless Data Processing with Dataflow: Pipelines - 日本語版 with these activities:

Organize and Review Course Materials

Show steps

Stay organized and enhance your learning by compiling and reviewing course materials regularly

Show steps

Create a system for organizing notes, assignments, and other materials
Regularly review and summarize key concepts covered in the course
Annotate materials to highlight important information and connections

Review Python for Apache Beam

Show steps

Ensure a solid footing in Python programming before the course begins to make the most of the content

Browse courses on Apache Beam

Show steps

Read through documentation on Python basics
Complete coding exercises to practice syntax and core concepts
Review online resources for best practices in Python programming
Build a small Python project to apply your understanding

Learn about Streaming Data Processing with Apache Beam

Show steps

Explore guided tutorials to familiarize yourself with the concepts of streaming data processing and Apache Beam

Browse courses on Streaming Data Processing

Show steps

Follow online tutorials on streaming data processing with Apache Beam
Work through examples provided in the official documentation
Connect with online communities and forums for discussions and support

Two other activities

Expand to see all activities and additional details

Show all five activities

Build a Simple Data Transformation Pipeline

Show steps

Apply your understanding by building a simple data transformation pipeline using Apache Beam

Show steps

Design a data transformation pipeline that meets a specific need
Implement the pipeline using Apache Beam SDK
Test and validate the pipeline's functionality
Deploy the pipeline and monitor its performance

Develop a Data Processing Dashboard

Show steps

Demonstrate your skills by creating a data processing dashboard using Apache Beam

Show steps

Gather requirements and design the dashboard
Develop the dashboard using Apache Beam and relevant tools
Integrate the dashboard with data sources and pipelines
Test and validate the dashboard's functionality
Present the dashboard to stakeholders

Career center

Learners who complete Serverless Data Processing with Dataflow: Pipelines - 日本語版 will develop knowledge and skills that may be useful to these careers:

Data Engineer

A Data Engineer designs, builds, and maintains the infrastructure and tools used to store, process, and analyze data. Using this course, you can build serverless data processing pipelines that will help you make a smooth transition into this role. The course also covers performance optimization using best practices to maximize your work as a Data Engineer.

See salaries and explore the career path for Data Engineer

Data Analyst

A Data Analyst collects, analyzes, and interprets data to identify trends and patterns. Once you are a Data Analyst, you can use the learning from this course, such as data processing pipeline development using Beam SDK, to collect streaming data and use SQL to express your business logic.

See salaries and explore the career path for Data Analyst

Software Engineer

A Software Engineer designs, develops, and maintains software applications. As a Software Engineer, you would be responsible for writing code. The coding concepts you will learn from this course will help you build a great foundation.

See salaries and explore the career path for Software Engineer

Data Scientist

A Data Scientist is responsible for developing and implementing machine learning and statistical models to solve business problems. This course covers using Dataflow SQL to express business logic, which will be helpful for you as a Data Scientist.

See salaries and explore the career path for Data Scientist

Business Analyst

A Business Analyst identifies and solves business problems through data analysis. In this role, you would need to understand how to develop data processing pipelines, which is covered in this course.

See salaries and explore the career path for Business Analyst

Machine Learning Engineer

A Machine Learning Engineer develops and deploys machine learning models to solve business problems. In this role, you would develop data processing pipelines to prepare data for machine learning models.

See salaries and explore the career path for Machine Learning Engineer

Cloud Architect

A Cloud Architect designs and manages cloud computing solutions. In this role, you would need to understand how to build serverless data processing pipelines, which is covered in this course.

See salaries and explore the career path for Cloud Architect

Data Architect

A Data Architect designs and manages data architectures. This course may be helpful in understanding how to design data processing pipelines.

See salaries and explore the career path for Data Architect

Database Administrator

A Database Administrator manages and maintains databases. This course may be helpful in understanding how to build data processing pipelines that interact with databases.

See salaries and explore the career path for Database Administrator

Systems Engineer

A Systems Engineer designs, implements, and maintains computer systems. This course may be helpful in understanding how to build data processing pipelines.

See salaries and explore the career path for Systems Engineer

Network Engineer

A Network Engineer designs, implements, and maintains computer networks. This course may be helpful in understanding how to build data processing pipelines that use cloud networking services.

See salaries and explore the career path for Network Engineer

Security Engineer

A Security Engineer designs, implements, and maintains security systems. This course may be helpful in understanding how to build data processing pipelines that are secure.

See salaries and explore the career path for Security Engineer

Cloud Security Engineer

A Cloud Security Engineer designs, implements, and maintains security systems for cloud computing environments. This course may be helpful in understanding how to build data processing pipelines that are secure in the cloud.

See salaries and explore the career path for Cloud Security Engineer

DevOps Engineer

A DevOps Engineer works with developers to build and maintain software systems. This course may be helpful in understanding how to build data processing pipelines that are integrated with continuous integration and continuous delivery (CI/CD) systems.

See salaries and explore the career path for DevOps Engineer

Quality Assurance Engineer

A Quality Assurance Engineer tests and verifies the quality of software systems. This course may be helpful in understanding how to build data processing pipelines that are reliable and error-free.

See salaries and explore the career path for Quality Assurance Engineer

Reading list

We've selected seven books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Serverless Data Processing with Dataflow: Pipelines - 日本語版.

Hadoop: The Definitive Guide

Save

Provides a comprehensive overview of Hadoop and how it can be used for big data processing.

Hadoop: The Definitive Guide: Storage and Analysis...

Paperback

Hadoop: The Definitive Guide

Kindle Edition

Hadoop: The Definitive Guide

Paperback

Hadoop: The Definitive Guide

Paperback

Architecting Data and Machine Learning Platforms

Save

Provides a comprehensive overview of the Google Cloud Platform and how it can be used for data engineering tasks.

Architecting Data and Machine Learning Platforms

Paperback

Check price

Architecting Data and Machine Learning Platforms

Kindle Edition

Check price

Kimball's Data Warehouse Toolkit Classics, 3 Volume...

Save

Provides a comprehensive overview of data warehousing and how to design and build a data warehouse.

Kimball's Data Warehouse Toolkit Classics, 3 Volume...

Paperback

$$$$

NoSQL Distilled

Save

Provides a comprehensive overview of NoSQL databases and how to choose the right NoSQL database for your needs.

NoSQL Distilled: A Brief Guide to the Emerging...

Kindle Edition

Secret Colors

Save

Provides a comprehensive overview of the challenges and patterns involved in designing data-intensive applications. It covers topics such as data modeling, data storage, data processing, and data analytics.