Serverless Data Processing with Dataflow: Develop Pipelines em Português Brasileiro from Coursera

What's inside

Syllabus

Introduçao

Este módulo é uma introdução ao curso e ao conteúdo dele.

Resumo dos conceitos do Beam

Confira os principais conceitos do Apache Beam e como aplicá-los na criação dos seus próprios pipelines de processamento de dados.

Janelas, gatilhos de marcas d'água

Neste módulo, você aprenderá a processar dados em streaming com o Dataflow. Para fazer isso, você precisa entender três conceitos principais: como agrupar dados em janelas, a importância das marcas d’água para saber quando a janela está pronta para oferecer resultados e como definir quantas vezes a janela emitirá respostas e a frequência desse processo.

Origens e coletores

Neste módulo, você aprenderá sobre as origens e os coletores no Google Cloud Dataflow. Mostraremos alguns exemplos de DoFn divisível e de E/S de texto, arquivos, BigQuery, Pub/Sub, Kafka, BigTable e Avro. Além disso, mostraremos alguns recursos úteis associados a cada E/S.

Esquemas

Neste módulo, apresentaremos esquemas que são usados por desenvolvedores para expressar dados estruturados nos pipelines do Beam.

Estado e Timers

Neste módulo, falaremos sobre estado e timers, dois recursos avançados que você pode usar na DoFn para implementar transformações com estado.

Práticas Recomendadas

Neste módulo, falaremos sobre práticas recomendadas e padrões comuns que maximizam o desempenho dos seus pipelines do Dataflow.

Dataflow SQL e DataFrames

Neste módulo, apresentaremos duas novas APIs que representam sua lógica de negócios no Beam: SQL e DataFrames.

Notebooks do Beam

Este módulo é sobre os notebooks do Beam, uma interface para que os desenvolvedores que usam Python comecem a adotar o SDK da plataforma. Isso pode ser feito para criar pipelines de forma iterativa em um ambiente de notebooks do Jupyter.

Resumo

Este módulo é uma recapitulação do curso.

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Desenvolve habilidades avançadas, como estado e temporizadores, que são essenciais para pipelines de processamento de dados do mundo real

Oferece práticas recomendadas e padrões para otimizar o desempenho do pipeline

Apresenta SQL e DataFrames, opções poderosas para representar a lógica de negócios em pipelines do Beam

Inclui notebooks de Beam para desenvolvimento iterativo de pipelines em um ambiente conveniente

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Serverless Data Processing with Dataflow: Develop Pipelines em Português Brasileiro with these activities:

Review Data Processing Concepts

Show steps

Comprehensively review the core concepts of data processing prior to immersing yourself in the course content to establish a stronger basis for learning and understanding.

Browse courses on Data Analysis

Show steps

Review fundamental principles of data processing
Explore different data processing techniques and algorithms
Practice data manipulation and transformation

Practice Beam Pipeline Development

Show steps

Engage in hands-on exercises to develop Beam pipelines, strengthening your understanding of pipeline construction, execution, and optimization.

Browse courses on Data Processing Pipelines

Show steps

Follow guided tutorials on Beam pipeline development
Create simple pipelines to process sample datasets
Experiment with different pipeline configurations

Show all two activities

Career center

Learners who complete Serverless Data Processing with Dataflow: Develop Pipelines em Português Brasileiro will develop knowledge and skills that may be useful to these careers:

Software Developer

Develop pipelines by using the Beam SDK with the Dataflow course from Google Cloud. This course covers a broad range of topics that will help you in your career as a Software Developer. Throughout the course, you will learn how to process streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames. Additionally, this course will introduce you to state and timers. These are two advanced features that you can use in DoFn to implement transformations with states. This course requires some familiarity with coding.

See salaries and explore the career path for Software Developer

Data Analyst

The Dataflow course from Google Cloud can help you to advance your career as a Data Analyst. It covers advanced concepts like state and timers. These are two advanced features that you can use in DoFn to implement transformations with states. You will also learn about processing streaming data, sources and sinks, schemas, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Data Analyst

Data Engineer

The Dataflow course from Google Cloud may be useful if you want to work as a Data Engineer. This course provides more information on developing pipelines by using the Beam SDK. You will learn about a variety of topics, including processing streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Data Engineer

Business Analyst

The Dataflow course from Google Cloud may be useful for those looking to become a Business Analyst. As you go through this course, you will learn how to develop pipelines by using the Beam SDK. You will also learn about processing streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Business Analyst

Data Scientist

For those looking to become a Data Scientist, taking the Dataflow course from Google Cloud may be helpful. This course covers a variety of topics that will help you in your career, including developing pipelines by using the Beam SDK, processing streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Data Scientist

Software Engineer

If you are interested in becoming a Software Engineer, the Dataflow course from Google Cloud may be useful. As you progress through the course, you will learn how to develop pipelines by using the Beam SDK and process streaming data. You will also learn about sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Software Engineer

Cloud Architect

For those looking to become a Cloud Architect, the Dataflow course from Google Cloud may be helpful.This course provides more information on developing pipelines by using the Beam SDK. You will also learn about processing streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Cloud Architect

Backend Developer

The Dataflow course from Google Cloud may be useful to those looking to become a Backend Developer. You will learn to process streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames as you progress through the course.

See salaries and explore the career path for Backend Developer

Full-Stack Developer

Those looking to become a Full Stack Developer may find the Dataflow course from Google Cloud helpful. Throughout the course, you will learn about developing pipelines by using the Beam SDK. You will also learn how to process streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Full-Stack Developer

Database Administrator

If you are interested in becoming a Database Administrator, the Dataflow course from Google Cloud may be useful. This course covers a range of topics, including developing pipelines by using the Beam SDK, processing streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Database Administrator

Data Warehouse Engineer

For those looking to become a Data Warehouse Engineer, the Dataflow course from Google Cloud may be helpful. You will learn how to develop pipelines by using the Beam SDK as you progress through the course. You will also learn about processing streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Data Warehouse Engineer

Systems Engineer

The Dataflow course from Google Cloud may be useful to those looking to become a Systems Engineer. You will learn to process streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames as you progress through the course.

See salaries and explore the career path for Systems Engineer

Network Engineer

The Dataflow course from Google Cloud may be useful for those looking to become a Network Engineer. As you progress through the course, you will learn how to develop pipelines by using the Beam SDK. You will also learn about processing streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Network Engineer

IT Manager

The Dataflow course from Google Cloud may be useful to those looking to become a IT Manager. You will learn to process streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames as you progress through the course.

See salaries and explore the career path for IT Manager

Information Security Analyst

For those looking to become a Information Security Analyst, the Dataflow course from Google Cloud may be helpful. This course provides more information on developing pipelines by using the Beam SDK. You will also learn about processing streaming data, sources and sinks, schemas, state and timers, best practices, and Dataflow SQL and DataFrames.

See salaries and explore the career path for Information Security Analyst