Serverless Data Processing with Dataflow: Operations em Português Brasileiro from Coursera

What's inside

Syllabus

Introduction

Este módulo mostra o resumo do curso

Monitoring

Neste módulo, aprendemos a usar a página Lista de jobs para filtrar os jobs que queremos monitorar ou investigar. Vimos como as guias Gráfico de job, Informações do job e Métricas do job mostram um resumo abrangente do job do Dataflow. Por fim, aprendemos a usar a integração do Dataflow com o Metrics Explorer e criar políticas de alertas para as métricas do Dataflow.

Logging e Error Reporting

Neste módulo, aprendemos a usar o painel Registro na parte inferior das páginas Gráfico de job e Métricas do job e mostramos a página centralizada Error Reporting.

Solução de problemas e depuração

Neste módulo, vamos aprender a resolver problemas e depurar pipelines do Dataflow. Também vamos analisar os quatro modos comuns de falha do Dataflow: ao criar o pipeline, ao iniciar o pipeline no Dataflow, durante execução do pipeline e problemas de desempenho.

Desempenho

Neste módulo, veremos as considerações de desempenho que devemos saber ao desenvolver pipelines em lote e de streaming no Dataflow.

Testes e CI/CD

Este módulo vai mostrar o teste de unidade dos pipelines do Dataflow. Também vamos abordar os frameworks e os recursos disponíveis para otimizar o fluxo de trabalho de CI/CD dos pipelines do Dataflow.

Confiabilidade

Neste módulo, veremos os métodos para criar sistemas resilientes a dados corrompidos e falhas temporárias do data center.

Modelos Flex

Este módulo aborda os modelos Flex, um recurso que ajuda as equipes de engenharia de dados a padronizar e reutilizar o código de pipeline do Dataflow. Vários desafios operacionais podem ser resolvidos com esses modelos.

Summary

Este módulo faz uma revisão dos temas abordados no curso.

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Analisa problemas e soluções comuns para criar pipelines de dados confiáveis e com desempenho otimizado

Descreve técnicas para monitorar e solucionar problemas no Google Dataflow

Aborda modelos Flex, permitindo a padronização e reutilização de código de pipeline do Dataflow

Indicado para profissionais com experiência em desenvolvimento de pipelines do Dataflow

Oferece práticas recomendadas de teste, implantação e confiabilidade para pipelines do Dataflow

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Serverless Data Processing with Dataflow: Operations em Português Brasileiro with these activities:

Revise Apache Beam Pipeline Components

Show steps

Review the concepts of Apache Beam pipelines. It will help you comprehend the operational components better.

Browse courses on Apache Beam

Show steps

Go through the documentation of Apache Beam Pipeline Components.
Refer to tutorials or online resources to understand their practical usage.

Job List and Monitoring Exercises

Show steps

Practicing job list filtering and monitoring will enhance your ability to manage and debug pipelines.

Show steps

Use the Dataflow web interface to access the Job List.
Filter jobs based on various parameters (e.g., status, start time).
Explore the Job Graph, Job Details, and Job Metrics sections.
Create and manage job alerts to track job status changes.

Error Reporting and Logging Analysis

Show steps

Understanding error reporting and logging will equip you with skills to troubleshoot and debug pipelines effectively.

Browse courses on Error Reporting

Show steps

Follow tutorials on error reporting and logging in Dataflow.
Explore the Error Reporting page in the Dataflow console.
Analyze log files generated by your pipelines.
Use the Metrics Explorer to identify potential errors.

Two other activities

Expand to see all activities and additional details

Show all five activities

Dataflow CI/CD Pipeline Development

Show steps

By creating a CI/CD pipeline for your Dataflow pipelines, you can ensure their reliability and maintainability.

Browse courses on CI/CD Pipeline

Show steps

Set up a CI/CD tool (e.g., Jenkins, Azure DevOps).
Define the pipeline stages (e.g., unit tests, integration tests, deployment).
Integrate your Dataflow pipelines into the CI/CD process.
Configure automated triggers to initiate the pipeline.
Monitor the pipeline execution and receive notifications.

Dataflow Flex Templates Study Group

Show steps

Engage in discussions with peers about the advantages and use cases of Dataflow Flex Templates.

Show steps

Join online forums or discussion groups related to Dataflow Flex Templates.
Share experiences and insights with other Dataflow users.
Collaborate on sample templates and explore different use cases.
Present and discuss findings within the study group.

Career center

Learners who complete Serverless Data Processing with Dataflow: Operations em Português Brasileiro will develop knowledge and skills that may be useful to these careers:

Software Engineer

Software Engineers design, develop, and maintain software applications. They work with customers to understand their needs and then design and implement software solutions that meet those needs. A course in Apache Beam and Google Cloud Dataflow can be useful for Software Engineers who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Software Engineer

Data Engineer

Data Engineers are responsible for the design and implementation of data pipelines. They build and maintain the infrastructure that moves data around an organization. This course may be useful to Data Engineers who want to learn how to use Apache Beam and Google Cloud Dataflow to build and manage data pipelines.

See salaries and explore the career path for Data Engineer

Cloud Architect

Cloud Architects design, build, and maintain cloud computing solutions. They work with customers to understand their business needs and then design and implement cloud solutions that meet those needs. A course in Apache Beam and Google Cloud Dataflow can be useful for Cloud Architects who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Cloud Architect

Data Analyst

Data Analysts collect, clean, and analyze data to identify trends and patterns. They use this information to make recommendations to businesses on how to improve their operations. This course may be useful to Data Analysts who want to learn how to use Apache Beam and Google Cloud Dataflow to build and manage data pipelines.

See salaries and explore the career path for Data Analyst

Data Scientist

Data Scientists use data to solve business problems. They collect, clean, and analyze data to identify trends and patterns. This course may be useful to Data Scientists who want to learn how to use Apache Beam and Google Cloud Dataflow to build and manage data pipelines.

See salaries and explore the career path for Data Scientist

Business Analyst

Business Analysts work with businesses to identify and solve business problems. They use data to understand the business and then develop and implement solutions that improve the business. A course in Apache Beam and Google Cloud Dataflow may be useful for Business Analysts who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Business Analyst

Project Manager

Project Managers are responsible for the planning and execution of projects. They work with stakeholders to define the project scope, timeline, and budget. A course in Apache Beam and Google Cloud Dataflow may be useful for Project Managers who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Project Manager

Database Administrator

Database Administrators are responsible for the maintenance and administration of databases. They work with databases to ensure that they are running smoothly and that data is safe and secure. A course in Apache Beam and Google Cloud Dataflow may be useful for Database Administrators who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Database Administrator

Network Administrator

Network Administrators are responsible for the maintenance and administration of computer networks. They work with networks to ensure that they are running smoothly and that data is safe and secure. A course in Apache Beam and Google Cloud Dataflow may be useful for Network Administrators who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Network Administrator

Product Manager

Product Managers are responsible for the development and management of products. They work with customers to understand their needs and then design and implement products that meet those needs. A course in Apache Beam and Google Cloud Dataflow may be useful for Product Managers who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Product Manager

Systems Administrator

Systems Administrators are responsible for the maintenance and administration of computer systems. They work with systems to ensure that they are running smoothly and that data is safe and secure. A course in Apache Beam and Google Cloud Dataflow may be useful for Systems Administrators who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Systems Administrator

Security Analyst

Security Analysts are responsible for the security of computer systems and networks. They work with systems and networks to identify and mitigate security risks. A course in Apache Beam and Google Cloud Dataflow may be useful for Security Analysts who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Security Analyst

Data Protection Officer

Data Protection Officers are responsible for the implementation and enforcement of data protection policies and procedures. They work with businesses to ensure that data is collected, used, and stored in a compliant and ethical manner. A course in Apache Beam and Google Cloud Dataflow may be useful for Data Protection Officers who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Data Protection Officer

Data Governance Analyst

Data Governance Analysts are responsible for the development and implementation of data governance policies and procedures. They work with businesses to ensure that data is used in a compliant and ethical manner. A course in Apache Beam and Google Cloud Dataflow may be useful for Data Governance Analysts who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Data Governance Analyst

Data Privacy Analyst

Data Privacy Analysts are responsible for the protection of data privacy. They work with businesses to ensure that data is collected, used, and stored in a compliant and ethical manner. A course in Apache Beam and Google Cloud Dataflow may be useful for Data Privacy Analysts who want to learn how to use these technologies to build and manage data pipelines.

See salaries and explore the career path for Data Privacy Analyst

Serverless Data Processing with Dataflow

Operations em Português Brasileiro

What's inside

Syllabus

Good to know

Save this course

Activities

Career center

Reading list

Share

Similar courses