We may earn an affiliate commission when you visit our partners.
Course image
Google Cloud Training

Na última parte da série de cursos do Dataflow, vamos abordar os componentes do modelo operacional do Dataflow. Veremos ferramentas e técnicas para solucionar problemas e otimizar o desempenho do pipeline. Depois analisaremos as práticas recomendadas de teste, implantação e confiabilidade para pipelines do Dataflow. Por fim, faremos uma revisão dos modelos, que facilitam o escalonamento dos pipelines do Dataflow para organizações com centenas de usuários. Essas lições garantem que a plataforma de dados seja estável e resiliente a circunstâncias imprevistas.

Enroll now

What's inside

Syllabus

Introduction
Este módulo mostra o resumo do curso
Monitoring
Neste módulo, aprendemos a usar a página Lista de jobs para filtrar os jobs que queremos monitorar ou investigar. Vimos como as guias Gráfico de job, Informações do job e Métricas do job mostram um resumo abrangente do job do Dataflow. Por fim, aprendemos a usar a integração do Dataflow com o Metrics Explorer e criar políticas de alertas para as métricas do Dataflow.
Read more
Logging e Error Reporting
Neste módulo, aprendemos a usar o painel Registro na parte inferior das páginas Gráfico de job e Métricas do job e mostramos a página centralizada Error Reporting.
Solução de problemas e depuração
Neste módulo, vamos aprender a resolver problemas e depurar pipelines do Dataflow. Também vamos analisar os quatro modos comuns de falha do Dataflow: ao criar o pipeline, ao iniciar o pipeline no Dataflow, durante execução do pipeline e problemas de desempenho.
Desempenho
Neste módulo, veremos as considerações de desempenho que devemos saber ao desenvolver pipelines em lote e de streaming no Dataflow.
Testes e CI/CD
Este módulo vai mostrar o teste de unidade dos pipelines do Dataflow. Também vamos abordar os frameworks e os recursos disponíveis para otimizar o fluxo de trabalho de CI/CD dos pipelines do Dataflow.
Confiabilidade
Neste módulo, veremos os métodos para criar sistemas resilientes a dados corrompidos e falhas temporárias do data center.
Modelos Flex
Este módulo aborda os modelos Flex, um recurso que ajuda as equipes de engenharia de dados a padronizar e reutilizar o código de pipeline do Dataflow. Vários desafios operacionais podem ser resolvidos com esses modelos.
Summary
Este módulo faz uma revisão dos temas abordados no curso.

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Analisa problemas e soluções comuns para criar pipelines de dados confiáveis e com desempenho otimizado
Descreve técnicas para monitorar e solucionar problemas no Google Dataflow
Aborda modelos Flex, permitindo a padronização e reutilização de código de pipeline do Dataflow
Indicado para profissionais com experiência em desenvolvimento de pipelines do Dataflow
Oferece práticas recomendadas de teste, implantação e confiabilidade para pipelines do Dataflow

Save this course

Save Serverless Data Processing with Dataflow: Operations em Português Brasileiro to your list so you can find it easily later:
Save

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Serverless Data Processing with Dataflow: Operations em Português Brasileiro with these activities:
Revise Apache Beam Pipeline Components
Review the concepts of Apache Beam pipelines. It will help you comprehend the operational components better.
Browse courses on Apache Beam
Show steps
  • Go through the documentation of Apache Beam Pipeline Components.
  • Refer to tutorials or online resources to understand their practical usage.
Job List and Monitoring Exercises
Practicing job list filtering and monitoring will enhance your ability to manage and debug pipelines.
Show steps
  • Use the Dataflow web interface to access the Job List.
  • Filter jobs based on various parameters (e.g., status, start time).
  • Explore the Job Graph, Job Details, and Job Metrics sections.
  • Create and manage job alerts to track job status changes.
Error Reporting and Logging Analysis
Understanding error reporting and logging will equip you with skills to troubleshoot and debug pipelines effectively.
Browse courses on Error Reporting
Show steps
  • Follow tutorials on error reporting and logging in Dataflow.
  • Explore the Error Reporting page in the Dataflow console.
  • Analyze log files generated by your pipelines.
  • Use the Metrics Explorer to identify potential errors.
Two other activities
Expand to see all activities and additional details
Show all five activities
Dataflow CI/CD Pipeline Development
By creating a CI/CD pipeline for your Dataflow pipelines, you can ensure their reliability and maintainability.
Browse courses on CI/CD Pipeline
Show steps
  • Set up a CI/CD tool (e.g., Jenkins, Azure DevOps).
  • Define the pipeline stages (e.g., unit tests, integration tests, deployment).
  • Integrate your Dataflow pipelines into the CI/CD process.
  • Configure automated triggers to initiate the pipeline.
  • Monitor the pipeline execution and receive notifications.
Dataflow Flex Templates Study Group
Engage in discussions with peers about the advantages and use cases of Dataflow Flex Templates.
Show steps
  • Join online forums or discussion groups related to Dataflow Flex Templates.
  • Share experiences and insights with other Dataflow users.
  • Collaborate on sample templates and explore different use cases.
  • Present and discuss findings within the study group.

Career center

Learners who complete Serverless Data Processing with Dataflow: Operations em Português Brasileiro will develop knowledge and skills that may be useful to these careers:
Software Engineer
Software Engineers design, develop, and maintain software applications. They work with customers to understand their needs and then design and implement software solutions that meet those needs. A course in Apache Beam and Google Cloud Dataflow can be useful for Software Engineers who want to learn how to use these technologies to build and manage data pipelines.
Data Engineer
Data Engineers are responsible for the design and implementation of data pipelines. They build and maintain the infrastructure that moves data around an organization. This course may be useful to Data Engineers who want to learn how to use Apache Beam and Google Cloud Dataflow to build and manage data pipelines.
Cloud Architect
Cloud Architects design, build, and maintain cloud computing solutions. They work with customers to understand their business needs and then design and implement cloud solutions that meet those needs. A course in Apache Beam and Google Cloud Dataflow can be useful for Cloud Architects who want to learn how to use these technologies to build and manage data pipelines.
Data Analyst
Data Analysts collect, clean, and analyze data to identify trends and patterns. They use this information to make recommendations to businesses on how to improve their operations. This course may be useful to Data Analysts who want to learn how to use Apache Beam and Google Cloud Dataflow to build and manage data pipelines.
Data Scientist
Data Scientists use data to solve business problems. They collect, clean, and analyze data to identify trends and patterns. This course may be useful to Data Scientists who want to learn how to use Apache Beam and Google Cloud Dataflow to build and manage data pipelines.
Business Analyst
Business Analysts work with businesses to identify and solve business problems. They use data to understand the business and then develop and implement solutions that improve the business. A course in Apache Beam and Google Cloud Dataflow may be useful for Business Analysts who want to learn how to use these technologies to build and manage data pipelines.
Project Manager
Project Managers are responsible for the planning and execution of projects. They work with stakeholders to define the project scope, timeline, and budget. A course in Apache Beam and Google Cloud Dataflow may be useful for Project Managers who want to learn how to use these technologies to build and manage data pipelines.
Database Administrator
Database Administrators are responsible for the maintenance and administration of databases. They work with databases to ensure that they are running smoothly and that data is safe and secure. A course in Apache Beam and Google Cloud Dataflow may be useful for Database Administrators who want to learn how to use these technologies to build and manage data pipelines.
Network Administrator
Network Administrators are responsible for the maintenance and administration of computer networks. They work with networks to ensure that they are running smoothly and that data is safe and secure. A course in Apache Beam and Google Cloud Dataflow may be useful for Network Administrators who want to learn how to use these technologies to build and manage data pipelines.
Product Manager
Product Managers are responsible for the development and management of products. They work with customers to understand their needs and then design and implement products that meet those needs. A course in Apache Beam and Google Cloud Dataflow may be useful for Product Managers who want to learn how to use these technologies to build and manage data pipelines.
Systems Administrator
Systems Administrators are responsible for the maintenance and administration of computer systems. They work with systems to ensure that they are running smoothly and that data is safe and secure. A course in Apache Beam and Google Cloud Dataflow may be useful for Systems Administrators who want to learn how to use these technologies to build and manage data pipelines.
Security Analyst
Security Analysts are responsible for the security of computer systems and networks. They work with systems and networks to identify and mitigate security risks. A course in Apache Beam and Google Cloud Dataflow may be useful for Security Analysts who want to learn how to use these technologies to build and manage data pipelines.
Data Protection Officer
Data Protection Officers are responsible for the implementation and enforcement of data protection policies and procedures. They work with businesses to ensure that data is collected, used, and stored in a compliant and ethical manner. A course in Apache Beam and Google Cloud Dataflow may be useful for Data Protection Officers who want to learn how to use these technologies to build and manage data pipelines.
Data Governance Analyst
Data Governance Analysts are responsible for the development and implementation of data governance policies and procedures. They work with businesses to ensure that data is used in a compliant and ethical manner. A course in Apache Beam and Google Cloud Dataflow may be useful for Data Governance Analysts who want to learn how to use these technologies to build and manage data pipelines.
Data Privacy Analyst
Data Privacy Analysts are responsible for the protection of data privacy. They work with businesses to ensure that data is collected, used, and stored in a compliant and ethical manner. A course in Apache Beam and Google Cloud Dataflow may be useful for Data Privacy Analysts who want to learn how to use these technologies to build and manage data pipelines.

Reading list

We've selected six books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Serverless Data Processing with Dataflow: Operations em Português Brasileiro.
Este livro explora os princípios de projetar e construir aplicativos que processam grandes quantidades de dados. Ele abrange tópicos como armazenamento de dados, processamento de dados em lote e de streaming e gerenciamento de estado.
Este livro fornece uma introdução ao uso do Python para engenharia de dados. Ele aborda tópicos como manipulação de dados, processamento de dados de streaming e aprendizado de máquina.
Este livro aborda os princípios de projetar e construir soluções orientadas a dados. Ele complementa o curso fornecendo uma perspectiva de arquitetura e ajudando os alunos a entender como o Dataflow se encaixa em arquiteturas de dados mais amplas.
Este livro fornece uma introdução prática ao aprendizado profundo. Ele aborda tópicos como redes neurais, convoluções e redes neurais recorrentes.
Este livro explora o conceito de data mesh, que é uma abordagem emergente para gerenciar e usar dados. Ele complementa o curso abordando o papel do Dataflow no ecossistema de data mesh e fornecendo uma visão mais estratégica.
Este livro aborda a análise de dados em tempo real, que é um aspecto importante do processamento de dados em larga escala. Ele complementa o curso fornecendo uma visão geral das tecnologias e técnicas usadas para análise de dados em tempo real, incluindo o Dataflow.

Share

Help others find this course page by sharing it with your friends and followers:

Similar courses

Here are nine courses similar to Serverless Data Processing with Dataflow: Operations em Português Brasileiro.
Serverless Data Processing with Dataflow: Foundations em...
Most relevant
Building Batch Data Pipelines on GCP em Português...
Most relevant
ML Pipelines on Google Cloud - Português
Most relevant
Building Resilient Streaming Systems on GCP em Português...
Most relevant
Atrair e engajar clientes com marketing digital
Most relevant
Proficiência Em Arduino – O Mundos Dos Sensores
Most relevant
UX & Design Thinking: Experiência do Usuário nos negócios
Most relevant
Finanças Orientada a Dados
Most relevant
O sucesso por meio das avaliações: análise e medição de...
Most relevant
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2024 OpenCourser