Advanced Big Data Systems | 高级大数据系统 from edX

本课程将重点讲解高级大数据系统的实现、优化和应用，包括分布式文件系统、MapReduce/Spark、Storm/Spark streaming、Mahout等系统的原理、实现、策略优化。

近年来，人工智能技术正在快速地渗透进各个不同领域。因大数据系统是当今数据驱动人工智能的基础，而变得至关重要。本课程旨在引导学生了解大数据系统的基本概念，包括如何有效地存储、处理和分析数据。课程从分布式系统设计的一般原理出发。之后我们提供了如何在大数据系统中评定存储、计算和网络功能的框架。最后，为了使这些设计原则便于理解，我们的案例研究将使用真实的工业系统来演示基本设计原则如何应用于实际系统，以及该如何分析它们的性能以及局限性。

本课程将重点讲解高级大数据系统的实现、优化和应用，包括分布式文件系统、MapReduce/Spark、Storm/Spark streaming、Mahout等系统的原理、实现、策略优化。

Recent years have witnessed the rapid increase of the penetration of AI technology into different areas in the industry. Big data systems, the foundation that enables today’s data-driven AI, are thus becoming critically important. This course is dedicated to lead students into the basic concepts of big data systems, covering how data is effectively stored, processed and analyzed. We start from the general principles in the design of distributed systems; then we provide frameworks on how storage, computation, and network capabilities are scaled in big data systems; finally, to make such design principles easy to follow, our case studies use real industrial systems to demonstrate how the basic design principles are applied in real-world systems as well as how their performance and limitation are analyzed.

What's inside

Learning objectives

Basic concepts of big data systems
Principelsof designing distributed systems
Frameworks on scaling storage, computaion and network capabilities

Case studeis of recent industrial big data systems, including gfs, mapreduce and spark
Big data processing pipelines such as nosql, streaming, and graph data processing

Basic concepts of big data systems
Principelsof designing distributed systems
Frameworks on scaling storage, computaion and network capabilities
Case studeis of recent industrial big data systems, including gfs, mapreduce and spark
Big data processing pipelines such as nosql, streaming, and graph data processing

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

适合高级大数据系统相关学习者学习。

适合想要了解大数据系统基本概念的学习者。

适合想要提高数据存储、处理和分析技能的学习者学习。

授课者具有在大数据系统方面的丰富经验。

课程内容涵盖了大数据系统的设计、实现和应用等核心知识点。

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Advanced Big Data Systems | 高级大数据系统 with these activities:

学习 Spark Streaming

Show steps

掌握 Spark Streaming 的基本概念和应用，增强对大数据实时处理能力的理解。

Browse courses on Spark Streaming

Show steps

查找并参加 Spark Streaming 相关教程
完成教程并练习示例代码
尝试一个小型流数据处理项目

Show all one activities

Career center

Learners who complete Advanced Big Data Systems | 高级大数据系统 will develop knowledge and skills that may be useful to these careers:

Business Intelligence Analyst

Business Intelligence Analysts use data to help businesses make better decisions. This course can help aspiring Business Intelligence Analysts build a foundation in the concepts and principles of big data systems, including distributed file systems, MapReduce/Spark, Storm/Spark streaming, and Mahout. It also covers the optimization and application of these systems, making you a more well-rounded and effective Business Intelligence Analyst.

See salaries and explore the career path for Business Intelligence Analyst

Quantitative Analyst

Quantitative Analysts use mathematical and statistical models to analyze data. This course can help aspiring Quantitative Analysts build a foundation in the concepts and principles of big data systems, including distributed file systems, MapReduce/Spark, Storm/Spark streaming, and Mahout. It also covers the optimization and application of these systems, making you a more well-rounded and effective Quantitative Analyst.

See salaries and explore the career path for Quantitative Analyst

Data Mining Analyst

Data Mining Analysts use data mining techniques to extract knowledge and insights from data. This course can help aspiring Data Mining Analysts build a foundation in the concepts and principles of big data systems, including distributed file systems, MapReduce/Spark, Storm/Spark streaming, and Mahout. It also covers the optimization and application of these systems, making you a more well-rounded and effective Data Mining Analyst.

See salaries and explore the career path for Data Mining Analyst

Software Engineer

Software Engineers design, develop and maintain software systems. This course can help aspiring Software Engineers build a foundation in the concepts and principles of big data systems, including distributed file systems, MapReduce/Spark, Storm/Spark streaming, and Mahout. It also covers the optimization and application of these systems, making you a more well-rounded and effective Software Engineer.

See salaries and explore the career path for Software Engineer

Data Analyst

Data Analysts collect, clean and analyze data to identify trends and patterns. This course can help aspiring Data Analysts build a foundation in the concepts and principles of big data systems, including distributed file systems, MapReduce/Spark, Storm/Spark streaming, and Mahout. It also covers the optimization and application of these systems, making you a more well-rounded and effective Data Analyst.

See salaries and explore the career path for Data Analyst

Cloud Engineer

Cloud Engineers design, build and maintain cloud computing systems and applications. This course can help aspiring Cloud Engineers build a foundation in the concepts and principles of big data systems, including distributed file systems, MapReduce/Spark, Storm/Spark streaming, and Mahout. It also covers the optimization and application of these systems, making you a more well-rounded and effective Cloud Engineer.

See salaries and explore the career path for Cloud Engineer

Data Engineer

Data Engineers design, build, and maintain data pipelines and infrastructure to support data-driven decision-making. This course can help aspiring Data Engineers build a foundation in the concepts and principles of big data systems, including distributed file systems, MapReduce/Spark, Storm/Spark streaming, and Mahout. It also covers the optimization and application of these systems, making you a more well-rounded and effective Data Engineer.

See salaries and explore the career path for Data Engineer

Data Scientist

Data Scientists use scientific methods, processes, algorithms and systems to extract knowledge and insights from data. This course can help aspiring Data Scientists understand the underlying principles and architectures of big data systems, which are essential for working with and analyzing large datasets. By taking this course, you will gain a competitive edge in the field of Data Science.

See salaries and explore the career path for Data Scientist

Machine Learning Engineer

Machine Learning Engineers design, develop and deploy machine learning models to solve real-world problems. This course can help aspiring Machine Learning Engineers understand the underlying principles and architectures of big data systems, which are essential for working with and analyzing large datasets used in machine learning. By taking this course, you will gain a competitive edge in the field of Machine Learning.

See salaries and explore the career path for Machine Learning Engineer

Cloud Architect

Cloud Architects design, build and manage cloud computing systems and applications. This course may be useful for aspiring Cloud Architects, as it covers the principles of designing distributed systems, as well as frameworks for scaling storage, computation and network capabilities, which are essential for designing and managing cloud-based systems and applications at scale.

See salaries and explore the career path for Cloud Architect

DevOps Engineer

DevOps Engineers work to bridge the gap between development and operations teams, ensuring that software is built, tested, and deployed efficiently and reliably. This course may be useful for aspiring DevOps Engineers, as it covers the principles of designing distributed systems, as well as frameworks for scaling storage, computation and network capabilities, which are essential for designing and managing the infrastructure that supports software development and deployment.

See salaries and explore the career path for DevOps Engineer

Systems Administrator

Systems Administrators are responsible for the management and maintenance of computer systems and networks. This course may be useful for aspiring Systems Administrators, as it covers the principles of designing distributed systems, as well as frameworks for scaling storage, computation and network capabilities, which are essential for managing and maintaining complex systems at scale.

See salaries and explore the career path for Systems Administrator

Software Architect

Software Architects design, build and maintain the overall architecture of software systems. This course may be useful for aspiring Software Architects, as it covers the principles of designing distributed systems, as well as frameworks for scaling storage, computation and network capabilities, which are essential for designing and managing complex software systems at scale.

See salaries and explore the career path for Software Architect

Data Architect

Data Architects plan, design, and build an organization's data systems. This course may be useful for aspiring Data Architects, as it will teach you the principles of designing distributed systems, as well as frameworks for scaling storage, computation and network capabilities, which are critical concepts for designing efficient and reliable data systems at scale. Furthermore, this course provides case studies of recent industrial big data systems such as GFS, MapReduce and Spark, which will give you practical insights into the design and implementation of real-world data systems, making you a more competitive candidate for Data Architect roles.

See salaries and explore the career path for Data Architect

Database Administrator

Database Administrators are responsible for the design, implementation, maintenance and security of database management systems. This course may be useful for aspiring Database Administrators, as it covers the principles of designing distributed systems, as well as frameworks for scaling storage, computation and network capabilities, which are essential for designing and managing efficient and reliable database systems at scale.

See salaries and explore the career path for Database Administrator