LAFF-On Programming for High Performance from edX

Is my code fast? Can it be faster? Scientific computing, machine learning, and data science are about solving problems that are compute intensive. Choosing the right algorithm, extracting parallelism at various levels, and amortizing the cost of data movement are vital to achieving scalable speedup and high performance.

In this course, the simple but important example of matrix-matrix multiplication is used to illustrate fundamental techniques for attaining high-performance on modern CPUs. A carefully designed and scaffolded sequence of exercises leads the learner from a naive implementation to one that effectively utilizes instruction level parallelism and culminates in a high-performance multithreaded implementation. Along the way, it is discovered that careful attention to data movement is key to efficient computing.

Prerequisites for this course are a basic understanding of matrix computations (roughly equivalent toWeeks 1-5 of Linear Algebra: Foundations to Frontiers on edX) and an exposure to programming. Hands-on exercises start with skeletal code in the C programming language that is progressively modified, so that extensive experience with C is not required. Access to a relatively recent x86 processor such as Intel Haswell or AMD Ryzen (or newer) running Linux is required.

MATLAB Online licenses will be made available to the participants free of charge for the duration of the course.

Join us to satisfy your need for speed!

What's inside

Learning objectives

Mapping algorithms to architectures
Extracting parallelism at multiple levels
Amortizing data movement over computation

Understanding performance data
Managing complexity through layering of software

Mapping algorithms to architectures
Extracting parallelism at multiple levels
Amortizing data movement over computation
Understanding performance data
Managing complexity through layering of software

Syllabus

0 Getting Started

1 Loops and More Loops

2 Start Your Engines

3 Pushing the Limits

4 Multithreaded Parallelism

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Teaches vital high-performance computing principles that are applicable across industries like scientific computing, machine learning, and big data

Provides a solid foundation in the fundamentals of high-performance computing, equipping learners with foundational knowledge and skills

Designed for individuals with a basic understanding of matrix computations and exposure to programming, addressing various skill levels

Emphasizes the importance of data movement efficiency, providing valuable insights for optimizing performance in real-world scenarios

Offers hands-on exercises that gradually build upon the learner's foundation, allowing them to apply concepts directly and enhance their understanding

Introduces key techniques for high-performance computing on modern CPU architectures, offering practical knowledge that is applicable in a variety of settings

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in LAFF-On Programming for High Performance with these activities:

Review fundamental matrix operations

Show steps

Reviewing matrix operations will provide a stronger foundation for understanding the complexities of matrix-matrix multiplication.

Browse courses on Matrix Operations

Show steps

Re-familiarize yourself with matrix addition, subtraction, and scalar multiplication
Practice multiplying matrices of different dimensions using the dot product rule
Review the concept of matrix inversion and its properties

Organize a study group with classmates

Show steps

Participating in a study group provides opportunities for peer-to-peer learning and reinforce concepts through discussions.

Show steps

Reach out to classmates and form a study group
Establish regular meeting times and study topics
Engage in discussions, solve problems, and quiz each other

Implement matrix-matrix multiplication in C

Show steps

Implementing matrix-matrix multiplication in C provides hands-on practice for applying the core concepts of the course.

Browse courses on Matrix-Matrix Multiplication

Show steps

Create a function to perform matrix-matrix multiplication using nested loops
Test your implementation with matrices of various dimensions
Analyze the performance of your implementation using different input sizes

Five other activities

Expand to see all activities and additional details

Show all eight activities

Attend a workshop on high-performance computing

Show steps

Attending a workshop on high-performance computing provides exposure to industry best practices and networking opportunities.

Browse courses on High-Performance Computing

Show steps

Research and identify relevant workshops
Register and attend the workshop
Actively participate in discussions and hands-on exercises

Read 'Matrix Computations' by Golub and Van Loan

Show steps

This book provides a comprehensive overview of matrix computations and is a valuable resource for consolidating the theoretical foundations of the course.

View Matrix Computations (Johns Hopkins Studies in... on Amazon

Show steps

Familiarize yourself with the basics of matrix theory and numerical linear algebra
Read and understand the concepts of matrix decomposition, conditioning, and stability
Solve exercises and practice problems from the book

Build a visual representation of matrix-matrix multiplication

Show steps

Creating a visual representation of matrix-matrix multiplication helps to solidify understanding of the process and its computational complexity.

Browse courses on Matrix-Matrix Multiplication

Show steps

Use a programming language or software to create a visualization
Animate the process to demonstrate how elements are multiplied and summed
Include visualizations for different matrix dimensions to illustrate scalability

Contribute to an open-source matrix library

Show steps

Contributing to an open-source matrix library provides practical experience and deepens understanding of matrix-related algorithms and data structures.

Browse courses on Open Source

Show steps

Identify an open-source matrix library to contribute to
Review the codebase and identify areas for improvement
Write and test code changes following the project's coding guidelines

Explore advanced techniques for optimizing matrix-matrix multiplication

Show steps

Exploring advanced optimization techniques provides insights into improving the efficiency of matrix-matrix multiplication algorithms.

Browse courses on Matrix-Matrix Multiplication

Show steps

Review Strassen's algorithm and its advantages over the naive approach
Investigate cache-aware algorithms and their impact on performance
Explore parallel programming techniques for multi-core architectures

Career center

Learners who complete LAFF-On Programming for High Performance will develop knowledge and skills that may be useful to these careers:

High Performance Computing Specialist

High Performance Computing Specialists design, develop, and deploy high-performance computing systems. They work in a variety of settings, including academia, industry, and government. As a High Performance Computing Specialist, you may specialize in a particular area of high-performance computing, such as algorithm optimization or system architecture. The course LAFF-On Programming for High Performance can help you build a foundation in the fundamentals of high-performance computing. This course will teach you how to write efficient code that can run on modern CPUs. You will also learn how to identify and eliminate performance bottlenecks in your code.

See salaries and explore the career path for High Performance Computing Specialist

Computer Scientist

Computer Scientists conduct research and develop new computing technologies. They work in a variety of settings, including academia, industry, and government. As a Computer Scientist, you may specialize in a particular area of computer science, such as high-performance computing. The course LAFF-On Programming for High Performance can help you build a foundation in the fundamentals of high-performance computing. This course will teach you how to write efficient code that can run on modern CPUs. You will also learn how to identify and eliminate performance bottlenecks in your code.

See salaries and explore the career path for Computer Scientist

Software Engineer

Software Engineers design, develop, and deploy software applications. They work closely with other engineers and scientists to translate ideas into working software on time and within budget. As a Software Engineer, you may specialize in a particular area of software development, such as high-performance computing. The course LAFF-On Programming for High Performance can help you build a foundation in the fundamentals of high-performance computing. This course will teach you how to write efficient code that can run on modern CPUs. You will also learn how to identify and eliminate performance bottlenecks in your code.

See salaries and explore the career path for Software Engineer

Performance Analyst

Performance Analysts identify and eliminate performance bottlenecks in software applications. They work closely with other engineers and scientists to ensure that software applications run efficiently and meet performance requirements. As a Performance Analyst, you may specialize in a particular area of performance analysis, such as high-performance computing. The course LAFF-On Programming for High Performance can help you build a foundation in the fundamentals of high-performance computing. This course will teach you how to write efficient code that can run on modern CPUs. You will also learn how to identify and eliminate performance bottlenecks in your code.

See salaries and explore the career path for Performance Analyst

Machine Learning Engineer

Machine Learning Engineers design, develop, and deploy machine learning models. They work closely with other engineers and scientists to translate ideas into working software on time and within budget. As a Machine Learning Engineer, you may specialize in a particular area of machine learning, such as high-performance computing. The course LAFF-On Programming for High Performance can help you build a foundation in the fundamentals of high-performance computing. This course will teach you how to write efficient code that can run on modern CPUs. You will also learn how to identify and eliminate performance bottlenecks in your code.

See salaries and explore the career path for Machine Learning Engineer

Data Scientist

Data Scientists use scientific methods and techniques to extract knowledge from data. They work in a variety of industries, including finance, healthcare, and manufacturing. As a Data Scientist, you may specialize in a particular area of data science, such as high-performance computing. The course LAFF-On Programming for High Performance can help you build a foundation in the fundamentals of high-performance computing. This course will teach you how to write efficient code that can run on modern CPUs. You will also learn how to identify and eliminate performance bottlenecks in your code.

See salaries and explore the career path for Data Scientist

Computational Scientist

Computational Scientists use computers to solve scientific problems. They work in a variety of fields, including physics, chemistry, and biology. As a Computational Scientist, you may specialize in a particular area of computational science, such as high-performance computing. The course LAFF-On Programming for High Performance can help you build a foundation in the fundamentals of high-performance computing. This course will teach you how to write efficient code that can run on modern CPUs. You will also learn how to identify and eliminate performance bottlenecks in your code.

See salaries and explore the career path for Computational Scientist

Systems Architect

Systems Architects design and develop the architecture of software systems. They work closely with other engineers and scientists to ensure that software systems meet the needs of users and stakeholders. As a Systems Architect, you may specialize in a particular area of systems architecture, such as high-performance computing. The course LAFF-On Programming for High Performance can help you build a foundation in the fundamentals of high-performance computing. This course will teach you how to write efficient code that can run on modern CPUs. You will also learn how to identify and eliminate performance bottlenecks in your code.

See salaries and explore the career path for Systems Architect

Software Developer

Software Developers design, develop, and deploy software applications. They work closely with other engineers and scientists to translate ideas into working software on time and within budget. As a Software Developer, you may specialize in a particular area of software development, such as high-performance computing. The course LAFF-On Programming for High Performance may be useful for you if you are interested in developing high-performance software applications.

See salaries and explore the career path for Software Developer

Operations Research Analyst

Operations Research Analysts use mathematical and statistical techniques to solve business problems. They work closely with other engineers and scientists to translate data into actionable insights. As an Operations Research Analyst, you may specialize in a particular area of operations research, such as high-performance computing. The course LAFF-On Programming for High Performance may be useful for you if you are interested in developing high-performance operations research applications.

See salaries and explore the career path for Operations Research Analyst

Computer Programmer

Computer Programmers write and maintain computer code. They work closely with other engineers and scientists to translate ideas into working software on time and within budget. As a Computer Programmer, you may specialize in a particular area of programming, such as high-performance computing. The course LAFF-On Programming for High Performance may be useful for you if you are interested in developing high-performance software applications.

See salaries and explore the career path for Computer Programmer

Management Analyst

Management Analysts use data to solve business problems. They work closely with other engineers and scientists to translate data into actionable insights. As a Management Analyst, you may specialize in a particular area of management analysis, such as high-performance computing. The course LAFF-On Programming for High Performance may be useful for you if you are interested in developing high-performance management analysis applications.

See salaries and explore the career path for Management Analyst

Data Analyst

Data Analysts use data to solve business problems. They work closely with other engineers and scientists to translate data into actionable insights. As a Data Analyst, you may specialize in a particular area of data analysis, such as high-performance computing. The course LAFF-On Programming for High Performance may be useful for you if you are interested in developing high-performance data analysis applications.

See salaries and explore the career path for Data Analyst

Business Analyst

Business Analysts use data to solve business problems. They work closely with other engineers and scientists to translate data into actionable insights. As a Business Analyst, you may specialize in a particular area of business analysis, such as high-performance computing. The course LAFF-On Programming for High Performance may be useful for you if you are interested in developing high-performance business analysis applications.

See salaries and explore the career path for Business Analyst

Financial Analyst

Financial Analysts use data to solve business problems. They work closely with other engineers and scientists to translate data into actionable insights. As a Financial Analyst, you may specialize in a particular area of financial analysis, such as high-performance computing. The course LAFF-On Programming for High Performance may be useful for you if you are interested in developing high-performance financial analysis applications.

See salaries and explore the career path for Financial Analyst