Programming Large-Scale Parallel Systems

Course Objective

At the end of the course, students will be able to

• Understand the main principles of High Performance Computing (Knowledge and understanding)
• Design and implement efficient parallel algorithms on distributed-memory parallel computers (Applying knowledge and understanding)
• Make use of different distributed parallel programming models to implement these algorithms, including message passing with MPI and the task-based model based on remote procedure calls available in the Julia programming language (Applying knowledge and understanding).
• Analyze the performance of parallel applications and network topologies (Making judgements)
• Identify the main parallelization challenges in selected scientific computing applications (Knowledge and understanding)

Course Content

In this course, students learn how to design and implement computer programs that solve a problem in parallel on a distributed-memory computing system such as the supercomputers available in the top500 list (https://top500.org/). This is key in many real world applications to reduce execution time and to solve very large problems that cannot fit in a single compute node. You will learn how to design such parallel algorithms in order to minimize different types of performance overheads such as communication overhead, search overhead, and load imbalance. You will also learn how to analyze the theoretical and experimental performance of the resulting codes and the underlying computer network topology. Among all parallel programming models available for high performance computing, this course focuses on distributed-memory parallelization. We consider different distributed parallel programming models, including established techniques such as message-passing with MPI, but also modern approaches like the task-based model available in the Julia programming language. The course also discusses the application of these techniques to a set of real-world scientific computing applications, including partial-differential equations in solid and fluid dynamics, and n-body problems in astrophysics.

Additional Information Teaching Methods

• There are 2 class sessions per week. Part of the course follows the flipped classroom method. Pre-recordings, lecture notes, and exercises are made available in advance for self study, and follow up discussions take place in class meetings to deepen understanding.
• The course also includes the opportunity to run experiments on the DAS-5 computer cluster (http://www.cs.vu.nl/das5).
• Computer codes used in the course are implemented in the Julia programming language. Previous knowledge of Julia is not required as it will be learned as part of the course.

Method of Assessment

There is a final written exam (E) and a programming assignment (A). Their grades are combined as G = 0.1*A + W(E)*E, where

• W(E) = 1 for E < 7, and
• W(E) = 1-(E-7)/30 otherwise.

The grades E and A are at most 10 points each.

According to this formula the programming assignment is "quasi optional". There is little or no penalty for skipping the assignment in most of the cases. However, high-scoring students in the exam need the assignment for a perfect score.

Literature

Course material will be made available on Canvas. Part of the course is based on the lecture notes available at https://www.francescverdugo.com/XM_40017/dev/.

Additional Information Target Audience

Master Business Analytics
Master Computational Science
Master Computer Science
Master Computer Security

Additional Information

There is also a separate course "Parallel Programming Practical" (6 EC) in Period 3, the aim of which is to complement the contents of this course with further practical skills and experience.