Programming Large-Scale Parallel Systems

Course Objective

At the end of the course, students will be able to

• Understand the main principles of High Performance Computing (Knowledge and understanding)
• Design and implement efficient parallel algorithms on distributed-memory parallel computers (Applying knowledge and understanding)
• Make use of different distributed parallel programming models to implement these algorithms, including message passing with MPI and the task-based model based on remote procedure calls available in the Julia programming language (Applying knowledge and understanding).
• Analyze the performance of parallel applications and network topologies (Making judgements)
• Identify the main parallelization challenges in selected scientific computing applications (Knowledge and understanding)

Course Content

In this course, students learn how to design and implement computer programs that solve a problem in parallel on a distributed-memory computing system such as the supercomputers available in the top500 list (https://top500.org/). This is key in many real world applications to reduce execution time and to solve very large problems that cannot fit in a single compute node. You will learn how to design such parallel algorithms in order to minimize different types of performance overheads such as communication overhead, search overhead, and load imbalance. You will also learn how to analyze the theoretical performance of distributed parallel algorithms and computer network topologies, and how to evaluate the actual performance of parallel implementations using the DAS-5 super computer. Among all parallel programming models available for high performance computing, this course focuses on distributed-memory parallelization. We consider different distributed parallel programming models, including established techniques such as message-passing with MPI, but also modern approaches like the task-based model available in the Julia programming language. The course also discusses the application of these techniques to a set of real-world scientific computing applications.

Additional Information Teaching Methods

• There are 2 class sessions per week. Part of the course follows the flipped classroom method. Lecture notes and exercises are made available in advance for self study, and follow up discussions take place in class meetings to deepen understanding.
• The course includes running experiments on the DAS-5 computer cluster (http://www.cs.vu.nl/das5).
• The parallel algorithms discussed in the course are implemented in the Julia programming language. Previous knowledge of Julia is not required as it will be learned as part of the course.

Method of Assessment

There is a final written exam (E) and a programming assignment (A). The grades E and A are at most 10 points each and are combined into a weighted grade G = 0.8*E + 0.2*A, giving 0.8 weight to the exam and 0.2 weight to the assignment. The final grade (F) is computed according to these cases:

• if E < 5.5, then F=E (if you fail the exam, you fail the course).
• if G < 5.5 and E >= 5.5, then F=5.5 (if you pass the exam, you pass the course).
• Otherwise: F=G.

Additional remarks:

• The final grade (F) is rounded according to this rule: If the final grade is between 5 and 6, then it is rounded to the closest integer (>=5.5 rounds to 6; <5.5 rounds to 5). Otherwise, it is rounded to the closest 0.5.
• There is a resit opportunity for the exam, but there is no resit opportunity for the assignment.
• The exam is individual and the assignment is in groups of two students.

Literature

Course material will be made available on Canvas. Part of the course is based on the lecture notes available at https://www.francescverdugo.com/XM_40017/dev/.

Additional Information Target Audience

Master Business Analytics
Master Computational Science
Master Computer Science
Master Computer Security

Additional Information

There is also a separate course "Parallel Programming Practical" (6 EC) in Period 3, the aim of which is to complement the contents of this course with further practical skills and experience.

Level 2b Use of generative AI is permitted in a limited way, with disclosure

In this course, generative AI (GenAI), such as ChatGPT or similar tools, may be used in a limited way, for example for editorial support, rephrasing, or structure. Substantive contributions from GenAI may only be used within the limits set by the lecturer. Any use of GenAI must always be explicitly disclosed, in accordance with the lecturer's instructions. The student remains fully responsible for the content, quality, and accuracy of the submitted work. Incorrect use of GenAI is considered fraud. If established, the submitted work or assessment result will be declared invalid, and the Examination Board may impose further measures.