Large Scale Data Engineering

Course Objective

The goal of the course is to gain insight into and experience with
algorithms and infrastructures for managing big data. (Knowledge and
understanding) (Applying knowledge and understanding) (Making
judgements) (Lifelong learning skills)

Course Content

This course confronts the students with some data management tasks,
where the challenge is that the mere size of this data causes naive
solutions, and/or solutions that work only on a single machine, to stop
being practical. Solving such tasks requires the computer scientist to
have insight into the main factors that underlie algorithm performance
(data access patterns, hardware latency/bandwidth), as well as possess
certain skills and experience in managing large-scale computing
infrastructure. Apart from the data being of large volume, another
problem invariably is that data comes in strange forms and formats, is
polluted, and needs to be questioned, transformed, and cleaned.
The main part of the course is the second assignment: a large big data
analysis project where each student team tackles a different problem,
and while doing so gains experience in multiple aspects of large-scale
data engineering (critical thinking, data management technologies, cloud
infrastructure, visualization techniques, paper writing).

More information is found on http://event.cwi.nl/lsde - also check the
"showcase" section where you can see past project results
(visualizations and papers).

Additional Information Teaching Methods

There are two lectures per week, and the course requires significant
practical work. The practicals are done outside lecture hours, at the
discretion of the students who are supported remotely through slack,

Method of Assessment

In the first assignment (writing a hand-coded C/C++ program doing graph
analysis on a single machine) the students can work either on their own
laptops via a prepared VM, or in the cloud using an Amazon EC2 Micro
Instance, and there is an online competition for the best result. In the
fifth week, there is an interim test (partly multiple-choice) that
covers the lecture material (note: if by that time physical classes are
not possible, this test might be dropped).
At the end of the first assignment, groups of three are formed and we
switch from C++ to programming in Spark using Scala or Python.
The second assignment is done on large datasets (TB scale) using Spark
on a cluster. The student teams are expected to create a data product, a
visualization, and a report of 5-8 pages. The students also need to find
and read scientific papers related to their project. First, a project
proposal is written, and students peer-review these. There are two 1:1
meetings with the teacher (or TA) to discuss project progress and
challenges.
There is no written exam; the grade is based on the two assignment
grades, the written test (if held) and the grade for the final project
presentation; which is at the very end of the period.

Literature

The course website http://event.cwi.nl/lsde provides all this
information. In this website, each lecture is provided, but also its
main points are summarized. Further, each lecture page provides links to
useful videos and presentations, but also the scientific papers to be
read.

Additional Information Target Audience

MSc Computer Science
MSc Computer Security

Explanation Canvas

Canvas is used for forming practicum groups for assignment 2 (3 people
per group), for turning in assignments and for communicating grades and
feedback on assignments.

Recommended background knowledge

You must have programming experience in C or C++ (for assignment 1).
Willingness to learn Python and some Scala.
Theoretical knowledge of: database systems, computer architecture, and
operating systems. To be clear, a computer science background, typically
based on bachelor courses, that includes knowledge of these topics is
required.
Experience with Linux (i.e. not only Windows) and SQL.
Useful practical skills include: knowing how to work with a debugger
(gdb for C/C++) and git (github).
Assignment2 will be done in Spark; for this programming is typically
done in Python or Scala.