A micro-credential (aka micro-certificate) is a digital certificate attesting that you have gained knowledge and skills in a specific area. It should be internationally recognized and verifiable using an online EU-wide verification system.

A micro-credential can be obtained both by the university students and external participants.

External Participants

If you are not a university student, you will (hopefully soon) be able to apply to the Deep Learning micro-credential course and then attend the course along the university students. Upon successfully passing the course, you will obtain the micro-credential.

Given that the micro-credentials are relatively new, we are still working hard to finalize the online application process, including the exact price. Preliminarily, we expect the price to be approximately 5 000 Kč.

University Students

If you have passed the course as a part of your study plan (in academic year 2024/25 or later), you can obtain the micro-credential by paying only an administrative fee. The exact price is not known yet, but our estimate is around 250 Kč. More information will be sent to the course participants during the examination period.

The lecture content, including references to study materials. The main study material is the Deep Learning Book by Ian Goodfellow, Yoshua Bengio and Aaron Courville, (referred to as DLB).

References to study materials cover all theory required at the exam, and sometimes even more – the references in italics cover topics not required for the exam.

1. Introduction to Deep Learning

 Feb 18 Slides PDF Slides

Introduction to Deep Learning

Requirements

To pass the practicals, you need to obtain at least 80 points, excluding the bonus points. Note that all surplus points (both bonus and non-bonus) will be transfered to the exam. In total, assignments for at least 120 points (not including the bonus points) will be available, and if you solve all the assignments (any non-zero amount of points counts as solved), you automatically pass the exam with grade 1.

Environment

The tasks are evaluated automatically using the ReCodEx Code Examiner.

The evaluation is performed using Python 3.11, Keras 3.8.0, PyTorch 2.6.0, HF Transformers 4.48.0, and Gymnasium 1.0.0. You should install the exact version of these packages yourselves.

Teamwork

Solving assignments in teams (of size at most 3) is encouraged, but everyone has to participate (it is forbidden not to work on an assignment and then submit a solution created by other team members). All members of the team must submit in ReCodEx individually, but can have exactly the same sources/models/results. Each such solution must explicitly list all members of the team to allow plagiarism detection using this template.

No Cheating

Cheating is strictly prohibited and any student found cheating will be punished. The punishment can involve failing the whole course, or, in grave cases, being expelled from the faculty. While discussing assignments with any classmate is fine, each team must complete the assignments themselves, without using code they did not write (unless explicitly allowed). Of course, inside a team you are allowed to share code and submit identical solutions. Note that all students involved in cheating will be punished, so if you share your source code with a friend, both you and your friend will be punished. That also means that you should never publish your solutions.

In the competitions, your goal is to train a model, and then predict target values on the given unannotated test set.

Submitting to ReCodEx

When submitting a competition solution to ReCodEx, you can include any number of files of any kind, and either submit them individually or compess them in a .zip file. However, there should be exactly one text file with the test set annotation (.txt) and at least one Python source (.py/ipynb) containing the model training and prediction. The Python sources are not executed, but must be included for inspection.

Competition Evaluation

• For every submission, ReCodEx checks the above conditions (exactly one .txt, at least one .py/ipynb) and whether the given annotations can be evaluated without error. If not, it will report the corresponding error in the logs.

• Before the first deadline, ReCodEx prints the exact achieved performance, but only if it is worse than the baseline.

  If you surpass the baseline, the assignment is marked as solved in ReCodEx and you immediately get regular points for the assignment. However, ReCodEx does not print the reached performance.

• After the first deadline, the latest submission of every user surpassing the required baseline participates in a competition. Additional bonus points are then awarded according to the ordering of the performance of the participating submissions.

• After the competition results announcement, ReCodEx starts to show the exact performance for all the already submitted solutions and also for the solutions submitted later.

What Is Allowed

• You can use only the given annotated data for training and evaluation.
• You can use the given annotated training data in any way.
• You can use the given annotated development data for evaluation or hyperparameter tuning, but not for the training itself.
• Additionally, you can use any unannotated or manually created data for training and evaluation.
• The test set annotations must be the result of your system (so you cannot manually correct them; but your system can contain other parts than just trained models, like hand-written rules).
• Do not use test set annotations in any way, if you somehow get access to them.
• Unless stated otherwise, you can use any architecture to solve the competition task at hand, but the implementation must be created by you and you must understand it fully. You can of course take inspiration from any paper or existing implementation, but please reference it in that case.
  • You can of course use anything from the Keras/PyTorch packages (but not models from Keras CV, torchvision, …).
  • You can use any data augmentation (even implementations not written by you).
  • You can use any optimizer and any hyperparameter optimization method (even implementations not written by you).
• If you utilize an already trained model, it must be trained only on the allowed training data, unless stated otherwise.

Install

• Installing to central user packages repository

  You can install all required packages to central user packages repository using python3 -m pip install --user --no-cache-dir --extra-index-url=https://download.pytorch.org/whl/cu118 npfl138.

  On Linux and Windows, the above command installs CUDA 11.8 PyTorch build, but you can change cu118 to:

  • cpu to get CPU-only (smaller) version,
  • cu124 to get CUDA 12.4 build,
  • rocm6.2 to get AMD ROCm 6.2 build (Linux only).

  On macOS, the --extra-index-url has no effect and the Metal support is installed in any case.

• Installing to a virtual environment

  Python supports virtual environments, which are directories containing independent sets of installed packages. You can create a virtual environment by running python3 -m venv VENV_DIR followed by VENV_DIR/bin/pip install --no-cache-dir --extra-index-url=https://download.pytorch.org/whl/cu118 npfl138. (or VENV_DIR/Scripts/pip on Windows).

  Again, apart from the CUDA 11.8 build, you can change cu118 on Linux and Windows to:

  • cpu to get CPU-only (smaller) version,
  • cu124 to get CUDA 12.4 build,
  • rocm6.2 to get AMD ROCm 6.2 build (Linux only).

• Windows installation

  • On Windows, it can happen that python3 is not in PATH, while py command is – in that case you can use py -m venv VENV_DIR, which uses the newest Python available, or for example py -3.11 -m venv VENV_DIR, which uses Python version 3.11.

  • If you encounter a problem creating the logs in the args.logdir directory, a possible cause is that the path is longer than 260 characters, which is the default maximum length of a complete path on Windows. However, you can increase this limit on Windows 10, version 1607 or later, by following the instructions.

• GPU support on Linux and Windows

  PyTorch supports NVIDIA GPU or AMD GPU out of the box, you just need to select appropriate --extra-index-url when installing the packages.

  If you encounter problems loading CUDA or cuDNN libraries, make sure your LD_LIBRARY_PATH does not contain paths to older CUDA/cuDNN libraries.

MetaCentrum

• How to apply for MetaCentrum account?

  After reading the Terms and conditions, you can apply for an account here.

  After your account is created, please make sure that the directories containing your solutions are always private.

• How to activate Python 3.10 on MetaCentrum?

  On Metacentrum, currently the newest available Python is 3.10, which you need to activate in every session by running the following command:

  module add python/python-3.10.4-intel-19.0.4-sc7snnf
  

• How to install the required virtual environment on MetaCentrum?

  To create a virtual environment, you first need to decide where it will reside. Either you can find a permanent storage, where you have large-enough quota, or you can use scratch storage for a submitted job.

  TL;DR:

  • Run an interactive CPU job, asking for 16GB scratch space:

    qsub -l select=1:ncpus=1:mem=8gb:scratch_local=16gb -I
    

  • In the job, use the allocated scratch space as the temporary directory:

    export TMPDIR=$SCRATCHDIR
    

  • You should clear the scratch space before you exit using the clean_scratch command. You can instruct the shell to call it automatically by running:

    trap 'clean_scratch' TERM EXIT
    

  • Finally, create the virtual environment and install PyTorch in it:

    module add python/python-3.10.4-intel-19.0.4-sc7snnf
    python3 -m venv CHOSEN_VENV_DIR
    CHOSEN_VENV_DIR/bin/pip install --no-cache-dir --upgrade pip setuptools
    CHOSEN_VENV_DIR/bin/pip install --no-cache-dir --extra-index-url=https://download.pytorch.org/whl/cu118 npfl138
    

• How to run a GPU computation on MetaCentrum?

  First, read the official MetaCentrum documentation: Basic terms, Run simple job, GPU computing, GPU clusters.

  TL;DR: To run an interactive GPU job with 1 CPU, 1 GPU, 8GB RAM, and 16GB scatch space, run:

  qsub -q gpu -l select=1:ncpus=1:ngpus=1:mem=8gb:scratch_local=16gb -I
  

  To run a script in a non-interactive way, replace the -I option with the script to be executed.

  If you want to run a CPU-only computation, remove the -q gpu and ngpus=1: from the above commands.

AIC

• How to install required packages on AIC?

  The Python 3.11.7 is available /opt/python/3.11.7/bin/python3, so you should start by creating a virtual environment using

  /opt/python/3.11.7/bin/python3 -m venv VENV_DIR
  

  and then install the required packages in it using

  VENV_DIR/bin/pip install --no-cache-dir --extra-index-url=https://download.pytorch.org/whl/cu118 npfl138
  

• How to run a GPU computation on AIC?

  First, read the official AIC documentation: Submitting CPU Jobs, Submitting GPU Jobs.

  TL;DR: To run an interactive GPU job with 1 CPU, 1 GPU, and 16GB RAM, run:

  srun -p gpu -c1 -G1 --mem=16G --pty bash
  

  To run a shell script requiring a GPU in a non-interactive way, use

  sbatch -p gpu -c1 -G1 --mem=16G SCRIPT_PATH
  

  If you want to run a CPU-only computation, remove the -p gpu and -G1 from the above commands.

Git

• Is it possible to keep the solutions in a Git repository?

  Definitely. Keeping the solutions in a branch of your repository, where you merge them with the course repository, is probably a good idea. However, please keep the cloned repository with your solutions private.

• On GitHub, do not create a public fork with your solutions

  If you keep your solutions in a GitHub repository, please do not create a clone of the repository by using the Fork button – this way, the cloned repository would be public.

  Of course, if you just want to create a pull request, GitHub requires a public fork and that is fine – just do not store your solutions in it.

• How to clone the course repository?

  To clone the course repository, run

  git clone https://github.com/ufal/npfl138
  

  This creates the repository in the npfl138 subdirectory; if you want a different name, add it as a last parameter.

  To update the repository, run git pull inside the repository directory.

• How to keep the course repository as a branch in your repository?

  If you want to store the course repository just in a local branch of your existing repository, you can run the following command while in it:

  git remote add upstream https://github.com/ufal/npfl138
  git fetch upstream
  git checkout -t upstream/master
  

  This creates a branch master; if you want a different name, add -b BRANCH_NAME to the last command.

  In both cases, you can update your checkout by running git pull while in it.

• How to merge the course repository with your modifications?

  If you want to store your solutions in a branch merged with the course repository, you should start by

  git remote add upstream https://github.com/ufal/npfl138
  git pull upstream master
  

  which creates a branch master; if you want a different name, change the last argument to master:BRANCH_NAME.

  You can then commit to this branch and push it to your repository.

  To merge the current course repository with your branch, run

  git merge upstream master
  

  while in your branch. Of course, it might be necessary to resolve conflicts if both you and I modified the same place in the templates.

ReCodEx

• What files can be submitted to ReCodEx?

  You can submit multiple files of any type to ReCodEx. There is a limit of 20 files per submission, with a total size of 20MB.

• What file does ReCodEx execute and what arguments does it use?

  Exactly one file with py suffix must contain a line starting with def main(. Such a file is imported by ReCodEx and the main method is executed (during the import, __name__ == "__recodex__").

  The file must also export an argument parser called parser. ReCodEx uses its arguments and default values, but it overwrites some of the arguments depending on the test being executed – the template should always indicate which arguments are set by ReCodEx and which are left intact.

• What are the time and memory limits?

  The memory limit during evaluation is 1.5GB. The time limit varies, but it should be at least 10 seconds and at least twice the running time of my solution.

Finetuning

• How to make a part of the network frozen, so that its weights are not updated?

  Each keras.layers.Layer/keras.Model has a mutable trainable property indicating whether its variables should be updated – however, after changing it, you need to call .compile again (or otherwise make sure the list of trainable variables for the optimizer is updated).

  Note that once trainable == False, the insides of a layer are no longer considered, even if some its sub-layers have trainable == True. Therefore, if you want to freeze only some sub-layers of a layer you use in your model, the layer itself must have trainable == True.

• How to choose whether dropout/batch normalization is executed in training or inference regime?

  When calling a keras.layers.Layer/keras.Model, a named option training can be specified, indicating whether training or inference regime should be used. For a model, this option is automatically passed to its layers which require it, and Keras automatically passes it during model.{fit,evaluate,predict}.

  However, you can manually pass for example training=False to a layer when using Functional API, meaning that layer is executed in the inference regime even when the whole model is training.

• How does trainable and training interact?

  The only layer, which is influenced by both these options, is batch normalization, for which:

  • if trainable == False, the layer is always executed in inference regime;
  • if trainable == True, the training/inference regime is chosen according to the training option.

TensorBoard

• Cannot start TensorBoard after installation

  If tensorboard executable cannot be found, make sure the directory with pip installed packages is in your PATH (that directory is either in your virtual environment if you use a virtual environment, or it should be ~/.local/bin on Linux and %UserProfile%\AppData\Roaming\Python\Python311 and %UserProfile%\AppData\Roaming\Python\Python311\Scripts on Windows).

• What can be logged in TensorBoard? See the documentation of the SummaryWriter. Common possibilities are:

  • scalar values:

    summary_writer.add_scalar(name like "train/loss", value, step)
    

  • tensor values displayed as histograms or distributions:

    summary_writer.add_histogram(name like "train/output_layer", tensor, step)
    

  • images as tensors with shape [num_images, h, w, channels], where channels can be 1 (grayscale), 2 (grayscale + alpha), 3 (RGB), 4 (RGBA):

    summary_writer.add_images(name like "train/samples", images, step, dataformats="NHWC")
    

    Other dataformats are "HWC" (shape [h, w, channels]), "HW", "NCHW", "CHW".
  • possibly large amount of text (e.g., all hyperparameter values, sample translations in MT, …) in Markdown format:

    summary_writer.add_text(name like "hyperparameters", markdown, step)
    

  • audio as tensors with shape [1, samples] and values in $[-1,1]$ range:

    summary_writer.add_audio(name like "train/samples", clip, step, [sample_rate])
    

Requirements

To pass the practicals, you need to obtain at least 80 points, excluding the bonus points. Note that all surplus points (both bonus and non-bonus) will be transfered to the exam. In total, assignments for at least 120 points (not including the bonus points) will be available, and if you solve all the assignments (any non-zero amount of points counts as solved), you automatically pass the exam with grade 1.

To pass the exam, you need to obtain at least 60, 75, or 90 points out of 100-point exam to receive a grade 3, 2, or 1, respectively. The exam consists of 100-point-worth questions from the list below (the questions are randomly generated, but in such a way that there is at least one question from every but the first lecture). In addition, you can get surplus points from the practicals and at most 10 points for community work (i.e., fixing slides or reporting issues) – but only the points you already have at the time of the exam count. You can take the exam without passing the practicals first.

Exam Questions

Related Courses

Machine Learning for Greenhorns

Introductory course to machine learning, focusing both on theoretical foundations as well as on practical applications in Python.

Deep Reinforcement Learning

Course introducing reinforcement learning, from basic tabular methods to involvement of deep neural networks, focusing both on theory as well as on practical aspects.

Archive

Summer 2023/24

Summer 2022/23

Summer 2021/22

Summer 2020/21

Summer 2019/20

Summer 2018/19

Summer 2017/18

Winter 2016/17