diff --git a/README.md b/README.md index 1e37299e4d113dc8dd8b0ac3d457d292a8f6bf2a..32dfaafeb0c24a0ee697ec1d8a48b76d815270d9 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,26 @@ # Singularity + Docker + Jupyter In this recipe, you will learn how to create your own container-based environment that -you can use at home and on the supercomputer. It is compatible with windows as well as Linux and MacOS, however only if your host is of X86 architecture (new Macs might pose a problem). -It is composed of the following steps: +you can use at home and on the supercomputer. In this picture it is outlined: + +You will create a docker-based environment on your machine, that can be use to serve +a jupyter server. Then you transfer the docker image to the supercomputer and make it available +within Jupyter-JSC. Of course, this works without Jupyter, but is much less fun. + +In General, the usage of singularity containers on the HPC is highly recommended. If the containers from the NVIDIA container registry +[link](https://catalog.ngc.nvidia.com/) are used, you can be quite sure that you are as fast as it gets. + +The workflow is compatible with windows as well as Linux and MacOS, however only if your host is of X86 architecture (new Macs might pose a problem). +We however did not get a chance to try. + +You will perform the following steps: + 0. Install docker 1. Create a docker container that contains the environment 2. Run the docker container to serve a local jupyter server and to execute programs -3. Export the docker container for singularity. (Does this work on WSL?) -4. Recreate the environment on the supercomputers +3. Export the docker image, transfer it to the supercomputer and convert it to a singularity container +4. Run things in the singularity container. + +Finally, we say a few words about recommended workflows and give a few details that we have omitted before. # Install docker on your home machine This step depends on the OS you use. Please follow the instructions on the [docker web page](https://docs.docker.com/get-docker/). After that you should be able to @@ -65,6 +79,7 @@ cd singularity_docker_jupyter ``` In this step, you will export the docker image you have created as a singularity container. It requires the following steps: + **1.** Save the docker image in a tarball ```bash docker save singularity_docker_jupyter -o singularity_docker_jupyter.tar @@ -88,9 +103,9 @@ This will create a file `singularity_docker_jupyter.sif` in your local directory If you local machine is a Linux machine, you also have the option to create the singularity image `singularity_docker_jupyter.sif` on your local machine. - # Use the container with singularity on the supercomputer -We have created an example submission script, that you can submit to slurm. It is also possible to +The typical usage of a singularity container is `singularity run image.sif command --with-args`. We demonstrate the usage +in an example submission script `example_submission_script.sh`, that you can submit to slurm. It is also possible to execute it directly on the login node. Here is the output:  @@ -107,6 +122,11 @@ If you look at the file `jupyter-jsc/kernel.sh` you will see that it only is it is executed in the singularity container.  +# Workflow +Your first shot at creating an environment will never be your last one. Creating an environment is an iterative task, and you'll soon find a library or a tool that is required in +your workflow. An option that speeds up your productivity might be using a virtual environment that is stored outside of the container. This makes it much easier to add libraries. +In another recipe, we have covered, how to do this. In General, we discourage the use of `docker commit`, even though this might feel like a very easy way to develop an environment step by step. However, you give +up the great reproducibility you achieve with docker files. # Details about the docker container is started The script `run_jupyter.sh` does a few things that are untypical when using docker. Here are the most important points. diff --git a/img/Presentation1.pptx b/img/Presentation1.pptx new file mode 100644 index 0000000000000000000000000000000000000000..6f2ed37331adf62e798a1d905ddcee0dfad783e7 Binary files /dev/null and b/img/Presentation1.pptx differ diff --git a/img/outline.png b/img/outline.png new file mode 100644 index 0000000000000000000000000000000000000000..32bbfa5ef9b26aa6a5e9db67e7956f35cd3ee6b3 Binary files /dev/null and b/img/outline.png differ