diff --git a/README.md b/README.md index 8d7bd60eab6eda63b2e3780e86c32e0dc26934a4..1e37299e4d113dc8dd8b0ac3d457d292a8f6bf2a 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ It is composed of the following steps: 3. Export the docker container for singularity. (Does this work on WSL?) 4. Recreate the environment on the supercomputers -# Install docker +# Install docker on your home machine This step depends on the OS you use. Please follow the instructions on the [docker web page](https://docs.docker.com/get-docker/). After that you should be able to start a simple docker container: ```bash @@ -17,7 +17,7 @@ docker run -it ubuntu bash You can leave the container by typing `exit`. On Windows, it is highly recommended to install the [Windows Subsystem Linux](https://docs.microsoft.com/en-us/windows/wsl/about). This will provide you with the WSL console, where you have a linux-like -environment. Please check that you can execute the commands above. +environment. Please check that you can execute the commands above. On MacOS and Linux, you already have the required environment. For convenience, we recommend enabling the option to run docker without the sudo command. On Linux, you can follow [this procedure](https://docs.docker.com/engine/install/linux-postinstall/). Otherwise, you will need to adjust the scripts in the `docker` subdirectory. @@ -54,33 +54,61 @@ the script `run_command_in_container.sh`. If you execute ```bash ./run_command_in_container python3 --version ``` -you will see which python version has been installed into the container. +it will invoke the Python interpreter that has been installed into the container image. You will see which python version has been installed into the container. Instead, you can, of course, execute any other command and pass arbitrary arguments. + +# Export the docker container +Before you start into this step, ssh to the supercomputer and also clone the repository there. +```bash +cd /path/to +git clone https://gitlab.jsc.fz-juelich.de/AI_Recipe_Book/recipes/singularity_docker_jupyter +cd singularity_docker_jupyter +``` -# Export the docker container for singularity In this step, you will export the docker image you have created as a singularity container. It requires the following steps: -1. Save the docker image in a tarball +**1.** Save the docker image in a tarball ```bash docker save singularity_docker_jupyter -o singularity_docker_jupyter.tar ``` -1. Copy the image to one of the JSC machines: +Note that a tarball `singularity_docker_jupyter.tar` has been created in your local directory. + +**2.** Copy the image to one of the JSC machines. In this example, we place the file into the directory where we have +clone the repo before. ```bash -scp singularity_docker_jupyter.tar surname1@jusuf.fz-juelich.de:/path/to/image +scp singularity_docker_jupyter.tar surname1@jusuf.fz-juelich.de:/path/to/singularity_docker_jupyter ``` Note that this can take a while, depending on your connection. -1. ssh to the machine and convert the tarball into a singularity image. +**3.** ssh to the machine and convert the tarball into a singularity image. ```bash ssh surname1@jusuf.fz-juelich.de -cd /path/to/image +cd /path/to/singularity_docker_jupyter +module load Singularity-Tools singularity build singularity_docker_jupyter.sif docker-archive://singularity_docker_jupyter.tar ``` +This will create a file `singularity_docker_jupyter.sif` in your local directory + If you local machine is a Linux machine, you also have the option to create the singularity image `singularity_docker_jupyter.sif` on your local machine. # Use the container with singularity on the supercomputer +We have created an example submission script, that you can submit to slurm. It is also possible to +execute it directly on the login node. Here is the output: + +Furthermore, you can create a jupyter kernel for jupyter-jsc. We have created a small script that will +do it for you. Execute +```bash +./create_kernel.sh +``` +and it will create a file `~/.local/share/jupyter/kernels/singularity_docker_jupyter/kernel.json` that contains the description +of a kernel. After logging out of Jupyter-JSC and logging in again, you will be able to pick a kernel that +was made from your own singularity container + +If you look at the file `jupyter-jsc/kernel.sh` you will see that it only is it a thin wrapper, that makes sure the Kernel +is executed in the singularity container. + -# Details about the container is started +# Details about the docker container is started The script `run_jupyter.sh` does a few things that are untypical when using docker. Here are the most important points. * We do not store any information in the container. Your home directory is mounted into the container. This is done by using `-v $HOME:$HOME`. It is mounted to the same path as on the host computer. This ensures not path inconsistencies occur. diff --git a/img/jupyter_container.png b/img/jupyter_container.png new file mode 100644 index 0000000000000000000000000000000000000000..14b6a2003c2c6a6fbccb6b2479d3d01ec753dfa2 Binary files /dev/null and b/img/jupyter_container.png differ diff --git a/img/kernel.png b/img/kernel.png new file mode 100644 index 0000000000000000000000000000000000000000..49cf48dfa2331f79bdaeeb1329da1286c3ab7d08 Binary files /dev/null and b/img/kernel.png differ diff --git a/img/singularity1.png b/img/singularity1.png new file mode 100644 index 0000000000000000000000000000000000000000..87137fc45117d829a01ccddbb6365e909af50be0 Binary files /dev/null and b/img/singularity1.png differ