diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..d874ad67cc881f97e9cbc302a3bf49e5c2686e46 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +*.tar diff --git a/README.md b/README.md index 435c28bbc9dfb272b8c728999c2f53fb865ed8c6..88794008e775216fa16c94d2d17898b2369a2c12 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,99 @@ # singularity_docker_jupyter +In this recipe, you will learn how to create your own container-based environment that +you can use at home on on the supercomputer. It is compatible with windows as well as Linux and MacOS, however only if your host is of X86 architecture (new Macs might pose a problem). +It is composed of the following steps: +0. Install docker +1. Create a docker container that contains the environment +2. Run the docker container to serve a local jupyter server and to execute programs +3. Export the docker container for singularity. (Does this work on WSL?) +4. Recreate the environment on the supercomputers + +#Install docker +This step depends on the OS you use. Please follow the instructions on the [docker web page](https://docs.docker.com/get-docker/). After that you should be able to +start a simple docker container: +```bash +docker run -it ubuntu bash +``` +You can leave the container by typing `exit`. + +On Windows, it is highly recommended to install the [Windows Subsystem Linux](https://docs.microsoft.com/en-us/windows/wsl/about). This will provide you with the WSL console, where you have a linux-like +environment. Please check that you can execute the commands above. + +For convenience, we recommend enabling the option to run docker without the sudo command. On Linux, you can follow [this procedure](https://docs.docker.com/engine/install/linux-postinstall/). Otherwise, you will need +to adjust the scripts in the `docker` subdirectory. + +# Create a docker image and container +In this step, you will create a custom docker image and a docker container that contains the environment. +First clone this repository, and `cd` into it. Please pick a good path for that. You might keep this repository for a long time. Here we assume, it is `/path/to`. +```bash +cd /path/to +git clone https://gitlab.jsc.fz-juelich.de/AI_Recipe_Book/recipes/singularity_docker_jupyter +cd singularity_docker_jupyter +``` +Everything related to the docker image is in the subdir `docker`. The rules to build the docker image are found in the [Dockerfile](docker/Dockerfile). Look inside, you will +see that we start from a plain ubuntu-image, install Python with `apt-get` and install Jupyter with `pip`. Build the docker container with the following commands +```bash +cd docker +./build.sh +``` +The build script will build the container and tag it with `singularity_docker_jupyter`. Once the container is build, you can run the jupyter server with the script `run.sh`: +```bash +./run_jupyter.sh +``` +The run script will start a docker container hosting a jupyter server that you can access by navigating to http://localhost:8889/. It will be restarted automatically when your system reboots or the container exits. In order to permanently +remove it, execute +```bash +docker rm -f singularity_docker_jupyter_cont +``` +We also have created small scripts to run commands and an interactive shell in the container. Execute +```bash +./run_bash_in_container.sh +``` +to run an interactive bash shell inside of the container. Exit it by typing `exit`. Executing commands is possible with +the script `run_command_in_container.sh`. If you execute +```bash +./run_command_in_container python3 --version +``` +you will see which python version has been installed into the container. + +# Export the docker container for singularity +In this step, you will export the docker image you have created as a singularity container. It requires the following steps: +1. Save the docker image in a tarball +```bash +docker save singularity_docker_jupyter -o singularity_docker_jupyter.tar +``` +1. Copy the image to one of the JSC machines: +```bash +scp singularity_docker_jupyter.tar surname1@jusuf.fz-juelich.de:/path/to/image +``` +Note that this can take a while, depending on your connection. +1. ssh to the machine and convert the tarball into a singularity image. +```bash +ssh surname1@jusuf.fz-juelich.de +cd /path/to/image +singularity build singularity_docker_jupyter.sif docker-archive://singularity_docker_jupyter.tar +``` +If you local machine is a Linux machine, you also have the option to create the singularity image `singularity_docker_jupyter.sif` on your local machine. + + +# Execute the container with singularity on the supercomputer + + + +# Details about the container is started +The script `run_jupyter.sh` does a few things that are untypical when using docker. Here are the most important points. +* We do not store any information in the container. Your home directory is mounted into the container. This is done by using `-v $HOME:$HOME`. It is mounted to the same path as on the +host computer. This ensures not path inconsistencies occur. +* The `HOST` environment variable is exported into the container. This is done with the option `-e HOME` +* The user id and group id are not set to `root`/0 as typical for docker, but with the option `--user $(id -u $USER):$(id -g $USER)` we make sure that UID and GID inside the container are the same as the ones of the user who starts the container. +All files modified in the container will be accessed with the UID and GID of the same user. If you are the only user, you will not even realize you are inside a container. +* The port 8888 of the container is mapped to the port 8889 of your local computer. The jupyter server started in the container by default serves on port 8888. To avoid conflicts with another potentially +running jupyter environment on your local machine, the container-based server serves on port 8889. +* The container is given the name `singularity_docker_jupyter_cont` + +We start the jupyter server with a few options +* We don't restrict IPs that can use the server +* We don't open a browser +* We disable access tokens +* We use the `$HOME` directory as base directory for the jupyter server. -How to create a custom container-based environment with jupyter that works at home an @JSC \ No newline at end of file diff --git a/docker/Dockerfile b/docker/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..b5ea8338b62fecbdfccc5a520afd54f6cc7f0949 --- /dev/null +++ b/docker/Dockerfile @@ -0,0 +1,6 @@ +from ubuntu:20.04 + +run apt-get update +run apt-get install -y python3-dev python3-pip python3-venv +run pip3 install --upgrade pip +run pip3 install jupyter diff --git a/docker/build.sh b/docker/build.sh new file mode 100755 index 0000000000000000000000000000000000000000..ac8195d545bdb12c908db31dba94ccbb19799ee7 --- /dev/null +++ b/docker/build.sh @@ -0,0 +1,2 @@ +#!/bin/bash +docker build --tag singularity_docker_jupyter . diff --git a/docker/run_bash_in_container.sh b/docker/run_bash_in_container.sh new file mode 100755 index 0000000000000000000000000000000000000000..b3578b5c3197b18cb5191d92c93f9ddc40c03242 --- /dev/null +++ b/docker/run_bash_in_container.sh @@ -0,0 +1,4 @@ +#!/bin/bash +USERMAPPING="--user $(id -u $USER):$(id -g $USER)" + +docker run $USERMAPPING --rm -e USER -e HOME -v $HOME:$HOME -it singularity_docker_jupyter bash "$@" diff --git a/docker/run_command_in_container.sh b/docker/run_command_in_container.sh new file mode 100755 index 0000000000000000000000000000000000000000..dc4d62e27289b3bf88513b9c106b8f35fb2f4a85 --- /dev/null +++ b/docker/run_command_in_container.sh @@ -0,0 +1,4 @@ +#!/bin/bash +USERMAPPING="--user $(id -u $USER):$(id -g $USER)" + +docker run $USERMAPPING --rm -e USER -e HOME -v $HOME:$HOME singularity_docker_jupyter "$@" diff --git a/docker/run_jupyter.sh b/docker/run_jupyter.sh new file mode 100755 index 0000000000000000000000000000000000000000..b618f12e9f1aa02b9b67323a5708ae74f4f81584 --- /dev/null +++ b/docker/run_jupyter.sh @@ -0,0 +1,5 @@ +#!/bin/bash +USERMAPPING="--user $(id -u $USER):$(id -g $USER)" + +docker run -d --restart always $USERMAPPING -e HOME -v $HOME:$HOME -p 8889:8888 --name singularity_docker_jupyter_cont singularity_docker_jupyter \ + bash -c "jupyter notebook --ip=0.0.0.0 --no-browser --NotebookApp.token='' --notebook-dir=$HOME"