Skip to content
Snippets Groups Projects
Commit 30f2b968 authored by Tim Kreuzer's avatar Tim Kreuzer
Browse files

update dask parameter

parent 7961a3fa
Branches
Tags
No related merge requests found
Show whitespace changes
Inline Side-by-side
001-Jupyter/002-JupyterExtensions/Dask_JURECA.ipynb
+ 2
− 2
View file @ 30f2b968
%% Cell type:markdown id: tags:
# Dask Extension
%% Cell type:markdown id: tags:
## If you have problems with this tutorial, try to download the Notebook.
%% Cell type:code id: tags:
``` python
!wget https://jupyter-jsc.fz-juelich.de/static/files/Dask_JURECA.ipynb
```
%% Cell type:markdown id: tags:
This notebook will give you a short introduction into the Dask Extension on JURECA. It allows you to run Jobs on the compute nodes, even if your JupyterLab is running interactively on the login node.
First you have to define on which project and partition it should be running.
%% Cell type:code id: tags:
``` python
queue = "batch" # batch, gpus, develgpus, etc.
project = "zam" # your project: zam, training19xx, etc.
```
%% Cell type:markdown id: tags:
# Monte-Carlo Estimate of $\pi$
We want to estimate the number $\pi$ using a [Monte-Carlo method](https://en.wikipedia.org/wiki/Pi#Monte_Carlo_methods) exploiting that the area of a quarter circle of unit radius is $\pi/4$ and that hence the probability of any randomly chosen point in a unit square to lie in a unit circle centerd at a corner of the unit square is $\pi/4$ as well. So for N randomly chosen pairs $(x, y)$ with $x\in[0, 1)$ and $y\in[0, 1)$, we count the number $N_{circ}$ of pairs that also satisfy $(x^2 + y^2) < 1$ and estimage $\pi \approx 4 \cdot N_{circ} / N$.
[<img src="https://upload.wikimedia.org/wikipedia/commons/8/84/Pi_30K.gif"
width="50%"
align=top
alt="PI monte-carlo estimate">](https://en.wikipedia.org/wiki/Pi#Monte_Carlo_methods)
%% Cell type:markdown id: tags:
## Core Lessons
- setting up SLURM (and other jobqueue) clusters
- Scaling clusters
- Adaptive clusters
%% Cell type:markdown id: tags:
## Set up a Slurm cluster
We'll create a SLURM cluster and have a look at the job-script used to start workers on the HPC scheduler.
%% Cell type:code id: tags:
``` python
import dask
from dask.distributed import Client
from dask_jobqueue import SLURMCluster
import os
cluster = SLURMCluster(
cores=24,
processes=2,
memory="100GB",
shebang='#!/usr/bin/env bash',
queue=queue,
dashboard_address=":56755",
scheduler_options={"dashboard_address": ":56755"},
walltime="00:30:00",
local_directory='/tmp',
death_timeout="15s",
interface="ib0",
log_directory=f'{os.environ["HOME"]}/dask_jobqueue_logs/',
project=project)
```
%% Cell type:code id: tags:
``` python
print(cluster.job_script())
```
%% Cell type:code id: tags:
``` python
client = Client(cluster)
client
```
%% Cell type:markdown id: tags:
## You can visit the Dask Dashboard at the following url:
```
https://jupyter-jsc.fz-juelich.de/user/<user_name>/<lab_name>/proxy/<port>/status
```
%% Cell type:markdown id: tags:
## You can integrate it into your JupyterLab environment by putting the link into the Dask Extension
%% Cell type:markdown id: tags:
!["Dask"](https://zam10183.zam.kfa-juelich.de/hub/static/images/dask2.png "dask")
%% Cell type:markdown id: tags:
Afterwards you can press on the orange buttons to open a new tab in your JupyterLab Environment.
%% Cell type:markdown id: tags:
## Scale the cluster to two nodes
A look at the Dashboard reveals that there are no workers in the clusetr. Let's start 4 workers (in 2 SLURM jobs).
For the distiction between _workers_ and _jobs_, see [the Dask jobqueue docs](https://jobqueue.dask.org/en/latest/howitworks.html#workers-vs-jobs).
%% Cell type:code id: tags:
``` python
cluster.scale(4) # scale to 4 _workers_
```
%% Cell type:markdown id: tags:
## The Monte Carlo Method
%% Cell type:code id: tags:
``` python
import dask.array as da
import numpy as np
def calc_pi_mc(size_in_bytes, chunksize_in_bytes=200e6):
"""Calculate PI using a Monte Carlo estimate."""
size = int(size_in_bytes / 8)
chunksize = int(chunksize_in_bytes / 8)
xy = da.random.uniform(0, 1, size=(size / 2, 2), chunks=(chunksize / 2, 2))
in_circle = (xy ** 2).sum(axis=-1) < 1
pi = 4 * in_circle.mean()
return pi
def print_pi_stats(size, pi, time_delta, num_workers):
"""Print pi, calculate offset from true value, and print some stats."""
print(
f"{size / 1e9} GB\n"
f"\tMC pi: {pi : 13.11f}"
f"\tErr: {abs(pi - np.pi) : 10.3e}\n"
f"\tWorkers: {num_workers}"
f"\t\tTime: {time_delta : 7.3f}s"
)
```
%% Cell type:markdown id: tags:
## The actual calculations
We loop over different volumes of double-precision random numbers and estimate $\pi$ as described above.
%% Cell type:code id: tags:
``` python
from time import time, sleep
```
%% Cell type:code id: tags:
``` python
for size in (1e9 * n for n in (1, 10, 100)):
start = time()
pi = calc_pi_mc(size).compute()
elaps = time() - start
print_pi_stats(
size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers)
)
```
%% Cell type:markdown id: tags:
## Scaling the Cluster to twice its size
We increase the number of workers by 2 and the re-run the experiments.
%% Cell type:code id: tags:
``` python
new_num_workers = 2 * len(cluster.scheduler.workers)
print(f"Scaling from {len(cluster.scheduler.workers)} to {new_num_workers} workers.")
cluster.scale(new_num_workers)
sleep(10)
```
%% Cell type:code id: tags:
``` python
client
```
%% Cell type:markdown id: tags:
## Re-run same experiments with doubled cluster
%% Cell type:code id: tags:
``` python
for size in (1e9 * n for n in (1, 10, 100)):
start = time()
pi = calc_pi_mc(size).compute()
elaps = time() - start
print_pi_stats(
size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers)
)
```
%% Cell type:markdown id: tags:
## Automatically Scaling the Cluster
We want each calculation to take only a few seconds. Dask will try to add more workers to the cluster when workloads are high and remove workers when idling.
_**Watch** how the cluster will scale down to the minimum a few seconds after being made adaptive._
%% Cell type:code id: tags:
``` python
ca = cluster.adapt(minimum=4, maximum=100)
sleep(4) # Allow for scale-down
```
%% Cell type:code id: tags:
``` python
client
```
%% Cell type:markdown id: tags:
## Repeat the calculation from above with larger work loads
(And watch the dash board!)
%% Cell type:code id: tags:
``` python
for size in (n * 1e9 for n in (1, 10, 100)):
start = time()
pi = calc_pi_mc(size, min(size / 1000, 500e6)).compute()
elaps = time() - start
print_pi_stats(
size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers)
)
sleep(20) # allow for scale-down time
```
......
001-Jupyter/002-JupyterExtensions/Dask_JURON.ipynb
+ 2
− 2
View file @ 30f2b968
%% Cell type:markdown id: tags:
# Dask Extension
%% Cell type:markdown id: tags:
## If you have problems with this tutorial, try to download the Notebook.
%% Cell type:code id: tags:
``` python
!wget https://jupyter-jsc.fz-juelich.de/static/files/Dask_JURON.ipynb
```
%% Cell type:markdown id: tags:
This notebook will give you a short introduction into the Dask Extension on JURON. It allows you to run Jobs on the compute nodes, even if your JupyterLab is running interactively on the login node.
%% Cell type:markdown id: tags:
# Monte-Carlo Estimate of $\pi$
We want to estimate the number $\pi$ using a [Monte-Carlo method](https://en.wikipedia.org/wiki/Pi#Monte_Carlo_methods) exploiting that the area of a quarter circle of unit radius is $\pi/4$ and that hence the probability of any randomly chosen point in a unit square to lie in a unit circle centerd at a corner of the unit square is $\pi/4$ as well. So for N randomly chosen pairs $(x, y)$ with $x\in[0, 1)$ and $y\in[0, 1)$, we count the number $N_{circ}$ of pairs that also satisfy $(x^2 + y^2) < 1$ and estimage $\pi \approx 4 \cdot N_{circ} / N$.
[<img src="https://upload.wikimedia.org/wikipedia/commons/8/84/Pi_30K.gif"
width="50%"
align=top
alt="PI monte-carlo estimate">](https://en.wikipedia.org/wiki/Pi#Monte_Carlo_methods)
%% Cell type:markdown id: tags:
## Core Lessons
- setting up SLURM (and other jobqueue) clusters
- Scaling clusters
- Adaptive clusters
%% Cell type:markdown id: tags:
## Set up a Slurm cluster
We'll create a SLURM cluster and have a look at the job-script used to start workers on the HPC scheduler.
%% Cell type:code id: tags:
``` python
import dask
from dask.distributed import Client
from dask_jobqueue import LSFCluster
import os
dask.config.set({"jobqueue.lsf.use-stdin": True})
cluster = LSFCluster(
queue="normal",
walltime="60",
ncpus=2,
host="192.168.45.25",
dashboard_address="0.0.0.0:56755",
scheduler_options={"dashboard_address": "0.0.0.0:56755"},
death_timeout="15s",
mem=4 * 1024 * 1024 * 1024,
log_directory="{}/dask_jobqueue_logs".format(os.getenv("HOME")),
cores=4,
locals_directory="/tmp",
n_workers=4,
memory="128GB",
usestd_in=True
)
```
%% Cell type:code id: tags:
``` python
print(cluster.job_script())
```
%% Cell type:code id: tags:
``` python
client = Client(cluster)
client
```
%% Cell type:markdown id: tags:
## You can visit the Dask Dashboard at the following url:
```
https://jupyter-jsc.fz-juelich.de/user/<user_name>/<lab_name>/proxy/<port>/status
```
%% Cell type:markdown id: tags:
## You can integrate it into your JupyterLab environment by putting the link into the Dask Extension
%% Cell type:markdown id: tags:
!["Dask"](https://zam10183.zam.kfa-juelich.de/hub/static/images/dask2.png "dask")
%% Cell type:markdown id: tags:
Afterwards you can press on the orange buttons to open a new tab in your JupyterLab Environment.
%% Cell type:markdown id: tags:
## Scale the cluster to two nodes
A look at the Dashboard reveals that there are no workers in the clusetr. Let's start 4 workers (in 2 SLURM jobs).
For the distiction between _workers_ and _jobs_, see [the Dask jobqueue docs](https://jobqueue.dask.org/en/latest/howitworks.html#workers-vs-jobs).
%% Cell type:code id: tags:
``` python
cluster.scale(4) # scale to 4 _workers_
```
%% Cell type:markdown id: tags:
## The Monte Carlo Method
%% Cell type:code id: tags:
``` python
import dask.array as da
import numpy as np
def calc_pi_mc(size_in_bytes, chunksize_in_bytes=200e6):
"""Calculate PI using a Monte Carlo estimate."""
size = int(size_in_bytes / 8)
chunksize = int(chunksize_in_bytes / 8)
xy = da.random.uniform(0, 1, size=(size / 2, 2), chunks=(chunksize / 2, 2))
in_circle = (xy ** 2).sum(axis=-1) < 1
pi = 4 * in_circle.mean()
return pi
def print_pi_stats(size, pi, time_delta, num_workers):
"""Print pi, calculate offset from true value, and print some stats."""
print(
f"{size / 1e9} GB\n"
f"\tMC pi: {pi : 13.11f}"
f"\tErr: {abs(pi - np.pi) : 10.3e}\n"
f"\tWorkers: {num_workers}"
f"\t\tTime: {time_delta : 7.3f}s"
)
```
%% Cell type:markdown id: tags:
## The actual calculations
We loop over different volumes of double-precision random numbers and estimate $\pi$ as described above.
%% Cell type:code id: tags:
``` python
from time import time, sleep
```
%% Cell type:code id: tags:
``` python
for size in (1e9 * n for n in (1, 10, 100)):
start = time()
pi = calc_pi_mc(size).compute()
elaps = time() - start
print_pi_stats(
size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers)
)
```
%% Cell type:markdown id: tags:
## Scaling the Cluster to twice its size
We increase the number of workers by 2 and the re-run the experiments.
%% Cell type:code id: tags:
``` python
new_num_workers = 2 * len(cluster.scheduler.workers)
print(f"Scaling from {len(cluster.scheduler.workers)} to {new_num_workers} workers.")
cluster.scale(new_num_workers)
sleep(10)
```
%% Cell type:code id: tags:
``` python
client
```
%% Cell type:markdown id: tags:
## Re-run same experiments with doubled cluster
%% Cell type:code id: tags:
``` python
for size in (1e9 * n for n in (1, 10, 100)):
start = time()
pi = calc_pi_mc(size).compute()
elaps = time() - start
print_pi_stats(
size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers)
)
```
%% Cell type:markdown id: tags:
## Automatically Scaling the Cluster
We want each calculation to take only a few seconds. Dask will try to add more workers to the cluster when workloads are high and remove workers when idling.
_**Watch** how the cluster will scale down to the minimum a few seconds after being made adaptive._
%% Cell type:code id: tags:
``` python
ca = cluster.adapt(minimum=4, maximum=100)
sleep(4) # Allow for scale-down
```
%% Cell type:code id: tags:
``` python
client
```
%% Cell type:markdown id: tags:
## Repeat the calculation from above with larger work loads
(And watch the dash board!)
%% Cell type:code id: tags:
``` python
for size in (n * 1e9 for n in (1, 10, 100)):
start = time()
pi = calc_pi_mc(size, min(size / 1000, 500e6)).compute()
elaps = time() - start
print_pi_stats(
size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers)
)
sleep(20) # allow for scale-down time
```
......
001-Jupyter/002-JupyterExtensions/Dask_JUWELS.ipynb
+ 2
− 2
View file @ 30f2b968
%% Cell type:markdown id: tags:
# Dask Extension
%% Cell type:markdown id: tags:
## If you have problems with this tutorial, try to download the Notebook.
%% Cell type:code id: tags:
``` python
!wget https://jupyter-jsc.fz-juelich.de/static/files/Dask_JUWELS.ipynb
```
%% Cell type:markdown id: tags:
This notebook will give you a short introduction into the Dask Extension on JURECA. It allows you to run Jobs on the compute nodes, even if your JupyterLab is running interactively on the login node.
First you have to define on which project and partition it should be running.
%% Cell type:code id: tags:
``` python
queue = "batch" # batch, gpus, develgpus, etc.
project = "zam" # your project: zam, training19xx, etc.
```
%% Cell type:markdown id: tags:
# Monte-Carlo Estimate of $\pi$
We want to estimate the number $\pi$ using a [Monte-Carlo method](https://en.wikipedia.org/wiki/Pi#Monte_Carlo_methods) exploiting that the area of a quarter circle of unit radius is $\pi/4$ and that hence the probability of any randomly chosen point in a unit square to lie in a unit circle centerd at a corner of the unit square is $\pi/4$ as well. So for N randomly chosen pairs $(x, y)$ with $x\in[0, 1)$ and $y\in[0, 1)$, we count the number $N_{circ}$ of pairs that also satisfy $(x^2 + y^2) < 1$ and estimage $\pi \approx 4 \cdot N_{circ} / N$.
[<img src="https://upload.wikimedia.org/wikipedia/commons/8/84/Pi_30K.gif"
width="50%"
align=top
alt="PI monte-carlo estimate">](https://en.wikipedia.org/wiki/Pi#Monte_Carlo_methods)
%% Cell type:markdown id: tags:
## Core Lessons
- setting up SLURM (and other jobqueue) clusters
- Scaling clusters
- Adaptive clusters
%% Cell type:markdown id: tags:
## Set up a Slurm cluster
We'll create a SLURM cluster and have a look at the job-script used to start workers on the HPC scheduler.
%% Cell type:code id: tags:
``` python
import dask
from dask.distributed import Client
from dask_jobqueue import SLURMCluster
import os
cluster = SLURMCluster(
cores=24,
processes=2,
memory="100GB",
shebang="#!/usr/bin/env bash",
queue=queue,
dashboard_address=":56755",
scheduler_options={"dashboard_address": ":56755"},
walltime="00:30:00",
local_directory="/tmp",
death_timeout="15s",
interface="ib1",
log_directory=f'{os.environ["HOME"]}/dask_jobqueue_logs/',
project=project,
)
```
%% Cell type:code id: tags:
``` python
print(cluster.job_script())
```
%% Cell type:code id: tags:
``` python
client = Client(cluster)
client
```
%% Cell type:markdown id: tags:
## You can visit the Dask Dashboard at the following url:
```
https://jupyter-jsc.fz-juelich.de/user/<user_name>/<lab_name>/proxy/<port>/status
```
%% Cell type:markdown id: tags:
## You can integrate it into your JupyterLab environment by putting the link into the Dask Extension
%% Cell type:markdown id: tags:
!["Dask"](https://zam10183.zam.kfa-juelich.de/hub/static/images/dask2.png "dask")
%% Cell type:markdown id: tags:
Afterwards you can press on the orange buttons to open a new tab in your JupyterLab Environment.
%% Cell type:markdown id: tags:
## Scale the cluster to two nodes
A look at the Dashboard reveals that there are no workers in the clusetr. Let's start 4 workers (in 2 SLURM jobs).
For the distiction between _workers_ and _jobs_, see [the Dask jobqueue docs](https://jobqueue.dask.org/en/latest/howitworks.html#workers-vs-jobs).
%% Cell type:code id: tags:
``` python
cluster.scale(4) # scale to 4 _workers_
```
%% Cell type:markdown id: tags:
## The Monte Carlo Method
%% Cell type:code id: tags:
``` python
import dask.array as da
import numpy as np
def calc_pi_mc(size_in_bytes, chunksize_in_bytes=200e6):
"""Calculate PI using a Monte Carlo estimate."""
size = int(size_in_bytes / 8)
chunksize = int(chunksize_in_bytes / 8)
xy = da.random.uniform(0, 1, size=(size / 2, 2), chunks=(chunksize / 2, 2))
in_circle = (xy ** 2).sum(axis=-1) < 1
pi = 4 * in_circle.mean()
return pi
def print_pi_stats(size, pi, time_delta, num_workers):
"""Print pi, calculate offset from true value, and print some stats."""
print(
f"{size / 1e9} GB\n"
f"\tMC pi: {pi : 13.11f}"
f"\tErr: {abs(pi - np.pi) : 10.3e}\n"
f"\tWorkers: {num_workers}"
f"\t\tTime: {time_delta : 7.3f}s"
)
```
%% Cell type:markdown id: tags:
## The actual calculations
We loop over different volumes of double-precision random numbers and estimate $\pi$ as described above.
%% Cell type:code id: tags:
``` python
from time import time, sleep
```
%% Cell type:code id: tags:
``` python
for size in (1e9 * n for n in (1, 10, 100)):
start = time()
pi = calc_pi_mc(size).compute()
elaps = time() - start
print_pi_stats(
size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers)
)
```
%% Cell type:markdown id: tags:
## Is it running?
%% Cell type:markdown id: tags:
To check if something has been started for you just use the following command in a terminal:
```
squeue | grep ${USER}
```
%% Cell type:markdown id: tags:
## Scaling the Cluster to twice its size
We increase the number of workers by 2 and the re-run the experiments.
%% Cell type:code id: tags:
``` python
new_num_workers = 2 * len(cluster.scheduler.workers)
print(f"Scaling from {len(cluster.scheduler.workers)} to {new_num_workers} workers.")
cluster.scale(new_num_workers)
sleep(10)
```
%% Cell type:code id: tags:
``` python
client
```
%% Cell type:markdown id: tags:
## Re-run same experiments with doubled cluster
%% Cell type:code id: tags:
``` python
for size in (1e9 * n for n in (1, 10, 100)):
start = time()
pi = calc_pi_mc(size).compute()
elaps = time() - start
print_pi_stats(
size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers)
)
```
%% Cell type:markdown id: tags:
## Automatically Scaling the Cluster
We want each calculation to take only a few seconds. Dask will try to add more workers to the cluster when workloads are high and remove workers when idling.
_**Watch** how the cluster will scale down to the minimum a few seconds after being made adaptive._
%% Cell type:code id: tags:
``` python
ca = cluster.adapt(minimum=4, maximum=100)
sleep(4) # Allow for scale-down
```
%% Cell type:code id: tags:
``` python
client
```
%% Cell type:markdown id: tags:
## Repeat the calculation from above with larger work loads
(And watch the dash board!)
%% Cell type:code id: tags:
``` python
for size in (n * 1e9 for n in (1, 10, 100)):
start = time()
pi = calc_pi_mc(size, min(size / 1000, 500e6)).compute()
elaps = time() - start
print_pi_stats(
size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers)
)
sleep(20) # allow for scale-down time
```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment