Skip to content
Snippets Groups Projects
Commit a23c334b authored by Jens Henrik Goebbert's avatar Jens Henrik Goebbert
Browse files

add day 2

parent 019e4ae9
No related branches found
No related tags found
No related merge requests found
Showing
with 7785 additions and 0 deletions
File added
Source diff could not be displayed: it is too large. Options to address this: view the blob.
Source diff could not be displayed: it is too large. Options to address this: view the blob.
File added
%% Cell type:markdown id:racial-equality tags:
![jsc-logo.jpg](attachment:67258d94-84e6-4a0c-ae8f-c74332ec082e.jpg)
Author: [Jens Henrik Göbbert](mailto:j.goebbert@fz-juelich.de)
------------------------------------
%% Cell type:markdown id:republican-throat tags:
## Jupyter Kernel
List of available Jupyer kernel:
https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
-------------------------------------------------------
%% Cell type:markdown id:tested-acceptance tags:
#### Excercise 1:
Find all current search paths for Jupyter kernel
[more](https://jupyter.readthedocs.io/en/latest/use/jupyter-directories.html#envvar-JUPYTER_PATH)
%% Cell type:code id:conditional-sphere tags:
``` python
!jupyter --paths
```
%% Output
config:
/home/jovyan/.jupyter
/p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/etc/jupyter
/home/jovyan/.jupyter
/p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/etc/jupyter
/usr/local/etc/jupyter
/etc/jupyter
data:
/home/jovyan/.local/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Ruby/3.0.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-R/4.1.2-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-PyVisualization/1.0-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-PyQuantum/3.0-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-PyDeepLearning/1.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Octave/6.4.0-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Julia/1.7.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-JavaScript/5.2.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Cling/0.9-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Bash/0.7.2-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/share/jupyter
/home/jovyan/.local/share/jupyter
/p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/share/jupyter
/usr/local/share/jupyter
/usr/share/jupyter
runtime:
/home/jovyan/.local/share/jupyter/runtime
%% Cell type:markdown id:66bfbb03-f262-4e39-998c-e04ef362f842 tags:
------------------------------------
#### Exercise 2:
Check how JupyterLab knows all those paths.
- JupyerLab searches for kernels in all paths listed in `$JUPYTER_PATH`
%% Cell type:code id:right-funds tags:
``` python
!echo $JUPYTER_PATH | tr ":" "\n"
```
%% Output
/home/jovyan/.local/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Ruby/3.0.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-R/4.1.2-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-PyVisualization/1.0-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-PyQuantum/3.0-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-PyDeepLearning/1.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Octave/6.4.0-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Julia/1.7.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-JavaScript/5.2.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Cling/0.9-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Bash/0.7.2-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/share/jupyter
%% Cell type:markdown id:b4d73dc8-a56d-42ac-bd8d-4330f0da96b6 tags:
----------------------------------
#### Exercise 3:
Check how a normal kernel definition directory looks like.
%% Cell type:code id:potential-enforcement tags:
``` python
!ls -la /p/software/hdfcloud/stages/2022/software/JupyterKernel-Julia/1.7.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter/kernels/julia-1.7
```
%% Output
total 12
drwxr-sr-x 2 6304 6304 69 Sep 2 2022 .
drwxr-sr-x 3 6304 6304 23 Sep 2 2022 ..
-rw-r--r-- 1 6304 6304 448 Sep 2 2022 kernel.json
-r--r--r-- 1 6304 6304 1310 Sep 2 2022 logo-32x32.png
-r--r--r-- 1 6304 6304 2843 Sep 2 2022 logo-64x64.png
%% Cell type:markdown id:47c00abf-3975-4503-95aa-c80af62f9dc2 tags:
--------------------------------
#### Exercise 4:
Check how a normal `kernel.json` looks like.
%% Cell type:code id:continuing-component tags:
``` python
!cat /p/software/hdfcloud/stages/2022/software/JupyterKernel-Julia/1.7.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter/kernels/julia-1.7/kernel.json
```
%% Output
{
"display_name": "Julia 1.7.1",
"argv": [
"/p/software/hdfcloud/stages/2022/software/Julia/1.7.1-gcccoremkl-11.2.0-2021.4.0/bin/julia",
"-i",
"--color=yes",
"--project=@.",
"/p/software/hdfcloud/stages/2022/software/JupyterKernel-Julia/1.7.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/extensions/packages/IJulia/AQu2H/src/kernel.jl",
"{connection_file}"
],
"language": "julia",
"env": {},
"interrupt_mode": "signal"
}
%% Cell type:markdown id:0aa6e61b-a33a-4e7c-8591-0506aa2eae0b tags:
---------------------------------
%% Cell type:code id:66939f26-015d-442b-a9b8-caed967513d0 tags:
``` python
```
This diff is collapsed.
This diff is collapsed.
%% Cell type:markdown id: tags:
![fzj-jsc_logo.jpg](attachment:f3736838-48a6-4450-9c91-5730ded4aca0.jpg)
Author: [Jonathan Windgassen](mailto:j.windgassen@fz-juelich.de)
------------------------------------
%% Cell type:markdown id: tags:
# How to modify/extend a running Jupyter Kernel
There are many cases where one needs modules from JupyterLab for a project. But building your own kernel is often a detour from the original idea or is annoying when publishing your project.
By adding these 4 cells to the top of your project you can load modules for the project "on the fly".
Besides that this also adds a ways of installing python packages via pip without disrupting the uses packages or access to the system site-packages
-------
%% Cell type:markdown id: tags:
- First we create to temp-folders in the */tmp* directory, who will contain the venv where we install the required packages and a folder that stores the PYTHONPATH and LD_LIBRARY_PATH environment variable. This is necessary because loading modules manipulates these variables but we can't access the changes from within python, so we store the changed variables in a folder.
- Then we use a bash-shell to:
- Load the Modules
- Create a venv and installing ipykernel in there
- Write PYTHONPATH and LD_LIBRARY_PATH to the tempdir
- Beacause the Dynamic Linker of Python doesn't detect changes in LD_LIBRARY_PATH we need to reboot the Interpreter afterwards to carry these changes over. To gain access to the venv we will start Python from there.
- After that we install the required modules.
**Note**: The third cell **won't** show that it's completed and the Notebook will show `Python 3 | Starting` at the bottom, although the interpreter already reloaded compeltely. You can savely ignore this and continue with the third shell. As soon as this has finished the Notebook will show `Python3 | Idle` again.
%% Cell type:code id: tags:
``` python
import os, sys, tempfile
tempdir = tempfile.mkdtemp()
venv_folder = tempfile.mkdtemp()
print(tempdir, venv_folder)
```
%% Cell type:code id: tags:
``` python
%%bash -s "$tempdir" "$venv_folder" # Pass the paths to the bash-subshell
######################################################################
## The modules go here ##
## We will use Stage/2022 with Python 3.9 as a example ##
######################################################################
# Update to Stage/2022
module --force purge
module load Stages/2022
module load GCCcore/.11.2.0
module load Python/3.9.6
# Create a venv with the python from stage 2022 and install ipykernel there (needed for communicating with Jupyter)
# If you don't change Python above this should be a normal Python 3.9 venv
python -m venv --system-site-packages $2
source $2/bin/activate
pip install ipykernel==6.15.3
# Store the new variables to the temp-folder
echo $PYTHONPATH > $1/pythonpath
echo $LD_LIBRARY_PATH > $1/librarypath
```
%% Cell type:markdown id: tags:
### CURRENTLY FAILS - needs to be fixed
%% Cell type:code id: tags:
``` python
# The arguments for the new python interpreter. We need to initialize ipykernel or JupyterLab will fail to integrate the new process
args = [f"{venv_folder}/bin/python", "-m", "ipykernel"]
args.extend(sys.argv)
# Because we call "execve" instead of "execv" we get the option to set the environment variables in the process. We use this to smuggle in the changed LD_LIBRARY_PATH and PYTHONPATH.
# You also can pass the location of the 2 temp-folders as new environment variables if you want to delete them later for cleanup.
env = {"PYTHONPATH": open(f"{tempdir}/pythonpath").read(),
"LD_LIBRARY_PATH": open(f"{tempdir}/librarypath").read(),
"tempdir": tempdir,
"venv_folder": venv_folder}
!echo Restarting Interpreter from $venv_folder/bin/python. Please execute the next cell
os.execve(f"{venv_folder}/bin/python", args, env)
```
%% Cell type:code id: tags:
``` python
# Because we are in the venv now, we can safely install all packages that we need and don't come with the Python3-Kernel. No need to add --user
%pip install --quiet ...
```
%% Cell type:markdown id: tags:
-----
After that you can import all you libraries (remember that the Interpreter restarted and you need to reimport os/sys/tempfile if you need them) and start with the notebook
%% Cell type:markdown id:technical-milan tags:
Example from https://datascience-enthusiast.com/Miscellaneous/Jupyter_R_Python_Julia_Octave.html
%% Cell type:code id:confirmed-mistake tags:
``` julia
using Plots
r(θ)=sin.(θ).*sqrt.(abs.(cos.(θ)))./(sin.(θ).+7/5)-2sin.(θ).+2
for i in -6:6
θ=π+i*π/6;
y=r(θ);
end
θ=range(0,stop=2*π,length=1000)
x=r(θ).*cos.(θ)
y=r(θ).*sin.(θ)
pyplot()
plot(x,y, linecolor=:red, leg=false, xlims = (-3,3), ylims=(-4,1), title = "Heart with Julia")
```
%% Cell type:code id:australian-huntington tags:
``` julia
```
%% Cell type:markdown id:latter-print tags:
Example from https://datascience-enthusiast.com/Miscellaneous/Jupyter_R_Python_Julia_Octave.html
%% Cell type:code id:precious-career tags:
``` python
t=-10:0.01:10;
x=16*(power(sin(t),3));
y=(13*cos(t))-(5*cos(2*t))-(2*cos(3*t))-(cos(4*t));
plot(x,y)
title("Heart with Octave", 'Color','k','fontsize',18)
```
%% Output
warning: using the gnuplot graphics toolkit is discouraged
The gnuplot graphics toolkit is not actively maintained and has a number
of limitations that are ulikely to be fixed. Communication with gnuplot
uses a one-directional pipe and limited information is passed back to the
Octave interpreter so most changes made interactively in the plot window
will not be reflected in the graphics properties managed by Octave. For
example, if the plot window is closed with a mouse click, Octave will not
be notified and will not update it's internal list of open figure windows.
We recommend using the qt toolkit instead.
This diff is collapsed.
This diff is collapsed.
File added
This diff is collapsed.
This diff is collapsed.
File added
File added
%% Cell type:markdown id:diverse-activity tags:
![jsc-logo.jpg](attachment:67258d94-84e6-4a0c-ae8f-c74332ec082e.jpg)
Author: [Jens Henrik Göbbert](mailto:j.goebbert@fz-juelich.de)
------------------------------------
%% Cell type:markdown id:hourly-valentine tags:
### Accessing Arbitrary Ports or Hosts
If you already have a server running on localhost listening on a port, you can access it through the notebook at `<notebook-base>/proxy/<port>`. The URL will be rewritten to remove the above prefix.
You can disable URL rewriting by using `<notebook-base>/proxy/absolute/<port>` so your server will receive the full URL in the request.
This works for all ports listening on the local machine.
You can also specify arbitrary hosts in order to proxy traffic from another machine on the network `<notebook-base>/proxy/<host>:<port>`.
For security reasons the host must match an entry in the `host_allowlist` in your configuration.
[more](https://jupyter-server-proxy.readthedocs.io/en/latest/arbitrary-ports-hosts.html)
---------------------------------------------
%% Cell type:markdown id:potential-store tags:
#### Exercise 1:
1. Start a web-server
**ATTENTION:** the code below fails from within a Jupyter notebook, because Jupyter is already running a tornado event loop
**Solution:**
1) Create a Python file `hello-webserver.py`
%% Cell type:markdown id:7815ccff-fb70-4c44-9c34-2b43e7fa36e4 tags:
```python
### WILL FAIL FROM WITHIN A JUPYTER NOTEBOOK ####
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
def make_app():
return tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
app = make_app()
app.listen(12345)
tornado.ioloop.IOLoop.current().start()
```
%% Cell type:markdown id:313ec32a-1e6a-4f32-9a3c-d336fd65afc7 tags:
2) and run it in the terminal
```sh
python hello-webserver.py
```
%% Cell type:markdown id:dynamic-experience tags:
#### Exercise 2:
1. Connect to the running web-server from within your browser
- add `/proxy/<port>` to the `<notebook-base>`
- What is my `<notebook-base`?
- Go to the Jupyter-JSC landing page (eg. click on `Control Panel` in system tray)
- right-click on the `open` button and copy the link-url
- e.g. https://jupyter-jsc.fz-juelich.de/user/j.goebbert@fz-juelich.de/jusuf_login/proxy/8888
%% Cell type:markdown id:computational-prime tags:
**Imagine what that allows you to do without the hassle of ssh-tunnels!**
%% Cell type:markdown id:peripheral-leave tags:
#### Exercise 3:
1. Check how `jupyter-xprahtml5-proxy` is using the [jupyter-server-proxy](https://github.com/jupyterhub/jupyter-server-proxy) extension
- [more](https://github.com/FZJ-JSC/jupyter-xprahtml5-proxy)
%% Cell type:code id:d02626aa-d853-40f5-bd6a-489a5ed5ce75 tags:
``` python
```
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
def make_app():
return tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
app = make_app()
app.listen(12345)
tornado.ioloop.IOLoop.current().start()
\ No newline at end of file
File added
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment