Skip to content
Snippets Groups Projects
Commit 84b8b122 authored by Jens Henrik Goebbert's avatar Jens Henrik Goebbert
Browse files

update mount

parent 551cde26
No related branches found
No related tags found
No related merge requests found
%% Cell type:markdown id:exclusive-nursery tags: %% Cell type:markdown id:exclusive-nursery tags:
![jsc-logo.jpg](attachment:67258d94-84e6-4a0c-ae8f-c74332ec082e.jpg) ![jsc-logo.jpg](attachment:67258d94-84e6-4a0c-ae8f-c74332ec082e.jpg)
Author: [Jens Henrik Göbbert](mailto:j.goebbert@fz-juelich.de) Author: [Jens Henrik Göbbert](mailto:j.goebbert@fz-juelich.de)
------------------------------------ ------------------------------------
%% Cell type:markdown id:worldwide-presence tags: %% Cell type:markdown id:worldwide-presence tags:
# Supercomputing with JupyterLab - Tips & Tricks # Supercomputing with JupyterLab - Tips & Tricks
This is the first time you are using JupyterLab on an HPC system? Let us show you some tips & tricks. This is the first time you are using JupyterLab on an HPC system? Let us show you some tips & tricks.
This notebook might be special to the environment at Juelich Supercomputing Centre, Forschungszentrum Juelich, Germany.# JupyterLab Tour This notebook might be special to the environment at Juelich Supercomputing Centre, Forschungszentrum Juelich, Germany.# JupyterLab Tour
------------------------- -------------------------
%% Cell type:markdown id:retired-crest tags: %% Cell type:markdown id:retired-crest tags:
#### Exercise 1: #### Exercise 1:
**ONLY FOR HDF-CLOUD** **ONLY FOR HDF-CLOUD**
**!!DANGEROUS!! YOU ARE MOUNTING WRITEABLE** **!!DANGEROUS!! YOU ARE MOUNTING WRITEABLE**
Mount the $HOMEs from GPFS on the HDF Cloud Mount the $HOMEs from GPFS on the HDF Cloud
%% Cell type:code id:abd7420b-36b8-4f8b-91fc-cc10f6d45013 tags: %% Cell type:code id:abd7420b-36b8-4f8b-91fc-cc10f6d45013 tags:
``` python ``` python
import json import json
import os import os
from contextlib import closing from contextlib import closing
import pyunicore.client as unicore_client import pyunicore.client as unicore_client
import requests import requests
``` ```
%% Cell type:markdown id:2da2cb6f-d6c6-4bbe-a7bb-8dd488615181 tags: %% Cell type:markdown id:2da2cb6f-d6c6-4bbe-a7bb-8dd488615181 tags:
During the login process in Jupyter-JSC, you have authenticated yourself against the identity management system [*Unity-IdM*](https://www.unity-idm.eu) . During the login process in Jupyter-JSC, you have authenticated yourself against the identity management system [*Unity-IdM*](https://www.unity-idm.eu) .
This IdM issues temporary tokens (comparable to user+password) on request, which can be use for authentication to other services that are also connected to the same IdM. This IdM issues temporary tokens (comparable to user+password) on request, which can be use for authentication to other services that are also connected to the same IdM.
Here we use the JupyterHub-API (+your JUPYTERHUB_API_TOKEN) to get a new temporary token for the next steps. Here we use the JupyterHub-API (+your JUPYTERHUB_API_TOKEN) to get a new temporary token for the next steps.
%% Cell type:code id:643d4c6b-927e-4d25-a3b0-1e6755a6cae3 tags: %% Cell type:code id:643d4c6b-927e-4d25-a3b0-1e6755a6cae3 tags:
``` python ``` python
def get_access_token(): def get_access_token():
remote_node = os.getenv("REMOTE_NODE") remote_node = os.getenv("REMOTE_NODE")
remote_hub_port = os.getenv("REMOTE_PORT") remote_hub_port = os.getenv("REMOTE_PORT")
hub_api_url = f"http://{remote_node}:{remote_hub_port}/hub/api/user" hub_api_url = f"http://{remote_node}:{remote_hub_port}/hub/api/user"
headers = {"Authorization": "token {}".format(os.getenv("JUPYTERHUB_API_TOKEN"))} headers = {"Authorization": "token {}".format(os.getenv("JUPYTERHUB_API_TOKEN"))}
with closing(requests.get(hub_api_url, headers=headers)) as r: with closing(requests.get(hub_api_url, headers=headers)) as r:
if r.status_code == 200: if r.status_code == 200:
resp = json.loads(r.content.decode("utf-8")) resp = json.loads(r.content.decode("utf-8"))
else: else:
raise Exception( raise Exception(
"Could not receive access token: {} {}".format( "Could not receive access token: {} {}".format(
r.status_code, r.content.decode("utf-8") r.status_code, r.content.decode("utf-8")
) )
) )
# No HPC accounts -> no access token in this script # No HPC accounts -> no access token in this script
if ( if (
"auth_state" in resp.keys() "auth_state" in resp.keys()
and "oauth_user" in resp["auth_state"].keys() and "oauth_user" in resp["auth_state"].keys()
and "hpc_infos_attribute" in resp["auth_state"]["oauth_user"] and "hpc_infos_attribute" in resp["auth_state"]["oauth_user"]
and len(resp["auth_state"]["oauth_user"]["hpc_infos_attribute"]) > 0 and len(resp["auth_state"]["oauth_user"]["hpc_infos_attribute"]) > 0
): ):
return resp["auth_state"]["access_token"] return resp["auth_state"]["access_token"]
else: else:
return None return None
``` ```
%% Cell type:code id:76076cad-cc9c-4590-b51b-d0a51432fb80 tags:
``` python
def get_access_token():
api_url = os.getenv("JUPYTERHUB_API_URL")
user_api_url = f"{api_url}/user_oauth"
headers = {"Authorization": "token {}".format(os.getenv("JUPYTERHUB_API_TOKEN"))}
with closing(requests.get(user_api_url, headers=headers)) as r:
if r.status_code == 200:
resp = json.loads(r.content.decode("utf-8"))
else:
raise Exception(
"Could not receive access token: {} {}".format(
r.status_code, r.content.decode("utf-8")
)
)
# No HPC accounts -> no access token in this script
if (
"auth_state" in resp.keys()
and "access_token" in resp["auth_state"].keys()
):
return resp["auth_state"]["access_token"]
else:
return None
```
%% Cell type:markdown id:540563d1-59ca-4d28-86f7-171c98310a68 tags: %% Cell type:markdown id:540563d1-59ca-4d28-86f7-171c98310a68 tags:
Having that `access_token` we can now ask unicore for information about the HPC account of the user the token belongs to. Having that `access_token` we can now ask unicore for information about the HPC account of the user the token belongs to.
We will receive the `uid` and the base of all its home directories as `remote_base_dir`. We will receive the `uid` and the base of all its home directories as `remote_base_dir`.
That information is required to ask the service [*UFTP*](https://apps.fz-juelich.de/jsc/hps/judac/uftp.html) all details needed for the mound command. That information is required to ask the service [*UFTP*](https://apps.fz-juelich.de/jsc/hps/judac/uftp.html) all details needed for the mound command.
%% Cell type:markdown id:aa5c5156-9afd-4b7b-a78f-7bb40aef2222 tags: %% Cell type:markdown id:aa5c5156-9afd-4b7b-a78f-7bb40aef2222 tags:
**ATTENTION:** Set `ro` or `rw` for mounting readonly or writable **ATTENTION:** Set `ro` or `rw` for mounting readonly or writable
%% Cell type:code id:78c3f65d-1039-482f-900b-c9b0173b490d tags: %% Cell type:code id:78c3f65d-1039-482f-900b-c9b0173b490d tags:
``` python ``` python
def get_mount_command(access_token): def get_mount_command(access_token):
_auth = "https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/" _auth = "https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/"
_tr = unicore_client.Transport(access_token) _tr = unicore_client.Transport(access_token)
_info = _tr.get(url=_auth) _info = _tr.get(url=_auth)
_uid = _info["JUDAC"]["uid"] _uid = _info["JUDAC"]["uid"]
remote_base_dir = "/p/home/jusers/%s" % _uid remote_base_dir = "/p/home/jusers/%s" % _uid
# authenticate # authenticate
_req = { _req = {
"persistent": "true", "persistent": "true",
"serverPath": "%s/___UFTP___MULTI___FILE___SESSION___MODE___" % remote_base_dir, "serverPath": "%s/___UFTP___MULTI___FILE___SESSION___MODE___" % remote_base_dir,
} }
_reply = _tr.post(url=_auth + "/JUDAC", json=_req).json() _reply = _tr.post(url=_auth + "/JUDAC", json=_req).json()
uftp_pwd = _reply["secret"] uftp_pwd = _reply["secret"]
uftp_host = _reply["serverHost"] uftp_host = _reply["serverHost"]
uftp_port = _reply["serverPort"] uftp_port = _reply["serverPort"]
return f"curlftpfs -s -o rw,uid=1094,gid=100,ftp_method=singlecwd,enable_epsv,user=anonymous:{uftp_pwd} {uftp_host}:{uftp_port}" return f"curlftpfs -s -o rw,uid=1094,gid=100,ftp_method=singlecwd,enable_epsv,user=anonymous:{uftp_pwd} {uftp_host}:{uftp_port}"
``` ```
%% Cell type:markdown id:e7f6af90-bd33-4a55-9659-973bcaa3da67 tags: %% Cell type:markdown id:e7f6af90-bd33-4a55-9659-973bcaa3da67 tags:
------------------------------------ ------------------------------------
Let's bring all pieces together: Let's bring all pieces together:
%% Cell type:code id:b0fa1183-2983-4702-94b1-b4567fea4df3 tags: %% Cell type:code id:2cdd8bf4-aab5-4cd8-bfda-cd99ca82e086 tags:
``` python ``` python
access_token = get_access_token() access_token = get_access_token()
if not access_token: if not access_token:
print("Error: No access token") print("Error: No access token")
```
%% Cell type:code id:0368679f-e3dd-4499-8b2e-ed429ee5ed53 tags:
``` python
mount_cmd = get_mount_command(access_token) mount_cmd = get_mount_command(access_token)
# e.g. curlftpfs -s -o ro,uid=1094,gid=100,ftp_method=singlecwd,enable_epsv,user=anonymous:fwngibbflwngfllaycpc judacsrv.fz-juelich.de:64333 # e.g. curlftpfs -s -o ro,uid=1094,gid=100,ftp_method=singlecwd,enable_epsv,user=anonymous:fwngibbflwngfllaycpc judacsrv.fz-juelich.de:64333
print(mount_cmd.partition("anonymous:")[0], "<...>") # do not print the token print(mount_cmd.partition("anonymous:")[0], "<...>") # do not print the token
``` ```
%% Output %% Output
curlftpfs -s -o rw,uid=1094,gid=100,ftp_method=singlecwd,enable_epsv,user= <...> ---------------------------------------------------------------------------
HTTPError Traceback (most recent call last)
Cell In[7], line 1
----> 1 mount_cmd = get_mount_command(access_token)
2 # e.g. curlftpfs -s -o ro,uid=1094,gid=100,ftp_method=singlecwd,enable_epsv,user=anonymous:fwngibbflwngfllaycpc judacsrv.fz-juelich.de:64333
3 print(mount_cmd.partition("anonymous:")[0], "<...>") # do not print the token
Cell In[3], line 13, in get_mount_command(access_token)
8 # authenticate
9 _req = {
10 "persistent": "true",
11 "serverPath": "%s/___UFTP___MULTI___FILE___SESSION___MODE___" % remote_base_dir,
12 }
---> 13 _reply = _tr.post(url=_auth + "/JUDAC", json=_req).json()
14 uftp_pwd = _reply["secret"]
15 uftp_host = _reply["serverHost"]
File /p/software/jsccloud/stages/2023/software/JupyterLab/2023.3.6-GCCcore-11.3.0/lib/python3.10/site-packages/pyunicore/client.py:189, in Transport.post(self, **kwargs)
187 def post(self, **kwargs):
188 """do a POST and return the response"""
--> 189 return self.run_method(requests.post, **kwargs)
File /p/software/jsccloud/stages/2023/software/JupyterLab/2023.3.6-GCCcore-11.3.0/lib/python3.10/site-packages/pyunicore/client.py:165, in Transport.run_method(self, method, **args)
158 if self.repeat_required(res, _headers):
159 res = method(
160 headers=_headers,
161 verify=self.verify,
162 timeout=self.timeout,
163 **args,
164 )
--> 165 self.check_error(res)
166 if self.use_security_sessions:
167 self.last_session_id = res.headers.get("X-UNICORE-SecuritySession", None)
File /p/software/jsccloud/stages/2023/software/JupyterLab/2023.3.6-GCCcore-11.3.0/lib/python3.10/site-packages/pyunicore/client.py:143, in Transport.check_error(self, res)
141 pass
142 msg = f"{res.status_code} Server Error: {reason} for url: {res.url}"
--> 143 raise requests.HTTPError(msg, response=res)
144 else:
145 res.raise_for_status()
HTTPError: 400 Server Error: Bad Request for url: https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth//JUDAC
%% Cell type:code id:45c2ff24-a735-459a-8b53-ceeb74752635 tags: %% Cell type:code id:45c2ff24-a735-459a-8b53-ceeb74752635 tags:
``` python ``` python
%%bash -s "$mount_cmd" %%bash -s "$mount_cmd"
mkdir -p $HOME/uftp-mount-rw mkdir -p $HOME/uftp-mount-rw
$1 $HOME/uftp-mount-rw $1 $HOME/uftp-mount-rw
``` ```
%% Cell type:code id:30a5b1e5-4f11-465a-a27d-e0b2c5973ec0 tags: %% Cell type:code id:30a5b1e5-4f11-465a-a27d-e0b2c5973ec0 tags:
``` python ``` python
# ... # ...
!ls -l $HOME/uftp-mount-rw !ls -l $HOME/uftp-mount-rw
``` ```
%% Output %% Output
total 68 total 68
drwx------ 3 jovyan users 4096 Mar 14 23:53 deep drwx------ 3 jovyan users 4096 Mar 14 23:53 deep
drwx------ 3 jovyan users 4096 Mar 31 21:42 hdfml drwx------ 3 jovyan users 4096 Mar 31 21:42 hdfml
drwx------ 3 jovyan users 4096 Mar 25 11:11 judac drwx------ 3 jovyan users 4096 Mar 25 11:11 judac
drwx------ 3 jovyan users 16384 Apr 5 2022 jureca drwx------ 3 jovyan users 16384 Apr 5 2022 jureca
drwx------ 3 jovyan users 4096 Aug 14 2021 juropa3exp drwx------ 3 jovyan users 4096 Aug 14 2021 juropa3exp
drwx------ 3 jovyan users 16384 Mar 31 23:15 jusuf drwx------ 3 jovyan users 16384 Mar 31 23:15 jusuf
drwx------ 3 jovyan users 16384 Apr 1 10:45 juwels drwx------ 3 jovyan users 16384 Apr 1 10:45 juwels
drwx------ 3 jovyan users 4096 Mar 9 21:50 shared drwx------ 3 jovyan users 4096 Mar 9 21:50 shared
%% Cell type:code id:5a4ff2ac-8731-4ce2-ab79-e89b6d3887a6 tags: %% Cell type:code id:5a4ff2ac-8731-4ce2-ab79-e89b6d3887a6 tags:
``` python ``` python
# unmount # unmount
!fusermount -u $HOME/uftp-mount-rw !fusermount -u $HOME/uftp-mount-rw
``` ```
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment