Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
AMBS
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container registry
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
GitLab community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
esde
machine-learning
AMBS
Commits
45a7dd21
Commit
45a7dd21
authored
3 years ago
by
Michael Langguth
Browse files
Options
Downloads
Patches
Plain Diff
Corrections to create_env.sh.
parent
6959a649
No related branches found
No related tags found
No related merge requests found
Pipeline
#89792
failed
3 years ago
Stage: build
Stage: test
Stage: deploy
Changes
2
Pipelines
1
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
video_prediction_tools/deprecated/modules_train.sh
+0
-0
0 additions, 0 deletions
video_prediction_tools/deprecated/modules_train.sh
video_prediction_tools/env_setup/create_env.sh
+3
-48
3 additions, 48 deletions
video_prediction_tools/env_setup/create_env.sh
with
3 additions
and
48 deletions
video_prediction_tools/
env_setup
/modules_train.sh
→
video_prediction_tools/
deprecated
/modules_train.sh
+
0
−
0
View file @
45a7dd21
File moved
This diff is collapsed.
Click to expand it.
video_prediction_tools/env_setup/create_env.sh
+
3
−
48
View file @
45a7dd21
...
@@ -54,8 +54,8 @@ TF_CONTAINER=${WORKING_DIR}/HPC_scripts/tensorflow_21.09-tf1-py3.sif
...
@@ -54,8 +54,8 @@ TF_CONTAINER=${WORKING_DIR}/HPC_scripts/tensorflow_21.09-tf1-py3.sif
## perform sanity checks
## perform sanity checks
modules_purge
=
""
modules_purge
=
""
if
[[
!
-f
${
CONTAINER
_IMG
}
]]
;
then
if
[[
!
-f
${
TF_
CONTAINER
}
]]
;
then
echo
"ERROR: Cannot find required TF1.15 container image '
${
CONTAINER
_IMG
}
'."
echo
"ERROR: Cannot find required TF1.15 container image '
${
TF_
CONTAINER
}
'."
return
return
fi
fi
...
@@ -80,7 +80,7 @@ fi
...
@@ -80,7 +80,7 @@ fi
if
[[
"
${
HOST_NAME
}
"
==
hdfml
*
||
"
${
HOST_NAME
}
"
==
*
jwlogin
*
]]
;
then
if
[[
"
${
HOST_NAME
}
"
==
hdfml
*
||
"
${
HOST_NAME
}
"
==
*
jwlogin
*
]]
;
then
# load modules and check for their availability
# load modules and check for their availability
echo
"***** Checking modules required during the workflow... *****"
echo
"***** Checking modules required during the workflow... *****"
source
"
${
ENV_SETUP
_DIR
}
"
/modules_preprocess.sh purge
source
"
${
THIS
_DIR
}
"
/modules_preprocess.sh purge
else
else
echo
"ERROR: AMBS-workflow is currently only supported on the Juelich HPC-systems HDF-ML, Juwels and Juwels Booster"
echo
"ERROR: AMBS-workflow is currently only supported on the Juelich HPC-systems HDF-ML, Juwels and Juwels Booster"
return
return
...
@@ -94,52 +94,7 @@ if [[ "$ENV_EXIST" == 0 ]]; then
...
@@ -94,52 +94,7 @@ if [[ "$ENV_EXIST" == 0 ]]; then
echo
"Configuring and activating virtual environment on
${
HOST_NAME
}
"
echo
"Configuring and activating virtual environment on
${
HOST_NAME
}
"
singularity
exec
--nv
"
${
TF_CONTAINER
}
"
./install_venv_container.sh
"
${
ENV_DIR
}
"
singularity
exec
--nv
"
${
TF_CONTAINER
}
"
./install_venv_container.sh
"
${
ENV_DIR
}
"
else
# cretae virtual environemt here
python3
-m
venv
$ENV_DIR
activate_virt_env
=
${
ENV_DIR
}
/bin/activate
echo
"Activating virtual environment
${
ENV_DIR
}
to install required Python modules..."
source
${
activate_virt_env
}
# Install packages depending on host
if
[[
"
${
HOST_NAME
}
"
==
hdfml
*
||
"
${
HOST_NAME
}
"
==
*
juwels
*
||
"
${
HOST_NAME
}
"
==
*
jwlogin
*
]]
;
then
echo
"***** Start installing additional Python modules with pip... *****"
req_file
=
${
ENV_SETUP_DIR
}
/requirements.txt
if
[[
"
${
bool_container
}
"
>
0
]]
;
then
req_file
=
${
ENV_SETUP_DIR
}
/requirements_container.txt
;
fi
pip3
install
--no-cache-dir
-r
${
req_file
}
else
echo
"***** Start installing additional Python modules with pip... *****"
req_file
=
${
ENV_SETUP_DIR
}
/requirements_noHPC.txt
pip3
install
--upgrade
pip
pip3
install
--no-cache-dir
-r
${
req_file
}
fi
# expand PYTHONPATH...
export
PYTHONPATH
=
${
WORKING_DIR
}
:
$PYTHONPATH
>>
${
activate_virt_env
}
export
PYTHONPATH
=
${
WORKING_DIR
}
/utils:
$PYTHONPATH
>>
${
activate_virt_env
}
export
PYTHONPATH
=
${
WORKING_DIR
}
/external_package/lpips-tensorflow:
$PYTHONPATH
>>
${
activate_virt_env
}
export
PYTHONPATH
=
${
WORKING_DIR
}
/model_modules:
$PYTHONPATH
>>
${
activate_virt_env
}
export
PYTHONPATH
=
${
WORKING_DIR
}
/postprocess:
$PYTHONPATH
>>
${
activate_virt_env
}
if
[[
"
${
HOST_NAME
}
"
==
hdfml
*
||
"
${
HOST_NAME
}
"
==
*
jwlogin
*
]]
;
then
export
PYTHONPATH
=
${
ENV_DIR
}
/lib/python3.6/site-packages:
$PYTHONPATH
>>
${
activate_virt_env
}
fi
# ...and ensure that this also done when the
echo
""
>>
"
${
activate_virt_env
}
"
echo
"# Expand PYTHONPATH..."
>>
${
activate_virt_env
}
echo
"export PYTHONPATH=
${
WORKING_DIR
}
:
\$
PYTHONPATH"
>>
${
activate_virt_env
}
echo
"export PYTHONPATH=
${
WORKING_DIR
}
/utils/:
\$
PYTHONPATH"
>>
${
activate_virt_env
}
echo
"export PYTHONPATH=
${
WORKING_DIR
}
/model_modules:
$PYTHONPATH
"
>>
${
activate_virt_env
}
echo
"export PYTHONPATH=
${
WORKING_DIR
}
/external_package/lpips-tensorflow:
\$
PYTHONPATH"
>>
${
activate_virt_env
}
echo
"export PYTHONPATH=
${
WORKING_DIR
}
/postprocess:
$PYTHONPATH
"
>>
${
activate_virt_env
}
if
[[
"
${
HOST_NAME
}
"
==
hdfml
*
||
"
${
HOST_NAME
}
"
==
*
jwlogin
*
]]
;
then
echo
"export PYTHONPATH=
${
ENV_DIR
}
/lib/python3.6/site-packages:
\$
PYTHONPATH"
>>
${
activate_virt_env
}
fi
fi
info_str
=
"Virtual environment
${
ENV_DIR
}
has been set up successfully."
info_str
=
"Virtual environment
${
ENV_DIR
}
has been set up successfully."
elif
[[
"
$ENV_EXIST
"
==
1
]]
;
then
elif
[[
"
$ENV_EXIST
"
==
1
]]
;
then
# loading modules of postprocessing and activating virtual env are suifficient
# loading modules of postprocessing and activating virtual env are suifficient
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment