Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
AMBS
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container registry
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
GitLab community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
esde
machine-learning
AMBS
Commits
f54cdeec
Commit
f54cdeec
authored
3 years ago
by
Michael Langguth
Browse files
Options
Downloads
Patches
Plain Diff
Ensure that preprocessing modules are loaded when running create_env.sh.
parent
d83ffa77
No related branches found
No related tags found
No related merge requests found
Pipeline
#89842
failed
3 years ago
Stage: build
Stage: test
Stage: deploy
Changes
2
Pipelines
1
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
video_prediction_tools/deprecated/modules_postprocess.sh
+0
-0
0 additions, 0 deletions
video_prediction_tools/deprecated/modules_postprocess.sh
video_prediction_tools/env_setup/create_env.sh
+16
-26
16 additions, 26 deletions
video_prediction_tools/env_setup/create_env.sh
with
16 additions
and
26 deletions
video_prediction_tools/
env_setup
/modules_postprocess.sh
→
video_prediction_tools/
deprecated
/modules_postprocess.sh
+
0
−
0
View file @
f54cdeec
File moved
This diff is collapsed.
Click to expand it.
video_prediction_tools/env_setup/create_env.sh
+
16
−
26
View file @
f54cdeec
...
@@ -61,6 +61,7 @@ fi
...
@@ -61,6 +61,7 @@ fi
# further sanity checks:
# further sanity checks:
# * ensure execution from env_setup-directory
# * ensure execution from env_setup-directory
# * check host
# * check if virtual env has already been set up
# * check if virtual env has already been set up
if
[[
"
${
EXE_DIR
}
"
!=
"env_setup"
]]
;
then
if
[[
"
${
EXE_DIR
}
"
!=
"env_setup"
]]
;
then
...
@@ -68,6 +69,13 @@ if [[ "${EXE_DIR}" != "env_setup" ]]; then
...
@@ -68,6 +69,13 @@ if [[ "${EXE_DIR}" != "env_setup" ]]; then
return
return
fi
fi
if
!
[[
"
${
HOST_NAME
}
"
==
hdfml
*
||
"
${
HOST_NAME
}
"
==
*
jwlogin
*
]]
;
then
echo
"ERROR: AMBS-workflow is currently only supported on the Juelich HPC-systems HDF-ML, Juwels and Juwels Booster"
return
# unset PYTHONPATH on every other machine that is not a known HPC-system
# unset PYTHONPATH
fi
if
[[
-d
${
ENV_DIR
}
]]
;
then
if
[[
-d
${
ENV_DIR
}
]]
;
then
echo
"Virtual environment has already been set up under
${
ENV_DIR
}
. The present virtual environment will be activated now."
echo
"Virtual environment has already been set up under
${
ENV_DIR
}
. The present virtual environment will be activated now."
echo
"NOTE: If you wish to set up a new virtual environment, delete the existing one or provide a different name."
echo
"NOTE: If you wish to set up a new virtual environment, delete the existing one or provide a different name."
...
@@ -76,19 +84,7 @@ else
...
@@ -76,19 +84,7 @@ else
ENV_EXIST
=
0
ENV_EXIST
=
0
fi
fi
## check integratability of modules
## set up virtual environment if required
if
[[
"
${
HOST_NAME
}
"
==
hdfml
*
||
"
${
HOST_NAME
}
"
==
*
jwlogin
*
]]
;
then
# load modules and check for their availability
echo
"***** Checking modules required during the workflow... *****"
source
"
${
THIS_DIR
}
"
/modules_preprocess.sh purge
else
echo
"ERROR: AMBS-workflow is currently only supported on the Juelich HPC-systems HDF-ML, Juwels and Juwels Booster"
return
# unset PYTHONPATH on every other machine that is not a known HPC-system
# unset PYTHONPATH
fi
## set up virtual environment
if
[[
"
$ENV_EXIST
"
==
0
]]
;
then
if
[[
"
$ENV_EXIST
"
==
0
]]
;
then
# Activate virtual environment and install additional Python packages.
# Activate virtual environment and install additional Python packages.
echo
"Configuring and activating virtual environment on
${
HOST_NAME
}
"
echo
"Configuring and activating virtual environment on
${
HOST_NAME
}
"
...
@@ -97,27 +93,21 @@ if [[ "$ENV_EXIST" == 0 ]]; then
...
@@ -97,27 +93,21 @@ if [[ "$ENV_EXIST" == 0 ]]; then
info_str
=
"Virtual environment
${
ENV_DIR
}
has been set up successfully."
info_str
=
"Virtual environment
${
ENV_DIR
}
has been set up successfully."
elif
[[
"
$ENV_EXIST
"
==
1
]]
;
then
elif
[[
"
$ENV_EXIST
"
==
1
]]
;
then
# loading modules of postprocessing and activating virtual env are suifficient
info_str
=
"Virtual environment
${
ENV_DIR
}
alread exists"
source
${
ENV_SETUP_DIR
}
/modules_postprocess.sh
# activate virtual envirionment
source
${
ENV_DIR
}
/bin/activate
info_str
=
"Virtual environment
${
ENV_DIR
}
has been activated successfully."
fi
fi
## load modules (for running runscript-generator...
echo
"
${
info_str
}
"
source
${
THIS_DIR
}
/modules_preprocess.sh
## ... and prepare runscripts
echo
"Set up runscript template for user
${
USER
}
..."
echo
"Set up runscript template for user
${
USER
}
..."
if
[[
-z
"
${
base_outdir
}
"
]]
;
then
if
[[
-z
"
${
base_outdir
}
"
]]
;
then
"
${
WORKING_DIR
}
"
/utils/runscript_generator/setup_runscript_templates.sh
"
${
WORKING_DIR
}
"
/utils/runscript_generator/setup_runscript_templates.sh
else
else
"
${
WORKING_DIR
}
"
/utils/runscript_generator/setup_runscript_templates.sh
${
base_outdir
}
"
${
WORKING_DIR
}
"
/utils/runscript_generator/setup_runscript_templates.sh
"
${
base_outdir
}
"
fi
fi
echo
"******************************************** NOTE ********************************************"
echo
"******************************************** NOTE ********************************************"
echo
"
${
info_str
}
"
echo
"Make use of generate_runscript.py to generate customized runscripts of the workflow steps."
echo
"Make use of generate_runscript.py to generate customized runscripts of the workflow steps."
echo
"******************************************** NOTE ********************************************"
echo
"******************************************** NOTE ********************************************"
# finally clean up loaded modules (if we are not on Juwels)
#if [[ "${HOST_NAME}" == *hdfml* || "${HOST_NAME}" == *juwels* ]] && [[ "${HOST_NAME}" != jwlogin2[1-4]* ]]; then
# module --force purge
#fi
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment