Update IBG 3: JSC Supercomputer HowTo authored by Johannes Keller's avatar Johannes Keller
......@@ -69,121 +69,16 @@ https://gitlab.jsc.fz-juelich.de/sdlts/general-organisation/jsc-supercomputer-wi
## Data handling
Links:
- JSC, available file systems: https://apps.fz-juelich.de/jsc/hps/juwels/environment.html#available-file-systems
Next to your `$HOME` directory, you can add a personal directory on each file system (example for JUWELS, replace project code for other projects):
- compute project
- `mkdir $PROJECT_cjibg36/$USER`
- `mkdir $SCRATCH_cjibg36/$USER`
- data project
- `mkdir $DATA_jibg36/$USER`
- `mkdir $ARCHIVE_jibg36/$USER`
https://gitlab.jsc.fz-juelich.de/sdlts/general-organisation/jsc-supercomputer-wiki-for-ibg-3/-/wikis/Data-Handling-on-JSC-Supercomputers
### File Systems and How to Use Them
For detailed information about the JSC file systems, refer to https://apps.fz-juelich.de/jsc/hps/juwels/environment.html#available-file-systems.
We add some more specific information for the file systems used in our projects.
#### $HOME
- Minimal usage with very few files.
- Reserved for user program configuration files.
#### $PROJECT
- Regular backups available, but limited memory and file count.
- Store source codes, binaries, and important files.
- Avoid storing full simulation directories; include only necessary files for reproducing runs (e.g., scripts for data download, excluding actual data).
#### $SCRATCH
- No backups provided, but larger space compared to $PROJECT.
- Files are automatically removed after 3 months.
- Utilize for active simulations; move relevant outputs and reproduction-related files to `$PROJECT/$DATA/$ARCHIVE` after completion.
#### $DATA
- Few backups, offering ample storage space.
- Suitable for storing input files, large output files, and forcing data.
#### $ARCHIVE
- No quick file access, large storage capacity, limited file count.
- Archive files using "tar-ball" compression before moving here.
- Ideal for archiving significant simulations or those related to publications.
https://gitlab.jsc.fz-juelich.de/sdlts/general-organisation/jsc-supercomputer-wiki-for-ibg-3/-/wikis/Data-Handling-on-JSC-Supercomputers#file-systems-and-how-to-use-them
### Moving data to archive
**Remark:** The same procedure applies with `jibg36` for JUWELS users instead of `jicg41`.
1. **Apply for Access**: Request access to the data project `jicg41` on JuDoor by clicking the `Join a project` button and entering `jicg41` (without the `c`).
2. **Create Directory**: Once approved, create a directory named after your username under `$ARCHIVE_jicg41` using the following commands:
```shell
cd $ARCHIVE_jicg41
mkdir $USER
```
3. **Archive Files**: Before transfering files from `$PROJECT_cjicg41` to `$ARCHIVE_jicg41`, archive these files using `tar`:
```shell
cd $PROJECT_cjicg41/$USER
```
- Create tar-balls for the files (no compression):
```shell
tar -cvf NAME.tar PATH/TO/DIR/FILES
```
- Optionally, compress tar-balls (e.g., gzip):
```shell
tar -czvf NAME.tar.gz PATH/TO/DIR/FILES
```
4. **Move Files**: After archiving, move the tar-balls:
```shell
mv NAME.tar $ARCHIVE_jicg41/$USER
```
https://gitlab.jsc.fz-juelich.de/sdlts/general-organisation/jsc-supercomputer-wiki-for-ibg-3/-/wikis/Data-Handling-on-JSC-Supercomputers#moving-data-to-archive
### Checking memory and number of files
JSC's `jutil` command gives an overview of the memory and file usage on each partition. Oftentimes more fine-grained, directory-specific information is useful. The following two commands (`find`, `du`) can do this.
#### jutil: Check memory/file usage for full partition
To review personal information regarding your compute project usage, run the following commands on JUWELS/JURECA:
```shell
jutil user show
```
```shell
jutil user cpuquota -p cjicg41
```
(Replace `cjicg41` with your project key)
```shell
jutil user dataquota -p cjicg41 -U GB
```
(Replace `cjicg41` with your project key)
#### find: Check number of files
Check the number of files in the current directory:
```shell
find . -name "*" -type f | wc -l
```
Check the number of files in all subdirectories of the current directory
```shell
for d in ./*/ ; do echo $d; (find $d -type f | wc -l); done
```
#### du: Check memory
Check the memory in the current directory and subdirectories of level 1:
```shell
du -ha -d1 .
```
\ No newline at end of file
https://gitlab.jsc.fz-juelich.de/sdlts/general-organisation/jsc-supercomputer-wiki-for-ibg-3/-/wikis/Data-Handling-on-JSC-Supercomputers#checking-memory-and-number-of-files
\ No newline at end of file