diff --git a/README.md b/README.md index 7bef26f7182549c7000a5ad020e5f9c113763aaa..87f3669617a0487659007b06d03d327d6b1924e7 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,100 @@ -# In transit coupling of NEST and Elephant - master thesis +# In transit coupling of NEST and Elephant ** Kim Sontheimer, k.sontheimer@fz-juelich.de** +## **Thesis:** +### In Transit Coupling of Neuroscientific Simulation and Analysis on High Performance Computing Systems +High performance computing (HPC) is experiencing an increasing imbalance between +processing power and I/O capabilities. This imbalance has led to the challenge +of managing the large amounts of data produced by extreme-scale simulations. It +has become prohibitive expensive to store this data on disk for subsequent offline +analysis. In transit processing could perform this analysis on memory-resident data. +In this thesis, based on requirements of neuroscientific use cases, a framework +has been designed, implemented and tested on the JURECA supercomputer located +at the Forschungszentrum Jülich. In the framework, simulation and analysis are +connected in transit across compute nodes using a client-server model. Data is +transferred in a streaming manner, without disk I/O in between. The framework +fulfills the presented use case requirements. Dedicated experiments on algorithmic +solutions of the data transfer show, that no data is lost during transfer. +The design of the framework enables future integration of other software and thus +could serve as a basis for in transit coupling in neuroscientific workflows on HPC +systems. + + + +## **Framework -- in transit coupling, v2.1** + +### Tested with (locally / on JURECA): + +* Python 3.6.5 / 3.6.8 +* Numpy 1.16.4 / 1.15.2 +* mpi4py 2.0.0 / 3.0.1 + + +### Mandatory Folder/Files: + +| folder | files | +| ------ | ------ | +| *in_transit-coupling/* | *analysis.py, data_handler.py, server.py, simulation.py, timer.py* | +| *nest_to_file/* | *all .gdf -> download them from https://fz-juelich.sciebo.de/s/RrR4XBcL51uy0vA* | + + +### On JURECA +from folder: *in_transit_coupling* + +1) Interactive session (3 nodes): +`salloc --nodes=3 --time=00:15:00 --account=slns` + +2) get the <jobid>, connect interactively in two additional console-tabs: +`srun --cpu-bind=none --jobid=<jobid> --pty /bin/bash -i` + +3) load the modules in each tab: +`. ./load_modules.sh` + +4) On each node, in this order!, start server, analysis, simulation: + 1. in transit server with append (a) or sort (s) algorithm: + `srun -N 1 -n x python server.py [a,s]` + + 2. analysis with nn=number of neurons to 'analyze': + `srun -N 1 -n 1 python analysis.py [nn]` + + 3. simulation of a NEST simulation; arg1 = simtype; arg2 = number of lines per package: + `srun -N 1 -n 1 python simulation.py [short,long,small] [lines]` + + + +### EXAMPLE USAGE: + +`srun -N 1 -n 4 python server.py s` + +`srun -N 1 -n 1 python analysis.py 100` + +`srun -N 1 -n 1 python simulation.py short 10000` + + + +### Additional information: + +* simtype loads files from 'nest_to_file' folder: + * short = 10s sim, 1000 neurons, ~4.7M events + * long = 100s sim, 1000 neurons, ~47M events + * small = 100s sim, 100 neurons, ~3.4M events + + +* short module description: + * simulation.py: + simulates a NEST simulation; streams data via MPI to the in transit server + size of packages set by argument + * analysis.py: + receives data from the in transit server; interface for Elephant analysis + * server.py: + the in transit server; accepts connections from simulation and analysis + * data_handler.py: + the transfer logic; + ringbuffer and shared memory allocation, + receiving data on rank 0, + transpose,exchange,convert,send data on rank 1-x + ## **Important links:**