"- Combinations: [ipython on MPI](https://ipyparallel.readthedocs.io/en/latest/reference/mpi.html), [Dask on CUDA](https://docs.rapids.ai/api/dask-cuda/stable/), [Dask on Ray](https://docs.ray.io/en/latest/ray-more-libs/dask-on-ray.html), etc.\n",
"- [Pandas](https://pandas.pydata.org/) and its HPC derivatives: [cuDF](https://github.com/rapidsai/cudf), [modin](https://github.com/modin-project/modin), [vaex](https://vaex.io/)\n",
"- ML frameworks and derivatives: [TensorFlow](https://www.tensorflow.org/), [PyTorch](https://pytorch.org/), [JAX](https://github.com/google/jax), [HeAT](https://github.com/helmholtz-analytics/heat) (=Helmholtz Analytics Toolkit)\n",
"- indexing, hashing, precalculation for random access\n",
"- compression, memory mapped files for faster data availability\n",
"- [HDF5](https://docs.h5py.org/en/stable/), [NetCDF](https://github.com/Unidata/netcdf4-python), [SIONlib](https://www.fz-juelich.de/en/ias/jsc/services/user-support/jsc-software-tools/sionlib), [MPI-I/O](https://mpi4py.readthedocs.io/en/stable/tutorial.html#mpi-io) for parallel file access\n",
"- see also https://www.fz-juelich.de/en/ias/jsc/education/training-courses/training-materials/course-material-parallel-i-o-and-portable-data-formats\n",
"- scalable database management systems for complex data (many with python API):\n",
" - object-relational *,\n",
" - array *, \n",
" - graph *, \n",
" - in-memory *, \n",
" - key-value stores, \n",
" - object * \n",
" - etc. (*= database management system)"
]
},
{
"cell_type": "markdown",
"id": "7416f9c5-5919-4192-ae7a-30f6fb9eda72",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"## perhaps the next big waves on the HPC Python horizon:\n",
"\n",
"- [Python 3.13](https://docs.python.org/3.13/whatsnew/3.13.html) (due June 2024) will allow to build it either with no GIL or with a JIT compiler \n",
- Combinations: [ipython on MPI](https://ipyparallel.readthedocs.io/en/latest/reference/mpi.html), [Dask on CUDA](https://docs.rapids.ai/api/dask-cuda/stable/), [Dask on Ray](https://docs.ray.io/en/latest/ray-more-libs/dask-on-ray.html), etc.
-[Pandas](https://pandas.pydata.org/) and its HPC derivatives: [cuDF](https://github.com/rapidsai/cudf), [modin](https://github.com/modin-project/modin), [vaex](https://vaex.io/)
- ML frameworks and derivatives: [TensorFlow](https://www.tensorflow.org/), [PyTorch](https://pytorch.org/), [JAX](https://github.com/google/jax), [HeAT](https://github.com/helmholtz-analytics/heat)(=Helmholtz Analytics Toolkit)
- IDEs: [VS Code](https://code.visualstudio.com/), [PyCharm](https://www.jetbrains.com/pycharm/), etc.