diff --git a/docs/modules/FortranInterface.rst b/docs/modules/FortranInterface.rst
index 51901cb0e1233d2b0cf5f56c9cb2d5e2dd2a5896..badee0782188fe7098c6a47ebaa23ad6c60c346d 100644
--- a/docs/modules/FortranInterface.rst
+++ b/docs/modules/FortranInterface.rst
@@ -28,7 +28,7 @@
http://slms.pages.jsc.fz-juelich.de/websites/all-website/sphinx/api/ALL_module.html
Relevant Training Material
- https://youtu.be/-LCDEnYoFiQ?t=5394
+ `Webinar (YT) <https://www.youtube.com/watch?v=2K2YFdzIJF4&list=PLmhmpa4C4MzY02eaacXImTts2aGJHrdwQ&index=3>`_
Software Module Developed by
Stephan Schulz
@@ -70,7 +70,7 @@ library.
It is currently in use by the Fortran Multi Particle Method written for
the thesis of Stephan Schulz. This application of the interface is
-documented in the according :ref:`module<allMPMApplication>`.
+documented in the according :ref:`module<allMPMIntegration>`.
.. TODO:
diff --git a/docs/modules/MPMApplication.rst b/docs/modules/MPMApplication.rst
deleted file mode 100644
index 74b32e66c70e6e798e87051ac5432f48e23a864b..0000000000000000000000000000000000000000
--- a/docs/modules/MPMApplication.rst
+++ /dev/null
@@ -1,236 +0,0 @@
-.. In ReStructured Text (ReST) indentation and spacing are very important (it is how ReST knows what to do with your
- document). For ReST to understand what you intend and to render it correctly please to keep the structure of this
- template. Make sure that any time you use ReST syntax (such as for ".. sidebar::" below), it needs to be preceded
- and followed by white space (if you see warnings when this file is built they this is a common origin for problems).
-
-.. We allow the template to be standalone, so that the library maintainers add it in the right place
-
-:orphan:
-
-.. Firstly, let's add technical info as a sidebar and allow text below to wrap around it. This list is a work in
- progress, please help us improve it. We use *definition lists* of ReST_ to make this readable.
-
-.. sidebar:: Software Technical Information
-
- Name
- Name of the relevant software.
-
- Language
- Please indicate the primary language(s) used by the module. Please also state if interfaces for other languages are
- available.
-
- Licence
- Specify the licence under which the software is released. Provide a link to the full online description of the
- licence. You'll find descriptions of the most common licences at https://opensource.org/licenses .
- An example here would be: `GPL <https://opensource.org/licenses/gpl-license>`_ or (the more permissive)
- `MIT <https://opensource.org/licenses/mit-license>`_
-
- Documentation Tool
- All source code created for this module should be documented so please indicate what tool has been used for
- documentation. Doxygen covers most languages but for Fortran you might want to use
- `Ford <http://fortranwiki.org/fortran/show/FORD>`_, for Python ReST_, etc.
-
- Application Documentation
- Provide a link to any documentation for the application.
-
- Relevant Training Material
- Add a link to any relevant training material. If there currently is none then say 'Not currently available.'
-
- Software Module Developed by
- Add the name of the person who developed the software for this module here
-
-
-.. In the next line you have the name of how this module will be referenced in the main documentation (which you can
- reference, in this case, as ":ref:`example`"). You *MUST* change the reference below from "example" to something
- unique otherwise you will cause cross-referencing errors. The reference must come right before the heading for the
- reference to work (so don't insert a comment between).
-
-.. _example:
-
-####################
-E-CAM example module
-####################
-
-.. Let's add a local table of contents to help people navigate the page
-
-.. contents:: :local:
-
-.. Add an abstract for a *general* audience here. Write a few lines that explains the "helicopter view" of why you are
- creating this module. For example, you might say that "This module is a stepping stone to incorporating XXXX effects
- into YYYY process, which in turn should allow ZZZZ to be simulated. If successful, this could make it possible to
- produce compound AAAA while avoiding expensive process BBBB and CCCC."
-
-The E-CAM library is purely a set of documentation that describes software development efforts related to the project. A
-*module* for E-CAM is the documentation of the single development of effort associated to the project.In that sense, a
-module does not directly contain source code but instead contains links to source code, typically stored elsewhere. Each
-module references the source code changes to which it directly applies (usually via a URL), and provides detailed
-information on the relevant *application* for the changes as well as how to build and test the associated software.
-
-The original source of this page (:download:`readme.rst`) contains lots of additional comments to help you create your
-documentation *module* so please use this as a starting point. We use Sphinx_ (which in turn uses ReST_) to create this
-documentation. You are free to add any level of complexity you wish (within the bounds of what Sphinx_ and ReST_ can
-do). More general instructions for making your contribution can be found in ":ref:`contributing`".
-
-Remember that for a module to be accepted into the E-CAM repository, your source code changes in the target application
-must pass a number of acceptance criteria:
-
-* Style *(use meaningful variable names, no global variables,...)*
-
-* Source code documentation *(each function should be documented with each argument explained)*
-
-* Tests *(everything you add should have either unit or regression tests)*
-
-* Performance *(If what you introduce has a significant computational load you should make some performance optimisation
- effort using an appropriate tool. You should be able to verify that your changes have not introduced unexpected
- performance penalties, are threadsafe if needed,...)*
-
-Purpose of Module
-_________________
-
-.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
-
-Give a brief overview of why the module is/was being created, explaining a little of the scientific background and how
-it fits into the larger picture of what you want to achieve. The overview should be comprehensible to a scientist
-non-expert in the domain area of the software module.
-
-This section should also include the following (where appropriate):
-
-* Who will use the module? in what area(s) and in what context?
-
-* What kind of problems can be solved by the code?
-
-* Are there any real-world applications for it?
-
-* Has the module been interfaced with other packages?
-
-* Was it used in a thesis, a scientific collaboration, or was it cited in a publication?
-
-* If there are published results obtained using this code, describe them briefly in terms readable for non-expert users.
- If you have few pictures/graphs illustrating the power or utility of the module, please include them with
- corresponding explanatory captions.
-
-.. note::
-
- If the module is an ingredient for a more general workflow (e.g. the module was the necessary foundation for later
- code; the module is part of a group of modules that will be used to calculate certain property or have certain
- application, etc.) mention this, and point to the place where you specify the applications of the more general
- workflow (that could be in another module, in another section of this repository, an application’s website, etc.).
-
-.. note::
-
- If you are a post-doc who works in E-CAM, an obvious application for the module (or for the group of modules that
- this one is part of) is your pilot project. In this case, you could point to the pilot project page on the main
- website (and you must ensure that this module is linked there).
-
-If needed you can include latex mathematics like
-:math:`\frac{ \sum_{t=0}^{N}f(t,k) }{N}`
-which won't show up on GitLab/GitHub but will in final online documentation.
-
-If you want to add a citation, such as [CIT2009]_, please check the source code to see how this is done. Note that
-citations may get rearranged, e.g., to the bottom of the "page".
-
-.. [CIT2009] This is a citation (as often used in journals).
-
-Background Information
-______________________
-
-.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
-
-If the modifications are to an existing code base (which is typical) then this would be the place to name that
-application. List any relevant urls and explain how to get access to that code. There needs to be enough information
-here so that the person reading knows where to get the source code for the application, what version this information is
-relevant for, whether this requires any additional patches/plugins, etc.
-
-Overall, this module is supposed to be self-contained, but linking to specific URLs with more detailed information is
-encouraged. In other words, the reader should not need to do a websearch to understand the context of this module, all
-the links they need should be already in this module.
-
-Building and Testing
-____________________
-
-.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
-
-Provide the build information for the module here and explain how tests are run. This needs to be adequately detailed,
-explaining if necessary any deviations from the normal build procedure of the application (and links to information
-about the normal build process needs to be provided).
-
-Source Code
-___________
-
-.. Notice the syntax of a URL reference below `Text <URL>`_ the backticks matter!
-
-Here link the source code *that was created for the module*. If you are using Github or GitLab and the `Gitflow Workflow
-<https://www.atlassian.com/git/tutorials/comparing-workflows#gitflow-workflow>`_ you can point to your feature branch.
-Linking to your pull/merge requests is even better. Otherwise you can link to the explicit commits.
-
-* `Link to a merge request containing my source code changes
- <https://github.com/easybuilders/easybuild-easyblocks/pull/1106>`_
-
-There may be a situation where you cannot do such linking. In this case, I'll go through an example that uses a patch
-file to highlight my source code changes, for that reason I would need to explain what code (including exact version
-information), the source code is for.
-
-You can create a similar patch file by (for example if you are using git for your version control) making your changes
-for the module in a feature branch and then doing something like the following:
-
-.. Don't forget the white space around the "literal block" (a literal block keeps all spacing and is a good way to
- include terminal output, file contents, etc.)
-
-::
-
- [adam@mbp2600 example (master)]$ git checkout -b tmpsquash
- Switched to a new branch "tmpsquash"
-
- [adam@mbp2600 example (tmpsquash)]$ git merge --squash newlines
- Updating 4d2de39..b6768b2
- Fast forward
- Squash commit -- not updating HEAD
- test.txt | 2 ++
- 1 files changed, 2 insertions(+), 0 deletions(-)
-
- [adam@mbp2600 example (tmpsquash)]$ git commit -a -m "My squashed commits"
- [tmpsquash]: created 75b0a89: "My squashed commits"
- 1 files changed, 2 insertions(+), 0 deletions(-)
-
- [adam@mbp2600 example (tmpsquash)]$ git format-patch master
- 0001-My-squashed-commits.patch
-
-
-To include a patch file do something like the following (take a look at the source code of this document to see the
-syntax required to get this):
-
-.. Below I am telling Sphinx that the included file is C code, if possible it will then do syntax highlighting. I can
- even emphasise partiuclar lines (here 2 and 9-11)
-
-.. .. literalinclude:: ./simple.patch
- :language: c
- :emphasize-lines: 2,9-11
- :linenos:
-
-
-.. I can't highlight the language syntax of a patch though so I have to exclude
- :language: c
-
-.. literalinclude:: ./simple.patch
- :emphasize-lines: 2,9-11
- :linenos:
-
-If the patch is very long you will probably want to add it as a subpage which can be done as follows
-
-.. toctree::
- :glob:
- :maxdepth: 1
-
- patch
-
-.. Remember to change the reference "patch" for something unique in your patch file subpage or you will have
- cross-referencing problems
-
-you can reference it with :ref:`patch`
-
-.. Here are the URL references used (which is alternative method to the one described above)
-
-.. _ReST: http://www.sphinx-doc.org/en/stable/rest.html
-.. _Sphinx: http://www.sphinx-doc.org/en/stable/markup/index.html
-
-.. vim: et sw=2 ts=2 tw=74 spell spelllang=en_us:
diff --git a/docs/modules/MPMIntegration.F90 b/docs/modules/MPMIntegration.F90
new file mode 100644
index 0000000000000000000000000000000000000000..4b110cbdd9fb780313f6a172dd6f2a16d32c5ccd
--- /dev/null
+++ b/docs/modules/MPMIntegration.F90
@@ -0,0 +1,123 @@
+! The following additional functions were used:
+!
+! Additional information:
+! - LB_METHOD_PRE is set by the preprocessor to ALL_STAGGERED or ALL_TENSOR.
+! - The domain_bounds_old will only be used during initialisation for the
+! initial domain configuration.
+! - ``this_image()`` returns the current image index, i.e. current MPI rank+1.
+! - The work ist estimated using ``lb_estimate_work`` which takes the current
+! domain size and number of particles as arguments.
+
+ function domain_decomposition_jall(bounds, dh, num_images3, domain_bounds_old, work, output) result(domain_bounds)
+ use ISO_C_BINDING
+ type(boundingbox), intent(in) :: bounds !< simulation bounds
+ real (kind = real_kind), intent(in) :: dh !< grid width
+ integer, dimension(3), intent(in) :: num_images3 !< the 1 indexed number of images in 3D
+ type(boundingbox_aligned), intent(in) :: domain_bounds_old !< current domain bounds
+ real(real_kind), intent(in) :: work !< work of this domain
+ logical, intent(in) :: output !< output domain bounds to `vtk_outline` directory
+ type(boundingbox_aligned) :: domain_bounds
+ type(boundingbox), save :: domain_old
+ type(ALL_t), save :: jall ! ALL object which is initialized once
+ real (kind = real_kind), dimension(3,2) :: verts
+ integer, dimension(3), save :: this_image3 ! the 1 indexed image number in 3D
+ logical, save :: first_run = .true.
+ integer, save :: step = 1
+ logical, dimension(2,3), save :: domain_at_sim_bound ! true if domain is at the lower/upper simulation boundary
+ real (kind = real_kind), dimension(3), save :: min_size
+ integer(c_int), parameter :: LB_METHOD = LB_METHOD_PRE
+ character (len=ALL_ERROR_LENGTH) :: error_msg
+ if(first_run) then
+ ! calculate this_image3
+ block
+ integer :: x,y,z, cnt
+ cnt = 1
+ do z=1,num_images3(3)
+ do y=1,num_images3(2)
+ do x=1,num_images3(1)
+ if(this_image()==cnt) this_image3 = (/ x,y,z /)
+ cnt = cnt + 1
+ enddo
+ enddo
+ enddo
+ end block
+ call jall%init(LB_METHOD,3,4.0d0)
+ call jall%set_proc_grid_params(this_image3-1, num_images3)
+ call jall%set_proc_tag(this_image())
+ call jall%set_communicator(MPI_COMM_WORLD)
+ min_size(:) = (abs(Rcont_min)+abs(Rcont_max))*dh
+ call jall%set_min_domain_size(min_size)
+ domain_old%bounds_unaligned = domain_bounds_old%bounds_aligned
+ domain_at_sim_bound(1,:) = this_image3==1 ! first image in a direction is automatically at sim bound
+ domain_at_sim_bound(2,:) = this_image3==num_images3 ! last image likewise at sim bound
+ call jall%setup()
+ endif
+ call jall%set_work(real(work,real_kind))
+ !! The `domain_old` bounds are not the actual domain bounds, which
+ !! are aligned to grid widths, but what we got from the previous
+ !! iteration of load balancing. However, the simulation boundaries are
+ !! unchanged by the load balancing.
+ block
+ type(boundingbox_aligned) :: aligned_bnds
+ real (kind = real_kind), dimension(3) :: lowest_upper_bound, highest_lower_bound
+ !> Align the simulation boundaries to the grid and add an additional
+ !! grid width on the top. These may be used instead of our current
+ !! bounds, so they should align properly on the upper bound, if we
+ !! are a simulation boundary. If the simulation bounds have not
+ !! changed they should still coincide with the domain bounds.
+ aligned_bnds%bounds_aligned = floor(bounds%bounds_unaligned/dh)*dh
+ aligned_bnds%bounds_aligned(2,:) = aligned_bnds%bounds_aligned(2,:) + dh
+ !> To make sure, the shrinking domain is still always large enough
+ !! and in particular is not shrunk into the neighbouring domain.
+ !! This can happen if the bounding box is not present in the current
+ !! domain, so the outer bound is moved across the inner bound. This
+ !! must be avoided at all cost. Additionally, we also need to ensure
+ !! the minimum domain width. Also, the outer bound of all boundary
+ !! domains, must be the same. To achieve this, the outermost inner
+ !! bound is calculated in each direction. This then allows us to
+ !! compute the innermost position any outer bound may have to still
+ !! be the required distance from every next inner bound.
+ ! For the lowest domains:
+ lowest_upper_bound = comm_co_min_f(domain_old%bounds_unaligned(2,:))
+ aligned_bnds%bounds_aligned(1,:) = min(lowest_upper_bound-min_size, aligned_bnds%bounds_aligned(1,:))
+ ! For the highest domains:
+ highest_lower_bound = comm_co_max_f(domain_old%bounds_unaligned(1,:))
+ aligned_bnds%bounds_aligned(2,:) = max(highest_lower_bound+min_size, aligned_bnds%bounds_aligned(2,:))
+ ! And now set the boundary domains outer bounds to the new, fixed bounds
+ where(domain_at_sim_bound)
+ domain_old%bounds_unaligned = aligned_bnds%bounds_aligned
+ end where
+ end block
+ !> Make sure that the old domain bounds are sensible. we are only
+ !! updating them, based in the previous value. This also means
+ !! the first call must already contain a sensible approximation
+ !! (the equidistant (geometric) distribution suffices for that).
+ verts = transpose(domain_old%bounds_unaligned)
+ call jall%set_vertices(verts)
+ call jall%balance()
+ call jall%get_result_vertices(verts)
+ domain_bounds%bounds_aligned = transpose(verts)
+ domain_old%bounds_unaligned = domain_bounds%bounds_aligned
+ domain_bounds%bounds_aligned = nint(domain_bounds%bounds_aligned/dh)*dh
+ if(output) then
+ call ALL_reset_error()
+ call jall%print_vtk_outlines(step)
+ if(ALL_error() /= 0) then
+ error_msg = ALL_error_description()
+ print*, "Error in ALL detected:"
+ print*, error_msg
+ endif
+ endif
+ first_run = .false.
+ step = step + 1
+ call assert_domain_width(domain_bounds, dh)
+ end function
+
+ !> Estimate local work
+ function lb_estimate_work(n_part, domain_bounds_old) result(work)
+ integer, intent(in) :: n_part !< number of particles of this domain
+ type(boundingbox_aligned), intent(in) :: domain_bounds_old !< domain bounds
+ real(real_kind) :: work
+ real(real_kind), parameter :: beta = 0.128 ! empirically determined
+ work = n_part + beta*product( domain_bounds_old%bounds_aligned(2,:)-domain_bounds_old%bounds_aligned(1,:) )/grid%dh**3
+ end function
diff --git a/docs/modules/MPMIntegration.makefile b/docs/modules/MPMIntegration.makefile
new file mode 100644
index 0000000000000000000000000000000000000000..a56ec50cec48d70f69c932683e9db26798987008
--- /dev/null
+++ b/docs/modules/MPMIntegration.makefile
@@ -0,0 +1,22 @@
+MJOBS ?= $(shell getconf _NPROCESSORS_ONLN)
+JUELICH_ALL_INCLUDE := external/juelich_all_build/include/modules
+JUELICH_ALL_LIB := external/juelich_all_build/lib/libALL_fortran.a external/juelich_all_build/lib/libALL.a
+VTK_LIB := $(subst lib/,external/vtk_build/lib/, lib/libvtkFiltersProgrammable-7.1.a lib/libvtkIOParallelXML-7.1.a lib/libvtkIOXML-7.1.a lib/libvtkIOXMLParser-7.1.a lib/libvtkexpat-7.1.a lib/libvtkParallelMPI-7.1.a lib/libvtkParallelCore-7.1.a lib/libvtkIOLegacy-7.1.a lib/libvtkIOCore-7.1.a lib/libvtkCommonExecutionModel-7.1.a lib/libvtkCommonDataModel-7.1.a lib/libvtkCommonTransforms-7.1.a lib/libvtkCommonMisc-7.1.a lib/libvtkCommonMath-7.1.a lib/libvtkCommonSystem-7.1.a lib/libvtkCommonCore-7.1.a lib/libvtksys-7.1.a -ldl -lpthread lib/libvtkzlib-7.1.a)
+
+# ...
+
+# VTK
+VTKCONFIG_FILE := external/vtk_build/lib/cmake/vtk-7.1/VTKConfig.cmake
+$(VTKCONFIG_FILE):
+ mkdir -p external/vtk_build
+ cd external/vtk_build && CC=$(CC) CXX=$(CXX) cmake ../vtk -DCMAKE_INSTALL_PREFIX=`pwd` $(EXT_LTO_CMFLAGS) -DBUILD_SHARED_LIBS=OFF -DBUILD_TESTING=OFF -DCMAKE_BUILD_TYPE=Release -DVTK_Group_MPI=OFF -DVTK_Group_Rendering=OFF -DVTK_Group_StandAlone=OFF -DVTK_RENDERING_BACKEND=None -DVTK_USE_CXX11_FEATURES=ON -DModule_vtkCommonDataModel=ON -DModule_vtkFiltersProgrammable=ON -DModule_vtkIOParallelXML=ON -DModule_vtkParallelMPI=ON
+ $(MAKE) -j $(MJOBS) -C external/vtk_build
+ $(MAKE) -C external/vtk_build install
+
+# juelich_all
+$(JUELICH_ALL_LIB): $(VTKCONFIG_FILE)
+ mkdir -p external/juelich_all_build
+ cd external/juelich_all_build && CC=$(CC) CXX=$(CXX) cmake ../juelich_all $(EXT_LTO_CMFLAGS) -DCMAKE_Fortran_FLAGS="-Wall -Wextra -fbacktrace $(EXT_LTO)" -DCMAKE_INSTALL_PREFIX=`pwd` -DCM_ALL_FORTRAN=ON -DCM_ALL_USE_F08=$(ALL_USE_F08) -DCMAKE_BUILD_TYPE=Release -DCM_ALL_DEBUG=OFF -DCM_ALL_VTK_OUTPUT=ON -DVTK_DIR=`pwd`/../vtk_build/lib/cmake/vtk-7.1
+ $(MAKE) -C external/juelich_all_build
+ $(MAKE) -C external/juelich_all_build install
+
diff --git a/docs/modules/MPMIntegration.rst b/docs/modules/MPMIntegration.rst
new file mode 100644
index 0000000000000000000000000000000000000000..a0550b83d2229a3d75caf3b36b4f99a55c47851e
--- /dev/null
+++ b/docs/modules/MPMIntegration.rst
@@ -0,0 +1,228 @@
+.. In ReStructured Text (ReST) indentation and spacing are very important (it is how ReST knows what to do with your
+ document). For ReST to understand what you intend and to render it correctly please to keep the structure of this
+ template. Make sure that any time you use ReST syntax (such as for ".. sidebar::" below), it needs to be preceded
+ and followed by white space (if you see warnings when this file is built they this is a common origin for problems).
+
+.. We allow the template to be standalone, so that the library maintainers add it in the right place
+
+:orphan:
+
+.. Firstly, let's add technical info as a sidebar and allow text below to wrap around it. This list is a work in
+ progress, please help us improve it. We use *definition lists* of ReST_ to make this readable.
+
+.. sidebar:: Software Technical Information
+
+ Name
+ A Load Balancing Library (ALL)/GMPM-PoC
+
+ Language
+ Fortran/C/C++
+
+ Licence
+ TBD!
+
+ Documentation Tool
+ In source documentation using Doxygen, additional man pages and plain
+ text
+
+ Application Documentation
+ Non public/in repo
+
+ Relevant Training Material
+ `Webinar (YT) <https://www.youtube.com/watch?v=2K2YFdzIJF4&list=PLmhmpa4C4MzY02eaacXImTts2aGJHrdwQ&index=3>`_
+
+ Software Module Developed by
+ Stephan Schulz
+
+
+.. In the next line you have the name of how this module will be referenced in the main documentation (which you can
+ reference, in this case, as ":ref:`example`"). You *MUST* change the reference below from "example" to something
+ unique otherwise you will cause cross-referencing errors. The reference must come right before the heading for the
+ reference to work (so don't insert a comment between).
+
+.. _allMPMIntegration:
+
+###############
+MPM Integration
+###############
+
+.. Let's add a local table of contents to help people navigate the page
+
+.. contents:: :local:
+
+.. Add an abstract for a *general* audience here. Write a few lines that explains the "helicopter view" of why you are
+ creating this module. For example, you might say that "This module is a stepping stone to incorporating XXXX effects
+ into YYYY process, which in turn should allow ZZZZ to be simulated. If successful, this could make it possible to
+ produce compound AAAA while avoiding expensive process BBBB and CCCC."
+
+
+The material point method (MPM) is used to simulate continuous matter and
+is especially suited for the simulation of large deformations. Once large
+deformation are present, a dynamic load balancing solution is sensible to
+efficiently simulate large systems. Even if the initial work distribution
+is good, it is very often the case, that this distribution is much less so
+during the simulation run itself. The load balancing library ALL provides
+an easy plug and play solution to this problem and this module describes
+the details in how the library is integrated. Thanks to the good load
+balancing provided by the library larger systems can be simulated with
+less computational cost.
+
+Purpose of Module
+_________________
+
+.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
+
+This module shows the straight forwardness of including the load balancing
+library into already existing code. Depending on the simulation code
+additional precautions must be taken and those needed for the MPM
+simulation code are presented here. The prerequisites for the simulation
+code are also shown. Looking at these will help determine whether a
+simulation code is particularly suited for integrating ALL or if some
+further work is needed when integrating.
+
+This module also shows a real world application of the :ref:`Fortran
+interface<allFortranInterface>` provided with :ref:`ALL<ALL_background>`.
+
+The MPM simulation code with integrated ALL is used by Stephan Schulz in
+his thesis.
+
+.. If needed you can include latex mathematics like
+ :math:`\frac{ \sum_{t=0}^{N}f(t,k) }{N}`
+ which won't show up on GitLab/GitHub but will in final online documentation.
+
+
+Background Information
+______________________
+
+.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
+
+.. If the modifications are to an existing code base (which is typical) then this would be the place to name that
+ application. List any relevant urls and explain how to get access to that code. There needs to be enough information
+ here so that the person reading knows where to get the source code for the application, what version this information is
+ relevant for, whether this requires any additional patches/plugins, etc.
+
+.. Overall, this module is supposed to be self-contained, but linking to specific URLs with more detailed information is
+ encouraged. In other words, the reader should not need to do a websearch to understand the context of this module, all
+ the links they need should be already in this module.
+
+The load balancing library ALL is integrated into the material point
+method simulation code GMPM-PoC, which is written by Stephan Schulz during
+his thesis. The simulation code will be released to the public in the
+future.
+
+Certain aspects of the simulation code require additional treatment of the
+library, or additional features of the library. First, the open boundaries
+of the simulation code require continuous updates of the outer domain
+boundaries of the boundary domains. The system extent is adapted to the
+particle's bounding box each time step. This also means, the geometry
+generated in the last balance step by the library cannot be used directly.
+It is therefore saved by the simulation code, adapted to the new system
+extent and then given to the library as the basis for the new geometry.
+
+The communication is based on grid halos and only accommodates nearest
+neighbor communication. This causes the minimum domain size to be the
+width of exactly this halo. The library supports this feature and only the
+aforementioned outer domain bounds must be checked for compliance with the
+minimum size. The other domain boundaries are automatically sufficiently
+large due to the library.
+
+And lastly, the particle communication at the end of each time step also
+only accounts for nearest neighbor communication. This means, that a
+domain's boundary must not change so much, that it needs to retrieve
+particles from a process that is not its nearest neighbor. Due to the way
+the library moves boundaries in the staggered grid and tensor approaches,
+this is also already guaranteed to be true. There is always an overlap
+between the old domain's volume and the new domain's.
+
+However, the library also has a few requirements of the simulation code.
+Due to changing domains, particles must be able to be communicated across
+processes, which is implemented for all particle codes with moving
+particles. Depending on the load balancing method this communication may
+be more involved. In the case of the tensor method the topology does not
+change and every process has the same 26 neighbors during the entire
+simulation. If, however, the staggered grid approach is used, the
+communication must also handle changing number of neighbors and determine
+where they are and what regions they belong to. For example it is common,
+that one half of a boundary must be communicated to one process and the
+other to a different one. So the fixed relationship between boundaries and
+neighbors is broken up. The GMPM-PoC code was already designed with such a
+communication scheme in mind and provided the necessary flexibility to
+simply enable the staggered grid method after fixing a few communication
+bugs.
+
+
+Building and Testing
+____________________
+
+.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
+
+.. Provide the build information for the module here and explain how tests are run. This needs to be adequately detailed,
+ explaining if necessary any deviations from the normal build procedure of the application (and links to information
+ about the normal build process needs to be provided).
+
+To build the code just run ``make LB=ALL`` and everything should be build
+automatically including dependencies. Make sure the correct compiler are
+found in the path and if you want to use Intel compilers you need to set
+``COPMILER=INTEL`` as well. The normal caveats and required modules for
+some HPC systems are the described in the main code's ``README``.
+
+Source Code
+___________
+
+.. Notice the syntax of a URL reference below `Text <URL>`_ the backticks matter!
+
+.. Here link the source code *that was created for the module*. If you are using Github or GitLab and the `Gitflow Workflow
+ <https://www.atlassian.com/git/tutorials/comparing-workflows#gitflow-workflow>`_ you can point to your feature branch.
+ Linking to your pull/merge requests is even better. Otherwise you can link to the explicit commits.
+
+.. * `Link to a merge request containing my source code changes
+ <https://github.com/easybuilders/easybuild-easyblocks/pull/1106>`_
+
+.. There may be a situation where you cannot do such linking. In this case, I'll go through an example that uses a patch
+ file to highlight my source code changes, for that reason I would need to explain what code (including exact version
+ information), the source code is for.
+
+.. You can create a similar patch file by (for example if you are using git for your version control) making your changes
+ for the module in a feature branch and then doing something like the following:
+
+The main changes are the replacement of the original domain decomposition
+function which used to equi partition the system extent. Now, ALL is
+called to update the domain geometry.
+
+
+.. literalinclude:: ./MPMIntegration.F90
+ :linenos:
+ :language: Fortran
+
+To include the library and its VTK dependency into the existing make build
+system, the following snippets were used. This builds a 'minimal' VTK and
+links ALL against this. During the linking of the main simulation code VTK
+is linked using ``$(VTK_LIB)`` where the order is very important. The
+calling of make in this Makefile is deprecated and should be replaced by
+appropriate calls to ``cmake --build`` and ``cmake --install``.
+
+
+.. literalinclude:: ./MPMIntegration.makefile
+ :linenos:
+ :language: Makefile
+
+.. :emphasize-lines: 2,9-11
+.. If the patch is very long you will probably want to add it as a subpage which can be done as follows
+
+.. .. toctree::
+ :glob:
+ :maxdepth: 1
+
+.. patch
+
+.. Remember to change the reference "patch" for something unique in your patch file subpage or you will have
+ cross-referencing problems
+
+.. you can reference it with :ref:`patch`
+
+.. Here are the URL references used (which is alternative method to the one described above)
+
+.. _ReST: http://www.sphinx-doc.org/en/stable/rest.html
+.. _Sphinx: http://www.sphinx-doc.org/en/stable/markup/index.html
+
+.. vim: et sw=2 ts=2 tw=74 spell spelllang=en_us:
diff --git a/docs/modules/index.rst b/docs/modules/index.rst
index 1a97add19a53dd369b9c514cbf4b60009bafd890..1d68739305c026e2a1ab346d43afb56512bc89ce 100644
--- a/docs/modules/index.rst
+++ b/docs/modules/index.rst
@@ -6,6 +6,7 @@ E-CAM Module documentation
:caption: Modules
FortranInterface.rst
+ MPMIntegration.rst
.. vim: et sw=2 ts=2 tw=74 spell spelllang=en_us: