ParaGEMS: Integrating discrete exterior calculus (DEC) into ParaFEM for geometric analysis of solid mechanics

ARCHER2-eCSE01-12

---

This ARCHER2 eCSE project integrated a newly developed math library for discrete exterior calculus (DEC) into the open-source finite-element (FE) library ParaFEM. A series of five MiniApps were developed and optimised to model elasticity and diffusion on synthetic material micro-structures with existing or emerging heterogeneities and discontinuities. The implementation allows for both FE and DEC to be applied within the same framework, creating the possibility for coupled simulations in the future. Approximately 80% parallel efficiency was realised on 8000 cores for a problem involving 135M unknowns and trends indicate this efficiency can be extended to higher core counts with larger meshes.

Figure 1 - simulation process from material micro-structure to solution with emerging discontinuities

Impact: DEC is a relatively new mathematical framework built on a fundamentally discrete view of the world, which also retains global macroscopic properties. This makes it ideal for simulating non-smooth physical processes across length scales from the molecular (micro), through to the engineering (macro). Interest in DEC is growing internationally in science and engineering – the development of DEC for solid mechanics is a key research activity at Manchester. However, progress has been limited by the lack of efficient parallel software libraries to test new formulations at scale.

Integrating the newly developed DEC library into ParaFEM has enabled reuse of well-established code for I/O, partitioning & load balancing, inter-process communication, as well as scalable solvers. Furthermore, it will enable reuse of existing ParaFEM frameworks for multiscale and multiphysics problems in the future: for example, DEC can be used in place of cellular automata and/or microFE for grain-scale modelling. The outputs of this project are timely in that they will support new innovations promised by investigators in the recently funded UK Collaborative Computational Project on wave-structure interaction (CCP-WSI+). CCP-WSI+ brings together cutting-edge researchers in both fluids and computational solid mechanics who will work together over the next 5 years to advance research into offshore energy generation.

Codes: ParaFEM is a well-established open-source finite-element library for parallel solution of various types of problems, including stress analysis, heat flow, eigenvalue and forced vibrations. It is written in modern Fortran with MPI parallelism, and includes interfaces to several scalable libraries, such as BLAS, LAPACK, METIS and PETSc. Furthermore, it has been coupled with third party software packages, such as CGPACK and OpenFOAM, to enable multi-scale and multi-physics simulations. ParaFEM has been ported to many HPC systems and has excellent scaling on 10,000s cores.

The DEC library, ParaGEMS, was developed as part of the EPSRC Fellowship EP/N026136/1 and has already been applied independently to two field heat diffusion in cracked media. Like ParaFEM, the library is written in modern Fortran with MPI parallelism and interfaces to BLAS, LAPACK, and PETSc. The sparse structure of DEC operators yields a smaller memory footprint than traditional finite-elements and their form better facilitates changes in topology. ParaFEM/ParaGEMS on github; and paragems_devel on bitbucket

References:

• P Boom, A Seepujak, O Kosmas, L Margetts, A Jivkov, “Parallelized Discrete Exterior Calculus for Three-Dimensional Elliptic Problems,” Computer Programs in Physics, Vol 279, 108456, 2022
• Pieter D. Boom, Andrey P. Jivkov, Lee Margetts, ParaGEMS: Integrating discrete exterior calculus (DEC) into ParaFEM for geometric analysis of solid mechanics, SoftwareX, Volume 21, 2023, 101280, ISSN 2352-7110, https://doi.org/10.1016/j.softx.2022.101280. (https://www.sciencedirect.com/science/article/pii/S2352711022001984)

Information about the code

The integration of ParaGEMS and ParaFEM, including the associated MiniApps, is available for download.

This requires pre-download of the ParaFEM library, but does include the ParaGEMS (DEC) library as a subdirectory.

The development head of the ParaGEMS library, and its independent MiniApps (not involving ParaFEM), is also available for download.

Webinar

Webinar presenting this work

Technical Report

Download as PDF

Paper

Pieter D. Boom, Andrey P. Jivkov, Lee Margetts, ParaGEMS: Integrating discrete exterior calculus (DEC) into ParaFEM for geometric analysis of solid mechanics, SoftwareX, Volume 21, 2023, 101280, ISSN 2352-7110, .

DOIs

• SoftwareX paper: 10.1016/j.softx.2022.101280