Chemistry and Materials Earth Sciences and Environment Engineering and Energy Fundamental physics Mathematics and Computer Science All All by publication date Recently published
Containerisation of NEMO Employing Singularity (CONES)
ARCHER2-eCSE02-13 : Dr James Harle (National Oceanography Centre)Subject Area:
Earth Sciences and Environment Containerisation is the process of packaging up a computer program, and everything else required to run it. Containerisation makes it easy to transfer the program between even quite disparate computing environments, and to scale up or down in a consistent and reproducible manner, e.g. from a local cluster to a national High Performance Computing facility such as ARCHER2, or even the cloud. The user needs minimal set-up knowledge to start using the program. This eCSE project investigated the containerisation of the NEMO ocean modelling program using the open-source Singularity software (now Apptainer). The containerised version was tested on ARCHER2, and its performance was compared to the conventional method of compiling and running. The project aimed to widen access to the NEMO ocean model within the ARCHER2 user community, and to provide a template for others wishing to containerise their code on ARCHER2. Read more...
Scaling up and coupling adaptive kinetic Monte Carlo with large-scale DFT
ARCHER2-eCSE05-04 : Prof Chris-Kriton Skylaris (University of Southampton)Subject Area:
Chemistry and Materials Diffusion of atoms and ions in solid state materials is essential for the function of products such as batteries, fuel cells and magnetic materials. Understanding atomic and/or ionic diffusion is therefore essential for the design and controlled manufacture of novel materials. However, computational techniques to study and analyse solid state diffusion are limited. The adaptive kinetic Monte Carlo method (aKMC) based on empirical potentials allows for material properties to be calculated efficiently, but it has clear limitations in its accuracy for certain chemical processes. In contrast, Density Functional Theory (DFT) calculations overcome these limitations, but are computationally more expensive. The recent availability of large numbers of cores on ARCHER2 provides an opportunity to combine these two approaches and develop methodologies capable of harnessing in excess of 100,000 cores. This eCSE project developed an interface to combine an aKMC program (ACDC) and a linear-scaling DFT program (ONETEP), making it possible to undertake large-scale aKMC simulations with a higher degree of accuracy than was previously possible. Read more...
uDALES: towards exa-scale simulation of air quality and microclimate in urban environments
ARCHER2-eCSE05-03 : Prof Maarten van Reeuwijk (Imperial College London)Subject Area:
Earth Sciences and Environment uDALES is a large-eddy simulation (LES) code used to model air quality and microclimate in urban environments at metre-scale resolution. Modelling urban environments is challenging due to the heterogeneous characteristics of urban landscapes, the turbulent nature of air flow within them and the effects of solar radiation on surface temperatures and turbulence. In this eCSE project the parallelisation of uDALES was upgraded to be able to fully exploit the large number of processors on ARCHER2. Tests with up to 130k cores show that the new code (uDALES V2.0) has excellent parallel performance, making it possible to perform simulations for longer durations (e.g. diurnal cycles) at much higher resolution, or study much larger areas. In addition, the limitation that buildings had to be aligned with the grid has been removed, which is particularly beneficial for simulations involving radiation, and when using the code to model realistic buildings. Read more...
Porting OptClim Optimisation system to ARCHER2
ARCHER2-eCSE04-07 : Prof Simon Tett (University of Edinburgh)Subject Area:
Earth Sciences and Environment OptClim is a software framework that uses optimisation algorithms to tune climate models. This project has enabled use of the OptClim software on ARCHER2, and enhanced the code to work with a wide range of models. This work will reduce the uncertainty in various climate and earth system models. Read more...
Optimising NEMO-ERSEM for High Resolution
ARCHER2-eCSE04-06 : Dr Jerry J C Blackford (Plymouth Marine Laboratory)Subject Area:
Earth Sciences and Environment NEMO-FABM-ERSEM is a software system for simulating the biogeochemistry of the marine environment. It is used by many research groups to address diverse topics from climate change to pollutants. However, the code is expensive to run, both in terms of compute time and storage requirements. Understanding which parts of the code are the most costly to run, or have the greatest impact on its performance, can guide future developments of the code, potentially leading to future simulations running more efficiently. This eCSE project focused on optimising the performance of the newest version of NEMO-FABM-ERSEM on ARCHER2, based on simulations in the Northwest European Shelf. A runtime reduction of up to 4% was achieved, and recommendations were produced for optimising both performance and resource costs on ARCHER2. Further, an analysis of the overall energy consumption of the code has been produced, which will help users to prioritise either optimal performance or optimal energy efficiency. Read more...
Scalable and robust Firedrake deployment on ARCHER2 and beyond
ARCHER2-eCSE04-05 : Dr David A Ham (Imperial College London)Subject Area:
Mathematics and Computer Science This project aimed to standardise and simplify the build procedure for Firedrake, a system for solving partial differential equations. Previously, expert HPC knowledge and significant time investment were required to carry out bespoke installation on HPC systems. Using Spack, a package manager specifically designed for installing popular software on HPC systems, a package was created which allows a non-expert user to quickly install Firedrake on an HPC system in a single line, saving a huge amount of time. Further Spack packages were also created for other applications that themselves depend on Firedrake, including applications for modelling weather, coastal ocean zones and glacier flow. Read more...
Upscaling ExaHyPE
ARCHER2-eCSE04-02 : Dr Tobias Weinzierl (University of Durham)Subject Area:
Mathematics and Computer Science ExaHyPE is an open-source simulation engine which is used for astrophysical challenges and seismic problems such as earthquake simulation. It hides most of the computing aspects from users, who need only enter the relevant scientific parameters when creating a new simulation code. This means a new simulation can be prepared in just a few hours, whereas it could take weeks, months, or even years to write from scratch. ExaHyPE is also designed to be able to exploit the future generation of exascale computers, which will have many more cores per node than current supercomputers. The need to harvest these cores makes task-based programming increasingly attractive. However, introducing tasks into codes also introduces other complexities. This eCSE project carried out modifications which have allowed the ExaHyPE code to run astrophysical simulations with unprecedented compute efficiency. As electricity costs rise and concerns grow over the environmental impact of supercomputing, there is an ever-growing focus on the need to increase the efficiency of simulation codes. Read more...
Realising a modular data interface to couple quantum mechanical calculators with external data-driven workflows
ARCHER2-eCSE03-10 : Dr Andrew Logsdail (Cardiff University)Subject Area:
Chemistry and Materials Electronic structure calculations are the workhorse of modern computational chemistry, allowing for the prediction of information for a range of vital societal applications, such as drug efficacy, composition of new solar cells, or environmental impact of pollutants. However, the intrinsically complicated nature of electron interactions leads to a great variety of computational chemistry software. Complex interoperation between computer codes can slow the progress of research. To address this, a new applications programming interface, Atomic Simulation Interface (ASI), was developed and then implemented in established electronic structure software packages. This offers a unified and efficient way to export and import large data structures used in electronic structure calculations and for classical molecular dynamics simulations. Read more...
Porting, Testing, Optimising, and Configuring the Community Earth System Model (CESM2) on ARCHER2
ARCHER2-eCSE03-06 : Dr Ioana Colfescu (National Centre for Atmospheric Science)Subject Area:
Earth Sciences and Environment The Community Earth System Model 2 (CESM2) is a world-leading global climate model that is widely used by the international research community for state-of-the-art simulations of the Earth’s past, present and future climate system. This project aimed at porting, optimising, testing, and validating the model’s key configurations and making them available to UK researchers through ARCHER2, thus enabling them to use a state-of-the-art climate model on a next-generation supercomputing service. Prior to this work, before a user could run their own simulations on ARCHER2, they would face a large amount of technical set-up work, at the expense of time spent on their research. This project allows UK researchers using ARCHER2 to confidently and reliably use CESM2 with minimum efforts. Read more...
Goal-oriented mesh adaptation for Firedrake
ARCHER2-eCSE03-04 : Prof Matthew Piggott (Imperial College London)Subject Area:
Earth Sciences and Environment Mesh adaptation can be a very powerful tool for improving the accuracy and/or efficiency of numerical simulations, e.g. for geoscientific modelling. However, it is still not widely used, largely because it requires users to have considerable experience and in-depth understanding. This eCSE project produced a new goal-oriented error estimation module, Pyroteus, which enables numerical modellers to use advanced techniques, without requiring expert knowledge. All that is required from the user is a set of instructions for how to run the numerical model and compute the user-specified Quantity of Interest, both of which are typically straightforward. With these, Pyroteus is able to perform mesh adaptation in a fully automated sense. To support the development of Pyroteus, this project also involved extending the mesh adaptation functionalities of the scientific software packages PETSc and Firedrake, upon which Pyroteus depends. Read more...
Developing in-situ analysis capabilities for pre-Exascale simulations with Xcompact3D
ARCHER2-eCSE03-02 : Dr Sylvain Laizet (Imperial College London)Subject Area:
Engineering and Energy Many of the environmental and energy-related issues we face today cannot possibly be tackled without a better understanding of the dynamics of fluids. The design of many engineering and industrial systems as well as the prediction of their impact on the environment greatly relies on the turbulent behaviour of fluid flows being properly quantified. Significant progress has been made recently using high performance computing, and Computational Fluid Dynamics is now a critical complement to experiments and theories. Xcompact3D is a high-order CFD framework designed for simulating turbulent flows on supercomputers, with applications ranging from fundamental turbulence studies to simulating and optimising wind farm designs. As computational power has increased, the rate at which data can be stored has not kept pace with the rate at which it can be generated, leading to the so-called “I/O bottleneck”. To address this in Xcompact3D, this project has implemented a new I/O system using the ADIOS2 library to facilitate user-defined in situ analyses. Read more...
Containerisation of NEMO Employing Singularity (CONES)
ARCHER2-eCSE02-13 : Dr James Harle (National Oceanography Centre)Subject Area:
Earth Sciences and Environment Containerisation is the process of packaging up a computer program, and everything else required to run it. Containerisation makes it easy to transfer the program between even quite disparate computing environments, and to scale up or down in a consistent and reproducible manner, e.g. from a local cluster to a national High Performance Computing facility such as ARCHER2, or even the cloud. The user needs minimal set-up knowledge to start using the program. This eCSE project investigated the containerisation of the NEMO ocean modelling program using the open-source Singularity software (now Apptainer). The containerised version was tested on ARCHER2, and its performance was compared to the conventional method of compiling and running. The project aimed to widen access to the NEMO ocean model within the ARCHER2 user community, and to provide a template for others wishing to containerise their code on ARCHER2. Read more...
Massively parallel and scalable electromagnetic solver for fast analysis of nonlinear optical processes in large clusters of nanoparticles
ARCHER2-eCSE02-12 : Prof Nicolae C. Panoiu (University College London)Subject Area:
Engineering and Energy OPTIMET-3D is a numerical electromagnetic (EM) solver for analysis of EM wave scattering from clusters of nanoparticles embedded in a homogeneous medium. This project introduced powerful new functionalities to OPTIMET, including the ability to handle some of the most common non-spherical particle morphologies encountered in nanotechnology applications, such as ellipsoids, cylinders and spheroids. Further, the number of particles that OPTIMET can now handle is at least an order of magnitude larger than it was previously, and it can now analyse clusters of nanoparticles many orders of magnitude faster than commercial solvers, which are based on the finite element method. OPTIMET is believed to be the first code adapted specially for the fast and accurate analysis of linear and nonlinear second harmonic (SH) optical phenomena pertaining to systems of very large number of particles. The addition of the new module has substantially expanded the scientific areas which can benefit from OPTIMET, as the SH interactions phenomenon lies at the heart of many applications, including optical sensors, plasmonic nanoantennae, photonic and plasmonic crystals, metamaterials and wavelength converters. Read more...
Optimizing BOUT++ MPI+OpenMP hybrid performance by refactoring compute kernels
ARCHER2-eCSE02-11 : Dr Joseph Parker (United Kingdom Atomic Energy Authority)Subject Area:
Engineering and Energy For 70 years, humans have been pursuing nuclear fusion as a clean, safe and near limitless source of energy production. Major experimental nuclear fusion reactors (“tokamaks”) already exist, and planning for demonstration power plants is now well underway. BOUT++ is a software framework for simulating the plasma inside a tokamak. It is designed to be both performant and easy-to-use, and can run across a range of computers, from laptops to national computing facilities like ARCHER2. However, to run on the world’s largest supercomputers, BOUT++ needs to further exploit parallelism – the ability to solve many interlinked components of a problem concurrently. This eCSE project improved the parallel efficiency of the BOUT++ code. Users can now produce higher-fidelity simulations of tokamaks, thereby increasing our understanding of critical plasma physics processes. Currently tractable simulations can be run in a shorter time, and larger ensembles of simulations can be run. This is crucial for enabling faster parameter scans and performing uncertainty quantification (UQ), a technique that is becoming increasingly important for assessing the robustness of physics results. Read more...
Optimising MITgcm on ARCHER2: efficient numerical simulation and data assimilation tools for studying the ocean, atmosphere, and cryosphere
ARCHER2-eCSE02-06 : Dr Emma JD Boland (British Antarctic Survey)Subject Area:
Earth Sciences and Environment MITgcm is a state-of-the-art general circulation model used to enhance understanding of the atmosphere and ocean and their interactions with land- and sea-ice. This project involved the porting to ARCHER2 of three exemplar models that are built on the MITgcm core, two addressing the Amundsen Sea in the Antarctic and one global ocean state estimate. These exemplar models were then optimised, achieving a speed-up of the order of 2-fold in a widely used setup. This work is already helping researchers address more complex problems with their models, as well as allowing for more efficient use of valuable computer resources. Read more...
Support for advanced transition state search techniques in CASTEP
ARCHER2-eCSE02-04 : Dr James R Kermode (University of Warwick)Subject Area:
Chemistry and Materials The CASTEP density functional theory code is a UK flagship code, specialised for solid materials, and is heavily used on ARCHER2 (300-400 active users). While support for obtaining the ground-state electronic and atomic configurations is now very good, computing transition states, reaction rates, and exploring free energy barriers with enhanced sampling were relatively poorly supported before this eCSE was completed, despite their importance for a wide range of chemistry and materials science applications. We have implemented two key new features in the CASTEP code to address these issues: (i) support for the widely used nudged elastic band (NEB) transition state search tool, augmented by a state-of-the-art robust optimizer; (ii) an interface to the i-PI universal force engine, which allows CASTEP to be connected efficiently to a range of external codes with enhanced sampling capabilities. The new tools will aid in reconciling experimental observations with atomic-scale behaviour, helping to guide and interpret future experiments. Read more...
Reducing UM-UKCA data output using flight-track simulation
ARCHER2-eCSE02-02 : Dr Nathan Luke Abraham (National Centre for Atmospheric Science (NCAS) & University of Cambridge)Subject Area:
Earth Sciences and Environment The UK Met Office Unified Model (UM) is a numerical model of the atmosphere used for both weather and climate applications. The main goal of this eCSE project was to allow for easy comparison between atmospheric model data, obtained from the United Kingdom Chemistry and Aerosols model used as part of the Unified Model (UM-UKCA), and observed data measured from research aircrafts. Prior to this work, this task involved the output of a large amount of model data, of which only a small part was used. Post-processing of this large dataset was very time-intensive and required the user to have in-depth knowledge of aircraft data, as well as technical and coding skills. This project developed a stand-alone “flight-track simulator” code which deals with all the required data processing while the model runs, greatly reducing the data storage requirement. The processed model data is also directly available as soon as the model completes, making it much quicker and easier for a wide variety of users to benefit. Although the flight-track simulator code was originally designed exclusively for use with UM-UKCA, it has been developed as a stand-alone code which can be used with any other atmospheric and climate model, therefore benefiting the wider atmospheric science community. Read more...
Multi-Resolution Coupling for Exascale Engineering
ARCHER2-eCSE01-28 : Prof Alistair Revell (University of Manchester)Subject Area:
Engineering and Energy Motivation for so-called multi-scale modelling is all around us in the everyday world, where small changes at a very small scale can impact on large, system-level, engineering processes, e.g. the impact of surface degradation on the performance of a heat exchanger, or the clogging up of porous surfaces over time. The ability for computational science to bridge scales from both macro- and microscale physics to mesoscale modelling is likely to benefit a broad range of computational scientists and engineers. This project has provided two fully functional examples of coupling between popular computational macro- and microscale simulation tools (Code_Saturne and LAMMPS respectively) and the mesoscale LUMA Lattice Boltzmann code. The work lays a solid foundation for future multi-scale simulations for industrial engineering applications. Read more...
A Partitioned Fluid-Structure Interaction Framework for Exascale
ARCHER2-eCSE01-22 : Dr Alex S Skillen (University of Manchester)Subject Area:
Engineering and Energy Fluid-structure interaction (FSI) occurs frequently in the field of renewable and low-carbon energy generation. Simulation of FSI problems is highly computationally demanding. This project developed an efficient and highly scalable FSI simulation tool, ParaSiF_CF, which can be used for a wide range of simulation cases, including extremely large problems involving “typical” operating conditions in the fields of offshore wind, marine turbines and nuclear energy, and the simulation of plastic polymer or composite material components that are involved in offshore turbines. Read more...
Scalable I/O and checkpointing for Firedrake
ARCHER2-eCSE01-20 : Dr David A Ham (Imperial College London)Subject Area:
Mathematics and Computer Science Firedrake is an automated system for the solution of partial differential equations using the finite element method (FEM). This project improved the input/output capability of Firedrake for the running and analysis of simulations such as geophysical fluid dynamics. Other applications of Firedrake include coastal ocean simulations, weather forecasting, and marine renewable energy supply. This project also extended the HDF5 interface of the Portable, Extensible Toolkit for Scientific Computation (PETSc). Read more...
Improving the performance of DL_MONTE for large-scale simulations
ARCHER2-eCSE01-19 : Prof Steve Parker (University of Bath)Subject Area:
Chemistry and Materials Monte Carlo molecular simulation (MCMS) entails using random numbers to calculate properties of solids or fluids at the atomic scale. It is the method of choice for studying various physical phenomena of key relevance to technology: common applications include quantifying the amount of a gas which adsorbs to a surface or material; and calculating the atomic-scale properties of fluids and mixtures. There is an insatiable demand for computer simulation to be able to model more complex systems, e.g. systems comprised of more atoms, or more complex molecules. Motivated by this, this project has enabled an open-source MCMS computer program, DL_MONTE, to simulate significantly more complex systems than it could previously. Read more...
Achieving the sustainability and scalability of numeric-atomic-orbital-based linear response and electron-phonon functionality in FHI-aims
ARCHER2-eCSE01-16 : Dr Reinhard Maurer (University of Warwick)Subject Area:
Chemistry and Materials Most calculations run on ARCHER2 will employ electronic structure software packages, which are designed to solve the Schrödinger equation for molecules and materials to obtain their ground state properties. FHI-aims is one such software package, designed to be efficient when running everything from small calculations on standard laptops to huge calculations involving millions of atoms on the largest High Performance Computing systems such as ARCHER2. It is of wide interest to also calculate how molecules and materials respond to atomic displacements, or to electric and magnetic fields. This is possible using density functional perturbation theory (DFPT). A portion of the existing DFPT infrastructure within FHI-aims has been overhauled within this project, to make it more modular, intuitive, scalable, and user and developer friendly. These changes will help researchers to discover and design the next generation of materials more quickly, more cheaply, and more efficiently. Read more...
Zacros Software Package Development: Towards Petascale Kinetic Monte Carlo Simulations with the Time-Warp Algorithm
ARCHER2-eCSE01-13 : Dr Michail Stamatakis (University College London)Subject Area:
Chemistry and Materials The chemical industry underpins virtually all sectors of the economy, from healthcare to construction. Catalytic materials, which accelerate reactions, are essential to this industry. Yet discovering catalytic materials and building catalytic processes is non-trivial. Theory and simulation can guide such efforts by delivering fundamental understanding of catalytic function. Zacros is a computational code that simulates chemical events on catalytic surfaces and enables scientists to understand how catalysts accelerate chemical reactions. This eCSE project investigated how the performance of the code is affected by different user-tuneable parameters that affect the efficiency of the parallel algorithm implemented in Zacros. The most important parameter affecting the efficiency of a run was thus identified, and tools were developed that allow users to optimise it for the systems they would be interested in simulating. The improvements made to the code will facilitate further development of improved algorithms and new features in the future. Read more...
ParaGEMS: Integrating discrete exterior calculus (DEC) into ParaFEM for geometric analysis of solid mechanics
ARCHER2-eCSE01-12 : Prof Lee Margetts (University of Manchester)Subject Area:
Engineering and Energy This project integrated the geometric and topological functions of the new discrete exterior calculus (DEC) library ParaGEMS into ParaFEM, a well-established open-source finite-element library. A series of five MiniApps was developed and optimised to model elasticity and diffusion on synthetic material micro-structures with existing or emerging heterogeneities and discontinuities. The outputs of the project will support new innovations promised by the recently funded UK Collaborative Computational Project on wave-structure interaction (CCP-WSI+), which brings together cutting-edge research in both fluids and computational solid mechanics to advance research into offshore energy generation. Read more...
Multi-Layered MPI parallelisation for the R-matrix with time-dependence code
ARCHER2-eCSE01-11 : Dr Andrew Brown (Queen’s University Belfast)Subject Area:
Fundamental physics Understanding and controlling light-induced charge transfer in molecules promises societal benefits in the context of medicine (vision), biology (photosynthesis) and energy (solar cells). However, describing these processes requires the investigation of intricate, time-dependent electronic dynamics in complex systems. Our research community develops and employs the R-matrix with time-dependence code (RMT) to study these phenomena, primarily in atoms at present. In this eCSE project, we have introduced new parallelism to RMT, facilitating cutting-edge calculations and enhancing the efficiency of ongoing applications. By enabling calculations for larger and more complex systems we approach the ultimate goal of understanding and controlling charge transfer in molecules with potential applications in medicine, biology and energy. Read more...
CASTEP Solvation Forces
ARCHER2-eCSE01-09 : Prof Matt Probert (University of York)Subject Area:
Chemistry and Materials Density-functional theory (DFT) is a quantum mechanical modelling method which is widely used in physics, chemistry and materials science to study the properties of materials where electrons dictate their behaviour. CASTEP is a UK flagship code which uses DFT to simulate a wide range of properties of materials, such as structure at the atomic level or the vibrational properties of the material. It is one of the most used codes on ARCHER2. Prior to this eCSE project, CASTEP was well adapted to investigating electronic and atomic configurations in solid materials, but not so well suited to investigating isolated molecules, or molecules in solvent, for which there is a wide range of chemistry and materials science applications. The aim of this eCSE project was to add new functionality to CASTEP to address this shortcoming, building on the work carried out in a previous ARCHER eCSE project. The result enables CASTEP users to accurately simulate molecules in solvent, without the cost of explicit solvent molecules. This will therefore open wider avenues of research with CASTEP, for instance, the study of pharmaceutical compounds and future battery designs. Read more...
High fidelity simulations of moving objects in a turbulent flow using a Cartesian mesh
ARCHER2-eCSE01-06 : Dr Sylvain Laizet (Imperial College London)Subject Area:
Engineering and Energy Developing accurate, efficient and scalable tools to simulate arbitrarily complex moving geometries in turbulent flows remains a considerable challenge in Computational Fluid Dynamics (CFD). Fluid-Structure Interaction (FSI) problems, where one or more complex solid structures interact and modify the behaviour of the surrounding fluid, pose an even greater challenge. FSIs are commonly found in nature and in many engineering fields such as energy (fixed and floating wind turbines, wave energy converters) and biomedical engineering (e.g. heart valves). In this project, new capabilities have been implemented in the high-order finite-difference framework Xcompact3d, and a Computer-Aided Design (CAD) interface has been developed, to facilitate high-fidelity simulations of incompressible turbulent flows. Read more...