One of the greatest challenges to running parallel applications on large numbers of processors is how to handle file IO. The IO routines provided by your favourite serial programming language are not designed with parallelism in mind, and IO overheads can grow to dominate the overall runtime. Parallel file systems, such as ARCHER2’s Lustre, are optimised for large data transfers but performance can be far from optimal if every process opens its own file or if all IO is funnelled through a single controller process.
Many libraries allow for true parallel IO, including MPI-IO, HDF5, NetCDF and ADIOS. However, the user must call them correctly and configure the filesystem appropriately to obtain good performance.
This talk will cover:
- the challenges of parallel IO;
- parallel filesystems and the Lustre configuration on ARCHER2;
- efficient use of Lustre;
- initial results from the ARCHER2 production system.
The presentation will be kept relatively short to allow for interactive discussions and a Q&A session.
This online session is open to all. It will use the Blackboard Collaborate platform.