File input and output often become a severe bottleneck when parallel applications run on large numbers of processors. Simple methods such as writing a separate file-per-process or performing all IO via a single controller process are no longer feasible at scale. In order to take advantage of the full potential of modern parallel file systems such as Lustre, IO also needs to be done in parallel.
This talk will cover investigations of the parallel IO performance that can be achieved on ARCHER2 and compare to results on its predecessor, ARCHER, which was a Cray XC30. Popular parallel IO libraries such as MPI-IO, HDF5 and NetCDF will be benchmarked.
Results are obtained using the simple benchio benchmark - https://github.com/davidhenty/benchio, which writes a large, regular, three-dimensional distributed dataset to file.
The aim is to be able to give users advice on the best parallel IO settings for their own applications. As the Lustre filesystem on ARCHER2 (an HPE Cray EX) has very different performance characteristics to that of ARCHER, previous experience may not be useful in achieving good IO rates.
This online session is open to all. It will use the Blackboard Collaborate platform.