This workshop will include two talks about optimizing, profiling, and benchmarking input/output operations using HDF5 and DLIO.
HDF5 is a widely used data model, file format, and I/O library, particularly in HPC applications for managing and storing large amounts of simulation data. This talk will focus on HDF5 usage on NCSA's Delta system and provide a brief overview of HDF5 (serial and parallel) with an emphasis on HDF5 HPC tuning techniques such as collective metadata I/O, data aggregation, asynchronous I/O, and other HDF5 tuning parameters and features. During the presentation, we will discuss various storage options available on Delta for HDF5 files and multiple post-simulation storage options, including cloud storage and other tools.
In this talk, we will be taking a dive into the unique features of I/O in deep learning application. We will present our ongoing efforts to understand the I/O characteristics of DL workloads using our DLIO profiler. Finally, we will present the DLIO Benchmark which is built to accurately represent the I/O characteristics in deep learning workloads.
Date and location: these talks will be presented virtually via Zoom on 3/12/2024, from 1 to 3 PM CT.