Conveners
HPC Technology: Storage and IO 1
- Jay Lofstead (Sandia National Laboratories)
HPC Technology: Storage and IO 2
- Jay Lofstead (Sandia National Laboratories)
HPC Technology
- Chair: Weronika Filinger (EPCC, The University of Edinburgh)
HPC Technology
- There are no conveners in this block
HPC Technology
- There are no conveners in this block
High-performance computing (HPC) applications generate massive amounts of data. However, the performance improvement of disk-based storage systems has been much slower than that of memory, creating a significant I/O performance gap. To reduce the performance gap, storage subsystems are under extensive changes, adopting new technologies and adding more layers into the memory/storage hierarchy....
The MeerKAT archive was made accessible from the internet earlier this year.
This allow researchers from across the world to pull data from the MeerKAT archive.
In this talk I'll describe the MeerKAT storage system that consists out of several petabytes of observations backed by a Ceph distributed storage system. The use of Ceph at SARAO and the infrastructure around the archive will be...
Accelerating discovery in computational science and high performance computing environments requires compute, network and storage to keep pace with technological innovations. Within a single organization, interdepartmental and multi-site sharing of assets has become more and more crucial to success. Furthermore, as the growth of data is constantly expanding, storage workflows are exceeding the...
The rise of AI/ML in HPC applications is also driving the need for suitable storage abstractions such as the key-value (KV) stores. These abstractions pose new challenges for the HPC I/O stack. Enterprise KV stores are not well suited for HPC applications, and entail customization and cumbersome end-to-end KV design to extract the applications needs. To this end, I will present BESPOKV, an...
Portable, Extensible Toolkit for Scientific Computation (PETSc) is a suite of
data structures and routines for the scalable (parallel) solution of scientific applications. Due to its solid mathematical grounding, careful software design, and most importantly, evolution resulting from the usage of many users in various application areas, PETSc is enabling engineers and scientists to solve...
Memory and storage read and write costs can lead to a significant loss of time and energy in current HPC systems. Byte-addressable non-volatile memory (NVM) could provide considerable improvements in both time and energy requirements over conventional DRAM memory. Using Optane DCPMM, Intel's new byte-addressable and persistent memory, the NEXTGenIO project investigated the performance of NVRAM...
We investigate the parallelisation and performance analyses of search and planning algorithms for artificial intelligence, machine learning, and software verification. These applications involve the exploration of large state spaces, which requires at its core a combinatorial search algorithm. Much of our work, therefore, focuses on evaluating and improving the scalability of algorithms used...
Machine Learning methodologies and tools have delivered new approaches to scientific computing ranging from new approximation methods to solve differential equations to leveraging advantages of ML hardware over traditional HPC hardware. It is not unlikely that such approaches will be helpful to computational problems that have seen little progress for decades. We will discuss a few examples,...
Two of the typical points of interest with elevation data, or Geographic Information Systems
(GIS) data in general, are storage and query costs. The former is typically addressed by
integrating standard compression schemes into already existing storage mechanisms, such
as GZIP in HDF5. Space-Filling Curves (SFCs) have already been used to reduce access
time for spatial operations on point...
Generally, scientific simulations load the entire simulation domain into memory because most, if not all, of the data changes with each timestep. This has driven application structures that have, in turn, affected the design of popular IO libraries, such as HDF-5, ADIOS, and NetCDF. All of these libraries assume that each output written will be a complete simulation domain. While a time...
