Several petabytes of Earth Science data are already in the HDF file formats and the collections are still growing. The HDF Group is strategically committed to support producers and users of these data as their access patterns vary throughout the data use lifecycle with evolving applications, computing frameworks, and backend storage systems. This is especially important for seamless transition from on-prem filesystem-based to cloud computing. This commitment is reflected in HDF Group’s long-time collaboration with the netCDF library developers, and more recent work to support efficient access to HDF5 files for the Zarr-based applications. Both of these data formats are popular in the ESIP community.
View Notes
• Elena Pourmal (NASA EED-2 / HDF Group): HDF - Current Status and Future Directions
• J. P. Swinski (NASA): H5Coro: The HDF5 Cloud-Optimized Read-Only Library
NASA’s migration of science data products and services to AWS has sparked a debate on the best way to access science data stored in the cloud. Given that a large portion of NASA’s science data is in the HDF5 format or one of its derivatives, a growing number of efforts are looking at ways to efficiently access H5 files residing in S3. This presentation describes one of those efforts, H5Coro, and argues for the creation of a standardized subset of the HDF5 specification targeting cloud environments. H5Coro is an open-source C++ module written from scratch that implements a performant HDF5 reader for H5 files that reside in S3. It targets high latency/high throughput environments by minimizing the number of I/O operations through caching and intelligent range GETs. H5Coro is currently available as a C library and includes Python bindings.
• John Readey (The HDF Group): HDF for the Cloud - Serverless HDFThe HDF Server (HSDS) provides a convenient method for running HDF applications in the cloud (utilizing scalable compute and object-based storage), but sometimes setting up a server is just too much to deal with due to cost, time, or management concerns. In this talk we'll discuss two alternative ways to utilize HSDS technology but leaving the server aspect behind. The first, HSDS for AWS Lambda supports the HDF REST API but runs entirely using Lambda functions. The second approach is "HSDS Direct Access", a client-side library that enables HSDS-like features exclusively on the client: Object storage read and write, multi-threading support, and sql-stye queries.
•
Ellen Johnson (MathWorks): MATLAB Modernization on HDF5 1.10, Support for SWMR and VDS, and Cloud Data Access
This talk presents our effort at MathWorks toward modernizing on HDF5 1.10.7 and adding support for the much-requested Single-Writer/Multiple-Reader and Virtual Dataset features. We will discuss our updated 1.10.7 HDF5 functionality available today for MATLAB users in the R2021b prerelease (with R2021b full release planned for September) and would like to hear early feedback from the community. We will also discuss MATLAB capabilities for working with HDF5 data hosted on S3, Azure, and Hadoop introduced in R2020b which we have now enabled for Virtual Datasets. We will wrap up with performance and compatibility considerations plus our tentative roadmap for future HDF5 enhancements.
• Joe Lee (NASA EED-2 / HDF Group): HDFEOS.org User Analysis, Updates, and Future
Video Stream
