The h5cpp library developed by DESY and ESS is a c++ wrapper for the HDF5 library. Using modern c++ features it simplifies creation of HDF5 files. The pninexus library adds a set of advance tools, e.g. a file structure builder from XML configuration. The libraries with their python binding are heavily used by the PETRA III experiment @ DESY in our detector Tango servers and our NeXus metadata...
The Open Standard for Particle-Mesh Data (openPMD) is a F.A.I.R. metadata standard for tabular (particle/dataframe) and structured mesh data in science and engineering.
We show the basic components of openPMD, its extensions to specific domains, applications from laser-plasma physics, particle accelerators, material physics to imaging and the ability to bridge multiple heterogeneous...
https://gitlab.com/helmholtz-berlin/nexuscreator
https://gitlab.com/helmholtz-berlin/nexuscreatorpy
The research data management group at Helmholtz-Zentrum Berlin is applying FAIR data management. Data starts to be moved from specific file formats into NeXus/HDF5 files. The standardization program involves the conversion of already generated data, and the automation for the creation of...
The materials science beamline, ID11, at the ESRF, was upgraded in 2020 to get a Dectris Eiger 4M pixel detector. This can record diffraction frames at 500 Hz while samples are rotated and scanned in a tiny (~150 nm) X-ray beam. Reconstruction of the diffraction data can eventually give detailed images of all the crystals inside the materials. The large quantities of data can be problematic to...
Serial crystallography (SX) has become an established technique for protein structure determination, especially when dealing with small or radiation-sensitive crystals and investigating fast or irreversible protein dynamics. The advent of newly developed multi-megapixel X-ray area detectors, capable of capturing over 1000 images per second, has brought about substantial benefits. However, this...
High bandwidth instruments (data production rates of GB/s) have proliferated in photon science experimental facilities in the last years across the globe. Some of them are planned to be operated 24/7. Data volumes thus produced exceed both the budget of storage facilities and sometimes even the ingest capacities of hardware.
In this talk, I'd like to highlight key challenges when...
HSDS (Highly Scalable Data Service) is a REST-based web service that supports most of the features of the HDF library, but running as a service. HSDS supports the standard HDF compressors as well as BLOSC-based compressors "out of the box". In addition, HSDS supports parallel compression/decompression and supports using compression with variable length datatypes. This talk will cover how...
