Speaker
Description
The materials science beamline, ID11, at the ESRF, was upgraded in 2020 to get a Dectris Eiger 4M pixel detector. This can record diffraction frames at 500 Hz while samples are rotated and scanned in a tiny (~150 nm) X-ray beam. Reconstruction of the diffraction data can eventually give detailed images of all the crystals inside the materials. The large quantities of data can be problematic to process, a single scan may contain millions of frames. This contribution will review our experience in the last few years working with these data. Raw frames are recorded into hdf5 files using bitshuffle and lz4 compression. For many of the larger datasets, the diffraction data are very sparse, so converting to a sparse format helps a lot. While this first step is bounded by IO and decompression, but the format conversion can be done in parallel over frames. The rest of our processing is based on these sparse data rather than full images.
| Website | https://github.com/jonwright/bslz4_to_sparse |
|---|
