Lossy and lossless compression for serial crystallography at synchrotrons and FELs

• Marina Galchenkova (FS-CFEL-1 (Forschung mit Photonen Experimente 1))

Protein crystallography is one of the most successful methods for biological structure determination. This technique requires many diffraction snapshots to get 3D structural information of the studied protein. Even more patterns are needed for studying fast protein dynamics that can be achieved using serial crystallography (SX). Fortunately, new X-ray facilities such as 4th generation synchrotrons and Free Electron Lasers (FELs) combined with newly developed X-ray detectors opened a way to carry out these experiments at a rate of more than 1000 images per second. The drawback of this increase in acquisition rate is the volume of collected data - up to 2 PB of data per experiment could be easily obtained. Therefore, new data reduction strategies have to be developed and deployed. Lossless data reduction methods will not change the data, but usually fail to achieve a high compression ratio. On the other hand, lossy compression methods can significantly reduce the amount of data, but they require careful evaluation of the resulting data quality. We have tested different approaches for both lossless and lossy compression applied to SX data, proposed some new ways for lossy compression and demonstrated appropriate methods for data quality assessment. By checking the resulting statistics of compressed data (like CC*/Rsplit, Rfree/Rwork) we have demonstrated that the volume of the measured data can be greatly reduced (10-100 times!) while the quality of the resulting data was kept almost constant. In addition, we tested lossy compression methods on the SAD dataset (thaumatin collected at 4.57 keV, measured at the SwissFEL) and demonstrated that even such very sensitive data can be successfully compressed. It allowed us to determine the limit of application for all considered lossy compressions. Some of the proposed compression strategies, tested on SX and MX datasets, can be used for other types of experiments, even with different sources (for example electron and neutron diffraction).

Real-time data reduction for the ESRF serial crystallography beamline ID29

• Jerome Kieffer (ESRF)

The serial crystallography beamline ID29 at the European Synchrotron (Grenoble, France) is operating a Jungfrau 4M detector at 1 kHz. The image flow it produces outperforms the storage speed (of ESRF) by one order of magnitude. This presentation summarizes the advancement in hardware-accelerated compression (lossless) and real-time data reduction (lossy) to be able to record data. The presentation will focus on GPU- accelerated algorithms for separation of Bragg-peaks from background, allowing to save mostly pixels contributing to peaks. The peak localization being obtained for free, the offline processing is also accelerated with the analysis starting at the indexation stage.