Speaker
Description
We have implemented a system for fully real-time processing of data from serial X-ray protein crystallography experiments. The system handles the full data rate from a 16 megapixel Dectris EIGER2 X detector at 133 frames per second, performing the standard serial crystallography data processing pipeline consisting of peak search, diffraction pattern indexing, spot “prediction” and integration. The processing time for each frame, on a single CPU, is much less than one second. Therefore, a small number of CPU cores (around 40) are sufficient to keep up with the data. This was achieved while handling the entire resolution of the detector: even higher speed is possible by binning the pixel values to reduce the effective number of pixels.
The system has been deployed at the P11 beamline of PETRA III, where it has been reliably used in a series of user experiments. It is based on the CrystFEL software for serial crystallography [1], in combination with the ASAP::O high-performance data framework developed at DESY [2]. Similar systems based on the same building blocks are now being tested at other experimental stations.
Real-time data processing offers many advantages. First, there is the obvious improvement in “situational awareness” during the experiment: the ability to spot problems, make improvements and know when enough data has been collected. In addition, since there is no technical need to store the detector readout data on disk, there is potential for drastic reductions in the high data storage costs which are currently associated with serial crystallography experiments. But are we ready to make the required changes to our established workflows?
This contribution will describe our real-time pipeline in detail, and discuss the implications for the way we perform experiments at large-scale facilities.
[1] https://desy.de/~twhite/crystfel/
[2] https://asapo.pages.desy.de/asapo/
| I plan to submit also conference proceedings | No |
|---|
