18–20 Sept 2024
Berlin Adlershof John v. Neumann Gebaeude
Europe/Berlin timezone

ASAPO: A high-speed streaming framework to support an automated data-processing pipeline.

Not scheduled
20m
RUD25 3.001 (Berlin Adlershof John v. Neumann Gebaeude)

RUD25 3.001

Berlin Adlershof John v. Neumann Gebaeude

Speaker

Mikhail Karnevskiy (IT (IT Scientific Computing))

Description

Modern scientific experiments often generate large amounts of data, posing challenges for real-time processing and analysis. ASAPO, a high-performance streaming framework developed at DESY, addresses these challenges by providing a robust solution for online and offline data processing. Leveraging TCP/IP and RDMA over Ethernet and Infiniband, ASAPO facilitates high-bandwidth communication between detectors, storage systems, and analysis processes.

ASAPO offers user-friendly interfaces for C/C++ and Python on all major platforms, streamlining the development of data processing pipelines. A high-level Python library reduces boilerplate code and enables the creation of complex analysis workflows with ease. Key features include automatic re-transfer, trivial parallelization on a per-image basis, support for multi-module detectors, and web-based monitoring capabilities.

Several experimental facilities at Petra III already benefit from ASAPO, employing it in various data-processing pipelines. Examples include azimuthal integration of X-ray scattering data, peak finding, and indexing of diffraction patterns. These applications demonstrate ASAPO's versatility and effectiveness in accelerating scientific discovery.

Speed talk: Normal speed talk selection

Primary author

Mikhail Karnevskiy (IT (IT Scientific Computing))

Co-authors

Martin Gasthuber (IT (IT Scientific Computing)) Tim Schoof (DESY)

Presentation materials

There are no materials yet.