European XFEL Data-processing workshop

Europe/Berlin
3.11 (European XFEL, AER19)

3.11

European XFEL, AER19

Albert-Einstein-Ring 19 22761 Hamburg
  • Wednesday 3 February
    • Welcome
      slides
    • Common Data Formats I

      pecial focus in this topic should be on the images produced by the fast 2D detectors, alongside auxiliary data needed from e.g. beam monitoring or instrument monitoring and how this is organized and configured. For the 2D detectors, data is by default decided as set of calibrated images, i.e in units of photons per pixel per pulse. Data will be ordered in C-ordering. Zero-suppression according to noise-thresholds is foreseen. Additionally, data may already be converted to event lists by the calibration and correction routines, depending on the observation type: i.e it is feasible to do this if speckles are foreseen to be observed but less feasible for ring-type diffraction patterns.

      • 1
        An introduction into foreseen data flows at European XFEL
        Speaker: Dr Steffen Hauf (European XFEL Gmbh)
        Slides
      • 2
        Data products from the fast 2D detectors: raw and calibrated data
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
      • 3
        Hierarchical data organization
        Speaker: Dr Djelloul Boukhelef (European XFEL GmbH)
        Slides
    • 10:30
      Coffee Break
    • Common Data Formats II

      The XFEL data formats use HDF5 for storage. Archival and streaming formats are foreseen, both of which will be able to handle data expressible as Karabo-hashes - this includes images and matrices of arbitrary rank. From within Karabo an I/O interface will be provided. Is there a need to foresee additional API interfaces for legacy software? How can it be guaranteed that data acquired through such interfaces follows XFEL data format standards, e.g. through documentation or through the API?

      • 4
        The PC-Layer: where data hits the disks
        Speaker: Dr Djelloul Boukhelef (European XFEL GmbH)
        Slides
      • 5
        The HDF5 format to be used at European XFEL
        Speaker: Dr Djelloul Boukhelef (European XFEL GmbH)
        Slides
      • 6
        Accessing data through queries to the Metadata catalogue
        Speaker: Mr Luis Maia (European XFEL)
        Slides
      • 7
        Retrieving queried data as a user
        Speakers: Dr Djelloul Boukhelef (European XFEL GmbH), Mr Luis Maia (European XFEL)
        Slides
      • 8
        Discussions
    • 13:00
      Lunch Break
    • Concurrency

      Concurrency: Karabo will handle message/data passing between individual device servers. Scalable topologies, and scatter/gather techniques are supported. What additional concurrency frameworks should be foreseen as being usable on a device level, e.g. ipyparallel, ØMQ? How could existing software best be integrated, using e.g. the aforementioned additional frameworks, into a concurrent Karabo environment? The goal of this topic would be to provide a list of concurrency scenarios which may occur, and ideally input to preparing a best-practices document on how to implement these.

      • 9
        Concurrent processing in Karabo
        Speaker: John Wiggins (XFEL - CAS)
        Slides
      • 10
        The XFEL calibration pipeline: An example of Karabo-p2p distributed computing mixed with thread-level parallelism in Python and GPU computing
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
        Slides
      • 11
        Discussions
    • 14:45
      Coffee Break
    • Existing processing solutions
      • 12
        X-ray detectors and diagnostics for HED instrument
        Speaker: Dr Motoaki Nakatsutsumi (European XFEL)
        Slides
      • 13
        Data processing requirements of the SQS instrument
        Speaker: Dr Thomas Baumann (European XFEL GmbH)
        Slides
      • 14
        Common approaches to data reduction in time-resolved X-ray diffuse scattering on liquids
        Speaker: Dr Dmitry Khakhulin (European XFEL GmbH)
        Slides
      • 15
        XPCS data analysis and requirements on the data treatment
        Speaker: Dr Jörg Hallmann (European XFEL GmbH)
        Slides
      • 16
        Data analysis workflow for synchrotron-CDI experiments mimicking FEL Single Particle Imaging
        Speaker: Klaus Giewekemeyer (European XFEL)
      • 17
        Developments in the Single Particle Imaging Analysis Pipeline at LCLS
        Speaker: Dr Chun Hong Yoon (SLAC/LCLS)
        Slides
      • 18
        Discussions
  • Thursday 4 February
    • Open issues and questions from the previous day
      • 19
        Serial diffraction experiments
        Speaker: Dr Anton Barty (CFEL/DESY)
      • 20
        What can we learn from the LCLS running experience for XFEL.EU
        Speaker: Dr Marc Messerschmidt (European XFEL GmbH)
        Slides
    • Throughput I
      • 21
        Available hardware performance
        Speaker: Dr Djelloul Boukhelef (European XFEL GmbH)
        Slides
      • 22
        Configuration of the online calibration pipelines - throughput limiting factors
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
        Slides
      • 23
        Offline calibration processing
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
        Slides
      • 24
        Discussions
    • 10:30
      Coffee Break
    • Throughput II

      Discussion on identifying common, cross-instrument requirements and optimizations thereof

    • Detector error modes
      • 25
        Detector error modes - a general picture
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
        Slides
      • 26
        Discussions
    • 27
      The Veto System
      Speaker: Dr Alexander Kaukher (European XFEL)
      Slides
    • 13:00
      Lunch Break
    • Conclusion