European XFEL Data-processing workshop

Europe/Berlin
3.11 (European XFEL, AER19)

3.11

European XFEL, AER19

Albert-Einstein-Ring 19 22761 Hamburg
  • Wednesday 3 February
    • 09:00 09:15
      Welcome
      slides
    • 09:15 10:30
      Common Data Formats I

      pecial focus in this topic should be on the images produced by the fast 2D detectors, alongside auxiliary data needed from e.g. beam monitoring or instrument monitoring and how this is organized and configured. For the 2D detectors, data is by default decided as set of calibrated images, i.e in units of photons per pixel per pulse. Data will be ordered in C-ordering. Zero-suppression according to noise-thresholds is foreseen. Additionally, data may already be converted to event lists by the calibration and correction routines, depending on the observation type: i.e it is feasible to do this if speckles are foreseen to be observed but less feasible for ring-type diffraction patterns.

      • 09:15
        An introduction into foreseen data flows at European XFEL 20m
        Speaker: Dr Steffen Hauf (European XFEL Gmbh)
        Slides
      • 09:35
        Data products from the fast 2D detectors: raw and calibrated data 20m
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
      • 09:55
        Hierarchical data organization 20m
        Speaker: Dr Djelloul Boukhelef (European XFEL GmbH)
        Slides
    • 10:30 10:45
      Coffee Break 15m
    • 10:45 13:00
      Common Data Formats II

      The XFEL data formats use HDF5 for storage. Archival and streaming formats are foreseen, both of which will be able to handle data expressible as Karabo-hashes - this includes images and matrices of arbitrary rank. From within Karabo an I/O interface will be provided. Is there a need to foresee additional API interfaces for legacy software? How can it be guaranteed that data acquired through such interfaces follows XFEL data format standards, e.g. through documentation or through the API?

      • 10:45
        The PC-Layer: where data hits the disks 20m
        Speaker: Dr Djelloul Boukhelef (European XFEL GmbH)
        Slides
      • 11:05
        The HDF5 format to be used at European XFEL 20m
        Speaker: Dr Djelloul Boukhelef (European XFEL GmbH)
        Slides
      • 11:25
        Accessing data through queries to the Metadata catalogue 20m
        Speaker: Mr Luis Maia (European XFEL)
        Slides
      • 11:45
        Retrieving queried data as a user 20m
        Speakers: Dr Djelloul Boukhelef (European XFEL GmbH), Mr Luis Maia (European XFEL)
        Slides
      • 12:05
        Discussions 55m
    • 13:00 14:00
      Lunch Break 1h
    • 14:00 14:45
      Concurrency

      Concurrency: Karabo will handle message/data passing between individual device servers. Scalable topologies, and scatter/gather techniques are supported. What additional concurrency frameworks should be foreseen as being usable on a device level, e.g. ipyparallel, ØMQ? How could existing software best be integrated, using e.g. the aforementioned additional frameworks, into a concurrent Karabo environment? The goal of this topic would be to provide a list of concurrency scenarios which may occur, and ideally input to preparing a best-practices document on how to implement these.

      • 14:00
        Concurrent processing in Karabo 20m
        Speaker: John Wiggins (XFEL - CAS)
        Slides
      • 14:20
        The XFEL calibration pipeline: An example of Karabo-p2p distributed computing mixed with thread-level parallelism in Python and GPU computing 15m
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
        Slides
      • 14:35
        Discussions 10m
    • 14:45 15:00
      Coffee Break 15m
    • 15:00 17:30
      Existing processing solutions
      • 15:00
        X-ray detectors and diagnostics for HED instrument 15m
        Speaker: Dr Motoaki Nakatsutsumi (European XFEL)
        Slides
      • 15:15
        Data processing requirements of the SQS instrument 15m
        Speaker: Dr Thomas Baumann (European XFEL GmbH)
        Slides
      • 15:30
        Common approaches to data reduction in time-resolved X-ray diffuse scattering on liquids 15m
        Speaker: Dr Dmitry Khakhulin (European XFEL GmbH)
        Slides
      • 15:45
        XPCS data analysis and requirements on the data treatment 15m
        Speaker: Dr Jörg Hallmann (European XFEL GmbH)
        Slides
      • 16:00
        Data analysis workflow for synchrotron-CDI experiments mimicking FEL Single Particle Imaging 15m
        Speaker: Klaus Giewekemeyer (European XFEL)
      • 16:15
        Developments in the Single Particle Imaging Analysis Pipeline at LCLS 15m
        Speaker: Dr Chun Hong Yoon (SLAC/LCLS)
        Slides
      • 17:00
        Discussions 30m
  • Thursday 4 February
    • 09:00 09:30
      Open issues and questions from the previous day
      • 09:00
        Serial diffraction experiments 15m
        Speaker: Dr Anton Barty (CFEL/DESY)
      • 09:15
        What can we learn from the LCLS running experience for XFEL.EU 15m
        Speaker: Dr Marc Messerschmidt (European XFEL GmbH)
        Slides
    • 09:30 10:30
      Throughput I
      • 09:30
        Available hardware performance 10m
        Speaker: Dr Djelloul Boukhelef (European XFEL GmbH)
        Slides
      • 09:40
        Configuration of the online calibration pipelines - throughput limiting factors 15m
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
        Slides
      • 09:55
        Offline calibration processing 10m
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
        Slides
      • 10:05
        Discussions 25m
    • 10:30 10:45
      Coffee Break 15m
    • 10:45 11:30
      Throughput II

      Discussion on identifying common, cross-instrument requirements and optimizations thereof

    • 11:30 12:30
      Detector error modes
      • 11:30
        Detector error modes - a general picture 20m
        Speaker: Dr Steffen Hauf (European XFEL GmbH)
        Slides
      • 11:50
        Discussions 40m
    • 12:30 13:00
      The Veto System 30m
      Speaker: Dr Alexander Kaukher (European XFEL)
      Slides
    • 13:00 14:00
      Lunch Break 1h
    • 14:00 15:00
      Conclusion