Online Data Analysis at the European XFEL
Thomas Michelat

The Karabo-bridge Python package provides a client interface to the Karabo bridge

Install the Karabo Bridge Python client:

$ pip install karabo-bridge
In [1]:
# import client
from karabo_bridge import Client

At object instantiation, the client connects to the karabo bridge server.

In [2]:
# Connect to a server: tcp://localhost:4343
bridge_client = Client('tcp://localhost:4343')

request the next data available on this server.

In [3]:
# Request data
data, metadata = bridge_client.next()

It returns 2 items of type dict:

  • data: contains the data associated with a XFEL train
  • metadata: source names, timestamp and trainId.
In [4]:
# Type of data container is dict
type(data)
Out[4]:
dict
In [5]:
# One entry per data source
data.keys()
Out[5]:
dict_keys(['SPB_DET_AGIPD1M-1/DET/detector'])

Each data source has its properties stored in a dict

In [6]:
# See properties for a data source
source = 'SPB_DET_AGIPD1M-1/DET/detector'
data[source].keys()
Out[6]:
dict_keys(['image.cellId', 'images.pulseId', 'sources', 'modulesPresent', 'image.passport', 'image.gain', 'image.data'])

All sources are associated with medata, containing: source name, train ID and UNIX epoch.

In [7]:
# show metadata for the same data source
metadata[source]
Out[7]:
{'source': 'SPB_DET_AGIPD1M-1/DET/detector',
 'timestamp.tid': 198425241,
 'timestamp': 1547733553.060967,
 'timestamp.sec': '1547733553',
 'timestamp.frac': '060967000000000000'}

All data types are:

  • build-in python or
  • numpy array for multidimensional arrays.

Example: getting detector image (numpy array) and display array informations.

In [8]:
# Getting ndarray
images = data[source]['image.data']
print(images.shape)
print(images.dtype)
print(images[0:2, 0:2, 0, 0])

(176, 16, 512, 128)
float32
[[ 28.592829  35.21756 ]
 [291.0153   265.3471  ]]

While data is flowing through the karabo pipeline, you can request data.

In [9]:
# Continuously requesting data
for i in range(3):
    data, meta = bridge_client.next()
    print(meta[source]['timestamp.tid'])

198425242
198425243
198425244

You can instantiate as many clients as you need (data will be distributed over the different clients).

In [10]:
# 2nd client
client_2 = Client('tcp://localhost:4343')
data, meta = client_2.next()
print(meta['SPB_DET_AGIPD1M-1/DET/detector']['timestamp.tid'])

198425245
In [11]:
# 3rd client
with Client('tcp://localhost:4343') as client_3:
    for data, meta in client_3:
        print(meta['SPB_DET_AGIPD1M-1/DET/detector']['timestamp.tid'])
        break

198425246

What can you do with data?

In [17]:
import matplotlib.pyplot as plt
import h5py
import numpy as np
import warnings
warnings.filterwarnings('ignore')

from kdata.karabo_data.geometry2 import AGIPD_1MGeometry

%matplotlib inline

# Detector data from one train
detector_data = data['SPB_DET_AGIPD1M-1/DET/detector']['image.data']

# Define geometry correction
geom = AGIPD_1MGeometry.from_quad_positions(quad_pos=[
        (-525, 625),
        (-550, -10),
        (520, -160),
        (542.5, 475),
    ])

# Apply correction to detector data
mean_image = np.nanmean(detector_data, axis=0)
res, centre = geom.position_modules_fast(mean_image)

# Plot
plt.figure(figsize=(12, 12))
plt.imshow(np.clip(res, 0, 800), cmap='Greys')
Out[17]:
<matplotlib.image.AxesImage at 0x2b6460288518>

The package provides also a few console tools:

  • Simulate a karabo bridge server with simulated data

    $ karabo-bridge-server-sim

  • Simulate a karabo bridge server from data reccorded at the EuXFEL

    $ karabo-bridge-serve-files

  • get information on the data sent by the Karabo Bridge

    $ karabo-bridge-glimpse

  • monitor the data sent by the Karabo Bridge

    $ karabo-bridge-monitor

Karabo Bridge C++ client:

Installation

$ git clone https://github.com/European-XFEL/karabo-bridge-cpp.git
$ cd karabo-bridge-cpp
$ ./autogen.sh install /YOUR/TARGET/FOLDER

Usage

#include "kb_client.hpp"

karabo_bridge::Client client;
client.connect("tcp://localhost:1234")

auto payload = client.next()
for (auto it = payload.begin(); it != payload.end(); ++it) {
    if (it->first == "SPB_DET_AGIPD1M-1/DET/detector") {
        karabo_bridge::kb_data data(it->second);
        auto images = data.array["image.data"].as<std::vector<uint32_t>>();
    }
}

Source, documentation, examples: https://github.com/European-XFEL/karabo-bridge-cpp/