Making SciCat scalable - first performance studies

Monday 15 Apr 2024, 10:00 → 11:00 Europe/Berlin

DESY, building 1, 2nd floor, SR 03a

Regina Hinzmann (IT (IT Information Fabrics))

https://desy.zoom.us/j/6389377212?pwd=a3pJUGZXb1UrNm5rVVJtN3phNC94dz09

 

10:00 → 10:20 Making SciCat scalable - first performance studies

First works were performed towards a larger scale deployment of SciCat at DESY beamlines. One initial study by Igor (FS-SC) is shown here as part of a bigger work to be done - once resources are clearer.

Speaker: Igor Khokhriakov (FS-SC (Scientific computing))

Presentation

minutes of SCT meeting on 2024-04-15

10:01-10:47

 

Intro: We started with conclusion in the email-discussion on the proposed way on how to get a solution on different IDs: no changes on the document.

 

Main talk: Igor presented his studies departing from the overall goal regarding SciCat at DESY with a nice introduction to this topic: He started with non-functional requirements that represent general actions: I/O and CPU which can stand for ingesting data into MongoDB or calculating hashing/routing, respectively. He studied requests-per-seconds (RPS) in 2 scenarios: single scicat instance and 8 instances + load balancer (nginx) in front. From his plots one could conclude

Avoid single instances: especially CPU performance drops already when one has 10 ms operations. In case of 8 nodes, 10 times improved behaviour can be expected. At DESY, we have as standard 3 nodes, which is already good and somewhere in between his scenarios.

Next step would be to set up performance monitoring tools on our demonstrator beamlines to get numbers of the mixture of I/O and CPU.

 

To-DOs:

Longer scale: find which monitoring tools we should use (Grafana, APM, etc).

Short scale: Get licence for FS-SC to set up CI/CD for pipeline with GitLab that allows mirroring from GitHub repos → allows quicker handle for development work.

Medium to short time scale: wait for next pyscicat release 4.5 due to incompatibilities of python versions available at DESY beamline environment.

Now: send (done at the time of writing) and discuss options for DESY case with SciCat community.

 

Participants:

1. Linus P
2. Igor K
3. Jan K
4. Martin G
5. Anjali A
6. Olga M
7. Dmytro K
8. Neele R
9. RH

10:20 → 10:40 The DESY case or how to disentangle our IDs

Paul M, Frank S and Regina H drafted a document to disentangle proposal and beamtime IDs. This document was sent out beforehand to DESY users for which we waited til after the meeting for users feedback. The attached document was sent after the deadline to the SciCat Project leader Max Novelli.

Speakers: Alexander Paul Millar (IT (Research and Innovation in Scientific Co)), Frank Schluenzen (IT (IT Benutzerservice)), Regina Hinzmann (IT (IT Information Fabrics))

DESYcase.pdf