The CDB evolution
part 2

Marco Emilio Poleggi

Marco.Poleggi@cern.ch

Outline

CDB overview

What's new

Boiling in the pot

Open issues

CDB overview

Three-tier architecture

SOAP client

cdbop interactive/batch shell

scripts

SOAP middle-ware

Apache + cdb-soap CGI

stateless: each connection conveys one command

CDB back-end

it's a library, not a server

templates compiled via cake

templates stored in CVS

stateful: partially transactional semantic through “sessions”

Outline

CDB overview

What's new

[tasks completed since last workshop at LAL]

State management through session metadata

Parallel and distributed compilation of profiles

Smarter rollback and commit

Handling of “dead” revisions

Boiling in the pot

Open issues

[tasks completed since last workshop at LAL]

Namespaces

X.509 authentication. And Kerberos5 too!

Access control lists (ACL), with namespace support

Client and SOAP middleware reworked to use one network connection to process list of items

State management through session metadata

Problem: template files in session directories are used for both data and state information

moving around data causes a loss of state information

Solution: using real state metadata

separate metadata files outside session directories

better control of concurrent modification conflicts:

templates really checked out only on “update” operations;

early conflict detection on “update”: avoids a lot of failed commits :-)

Status: under testing

Parallel compilation of profiles...

Problem: compilation of O(10K) profiles doesn't scale on a single CPU

takes ages to terminate

eats GBytes of memory

Solution: split the profile set in batches and process each batch on a different CPU

no dependencies among batches

common template base

locally on a multiprocessor board or multicore CPU

remotely on several machines. See #Distributed compilation of profiles

Note. It's not a concurrent compilation process: only one session holds the lock!

Status: local compilation on multicore processors under testing

...Distributed compilation of profiles

Problem: parallel compilation of batches over different machines

intrinsically fault-tolerant :-)

Solution: keep it as simple as possible!

orchestration of processes via SSH

data exchange: know nothing about it! Just place them wherever you prefer and share the location over a network file system

Status: to be tested

Smarter rollback and commit

Problem

Rolling back a whole session with tens of modified templates because only one is screwed up is frustrating!

Committing without knowing what's going on after the last change made the day before may fail spectacularly or produce unintended results. That's anyway a waste of CPU cycles!

Solution

selective rollback: just discard what I want, not everything

interactive commit: show what's to be done and ask for confirmation

Status: under testing

Handling of "dead" revisions

Problem: removed templates are buried forever in CVS :-(

“Ouch! I just misspelled the name, now what?”

administrator calls needed for resurrecting them

“retired” templates lay derelict in memory of the removed ones (R.I.P.)

Solution: the zombies are there, just wake them back up ;-)

Easy, but... please, don't use cvs status -v looong-list-of-files: it will run out of memory. Right, just find and scan RCS files... So, what's CVS for? Mmh, we should have a look at Subversion

Status: under testing

Outline

CDB overview

What's new

Boiling in the pot

[left behind since the last workshop @ LAL]

A common authentication service

Open issues

[left behind since the last workshop @ LAL]

Make CDB a Web service

Still interest in this?

Fine-grained CDB locking with fair queuing

The latter feature is really needed

Concurrent (was “parallel”) compilation of templates

I believe this should be dropped because too complex

mod_perl

no further investigation. Maybe once we have more recent Apache and mod_perl distributed with Scientific Linux. Fast CGI might be a better alternative

A common authentication service

Problem: different services use the same routines for authentication purpose

code duplication!

direct calls to internal cdb-soap's routines

Solution: provide an external authentication library

Outline

CDB overview

What's new

Boiling in the pot

Open issues

Open Issues...

CVS doesn't scale well!

For instance, with 24K templates:

cvs status -v checkout/ ~ O(10', out-of-memory)

for `cvs status -v checkout/<RCS file>` { grep stuff } ~ O(eternity, 10MB)

for `find <RCS files>` { grep stuff } ~ O(1'', 1MB)

Possible solutions

Perl-based CVS interface? Subversion? XML database?

Not relocatable (actually most of Quattor is affected)

limited portability

testing need a full installation or specific privileges

...Open Issues

Am I forgetting something?

Any RFE, wish, remark?

Open bugs in Savannah

managing multiple testbed with single CDB machine (bug #3550 )

CVS-like handling of cdbop (bug #9734)

pkg_repl does not work with "DEF","DEF","CERN_CC" arguments (bug #7335)

impossible to separate site-specific from general component information (bug #9735)

aii to use sindes to fetch certificates (bug #10437)

allow for 'vertical' data/template typing in pan (bug #10621)

extend CCM to host non-local profiles (bug #10628)

remove 'global' lock from CDB (bug #10731)

SWRep should check RPM signatures (bug #11061)

Extend AII support for other device naming schemas

Enable new authentication mechanisms: KRB, X509 certificates (bug #17676)

Generic 'provides' for NCM dependencies (bug #17681)

Create a complete partitioning code for the AII/KickStart configuration template (bug #17689)

CDBOP should allow using CVS tags (bug #17827)

add 'NOACTION_SUPPORTED' to NCM component (bug #20040)

2005-04-06T11:14:03