Deep Learning in Inverse Problems

23 Sept 2024, 09:00 → 24 Sept 2024, 17:00 Europe/Berlin

Flash Seminar Room (DESY)

This workshop will focus on novel developments on learning based methods for inverse problems, including supervised learning and unsupervised learning such as learned regularizations and generative models for Bayesian inversion. 
Particular emphasis will be put on the understanding of foundations of learning in this area as well as novel applications including problems in imaging and speech/audio processing.

→ The timetable can be found here.

Confirmed speakers

• Tatiana Bubba (University of Bath)
• Antoine Deleforge (INRIA Nancy)
  
• Berthy Feng (California Institute of Technology)
  
• Eloi Moliner (Aalto University)
  
• Ozan Öktem (KTH Stockholm)

 

We gratefully acknowledge support by the DFG funded priority programme Theoretical Foundations of Deep Learning and Helmholtz Imaging.

book_of_abstracts_dlip.pdf

Monday 23 September

09:15 Registration

10:15 Welcome

1 Scientific machine learning: An overview with applications to inverse problems

Scientific Machine Learning is an emerging research area within statistical learning that focuses on the opportunities and challenges of machine learning in the context of complex applications across science, engineering, and medicine. Challenges in these fields have attributes that make them very different in nature to computer science applications where data-driven machine learning has found success. The talk will try to provide an overview of this area from the viewpoint of solving large-scale ill-posed inverse problems.

Speaker: Ozan Öktem (KTH Stockholm)

2 Stationary Velocity Fields on Matrix Groups for Deformable Image Registration

We present an extension of the Stationary Velocity Field (SVF) approach to matrix groups, with a particular focus on the Special Euclidean group SE(3). The SVF method is a popular tool for parameterizing invertible deformation fields and is particularly effective when integrated into machine learning-based networks. However, it can struggle with modeling large deformations. By extending the SVF approach to matrix groups like SE(3), we move Euclidean transformations into the low-frequency domain, aligning them with the natural bias of many neural network architectures towards low-frequency components. This extension can result in improved handling of large motions, allowing for more robust and efficient recovery in tasks that involve significant deformations. This method addresses challenges in deformation modeling, making it particularly relevant for deep learning-based applications.

Speaker: Johannes Bostelmann (Universität zu Lübeck)

3 Robust Hybrid 3D Image Registration Based On Learned Segments

Variational image registration methods are a powerful tool in medical imaging. A drawback of these methods is the sensitivity to initialization, which often relies on user input as manual landmark detection. To overcome this difficulty we propose a fully-automated hybrid registration approach, which builds on the great success of artificial intelligence in segmentation of anatomical structures: Given segmentations, we derive features and perform a landmark-based registration followed by an intensity-guided registration. A new coupling regularization is used in both phases of the registration in order to ensure a seamless transformation. Experimental results in 3D show that our registration approach can be easily applied even to challenging medical data, such as lung CT imaging.

Speaker: Pia Franziska Schulz (Universität zu Lübeck)

12:30 Lunch

4 Proximal operators learning meets unrolling for limited angle tomography

In recent years, model-based strategies for solving ill-posed inverse problems, such as classical regularisation theory, have been successfully integrated with data-driven approaches, providing satisfying numerical results and insights into major theoretical and practical questions.
In this talk, I will present some recent results combining unrolling of a proximal gradient descent algorithm and the Gadient-Step denoiser formulation of a Plug-and-Play scheme, which allows to learn the proximal operator of the unfolded scheme. Particular effort is put into the efficient formulation of the algorithm, by introducing an extrapolation strategy in the unrolled scheme which allows to reduce the resources necessary to compute the reconstruction while preserving the theoretical guarantees. The advantages of our approach are demonstrated in the context of limited data tomography, a challenging inverse problem where only partial data are available for the reconstruction.

Speaker: Tatiana Bubba (University of Bath)

5 Early Stopping of Untrained Convolutional Neural Networks

In recent years, new regularization methods based on (deep) neural networks have shown very promising empirical performance for the numerical solution of ill-posed problems, such as in medical imaging and imaging science. Due to the nonlinearity of neural networks, these methods often lack satisfactory theoretical justification. In this work, we rigorously discuss the convergence of a successful unsupervised approach that utilizes untrained convolutional neural networks to represent solutions to linear ill-posed problems. Untrained neural networks have particular appeal for many applications because they do not require paired training data. The regularization property of the approach relies solely on the architecture of the neural network instead. Due to the vast over-parameterization of the employed neural network, suitable early stopping is essential for the success of the method. We establish that the classical discrepancy principle is an adequate method for early stopping of two-layer untrained convolutional neural networks learned by gradient descent, and furthermore, it yields an approximation with minimax optimal convergence rates.

Speaker: Tim Jahn (TU Berlin)

6 Accelerating Full Waveform Inversion By Transfer Learning

Full waveform inversion (FWI) is a powerful tool to reconstruct the material distribution with damage such as a void or a crack based on sparsely measured wave propagation data. FWI can be utilized in health monitoring of infrastructure to determine the location and severity of the damage. However, the reconstruction of damage from sensor data is an ill-posed and computationally expensive problem. A computationally cheaper alternative is to use the framework of Neural Networks (NN) trained by supervised learning. The training of the NN is performed on sparsely measured wave propagation data measured in domains with known defects. However, such a purely supervised approach can lead to wrong predictions for signals outside the training data set. Alternatively, NNs may be combined with classical FWI. Therein, NNs are used as the discretization of the material field, utilizing the adjoint formulation inherent to FWI. This combination of NN with FWI yields reliable predictions, whose results are improved wr.t. classical FWI in terms of quality. To further improve NN-based FWI, we propose to combine Transfer Learning and FWI . The proposed technique utilizes supervised pretraining to provide a better NN weight initialization leading to faster convergence of FWI. The network is pretrained to predict the unknown material field using the gradient information from the first iteration of the conventional FWI. We demonstrate that the use of gradient information in transfer-learning based FWI provides reliable predictions also for data outside the training set and reduces the number of iterations required for convergence [3]. Even in case where the initial guess is not accurate, transfer learning FWI can still recover the damaged shape although requiring more epochs.

Speaker: Divya Shyam Singh (TU Munich)

15:30 Coffee

7 KCT CBCT: Fast, Accurate, and Reliable Software for Algebraic CT Reconstruction

KCT CBCT is a high-performance software suite designed to tackle the inverse problem of tomographic reconstruction, specifically for CT and CBCT applications. This package offers a range of modern reconstruction algorithms, including Krylov-based LSQR and CGLS methods, as well as the widely used OS-SIRT technique. A key innovation of KCT CBCT is its implementation of the Cutting Voxel Projector (CVP), an algorithm developed by the author that directly computes voxel projections using volume integrals, significantly enhancing projector precision. The package also supports other projectors, such as the TT and Siddon projectors, with future updates planned for DD and TR projectors. In addition to these core features, KCT CBCT includes basic L2 regularization and preconditioning schemes for Krylov methods. Looking ahead, there are plans to implement advanced nonlinear regularization techniques, such as Total Variation (TV) regularization. The programs are written in C++ and OpenCL. Optimized for GPU acceleration, KCT CBCT is compatible with both AMD and NVIDIA architectures, utilizing OpenCL 1.2 for broad hardware support. Notably, the software fully supports parallel ray geometry, making it particularly suited for synchrotron applications.

Speaker: Vojtech Kulvait (Hereon (Helmholtz-Zentrum Hereon))

8 Stability analysis for Inverse Random Source Problems

The problem under investigation is to determine the strength of a random acoustic source from correlations of measurements distant from the source region. Specifically, we acquire measurements of the time-harmonic acoustic waves on a surface surrounding the source region and then average their correlation to approximate the covariance operator of the solution process on the measurement surface. A natural extension of the existing uniqueness results [1,4] are stability estimates. We were able to show in general settings that this problem is severely ill-posed. Particularly, we show two bounds the upper bound usually referred to as stability estimate and the lower bound called instability estimate. The stability estimate is shown by verifying a variational source condition [3] for the problem which in turn also provides convergence rates for a variety of spectral regularisation methods. Instability is based on a general entropy argument presented for operators of the type X -> L(H,H') with X some metric space and H some separable Hilbert space [2]. The talk will be finished with some numerical experiments that support our theoretical results.

[1] A.J. Devaney. The inverse problem for random sources. In: Journal of Mathematical Physics 20.8 (1979), pp. 1687-1691.

[2] M. Di Cristo and L. Rondi. Examples of exponential instability for inverse inclusion and scattering problems. In: Inverse Problems, 19 (2003), p. 685.

[3] T. Hohage and F. Weidling. Characterizations of variational source conditions, converse results, and maxisets of spectral regularization methods. In: SIAM Journal on Numerical Analysis, 55.2 (2017), pp. 698-630.

[4] T. Hohage, H.-G. Raumer, and C. Spehr. Uniqueness of an inverse source problem in experimental aeroacoustics. In: Inverse Problems 36(7) (2020)

Speaker: Philipp Mickan (University of Göttingen)

Poster Session: Poster Pitches

9 Neural Network Method for Dielectric Optical Coating Design

Dielectric optical coatings are crucial for ultrafast laser systems. Current methods for designing these coatings rely on brute force optimization, which is computationally intensive and demands expert knowledge. Our project introduces an AI-driven approach using deep neural networks to solve this complex inverse problem, offering faster, expert-independent solutions.

Speaker: Utsa Chattopadhyay (DESY/TUHH)

10 Speeding-up the extraction of morphological parameters in GISAXS data using a physics-informed deep learning approach

In-situ grazing-incidence small-angle X-ray scattering (GISAXS) is a powerful technique for investigating nanoscale structures with high time resolution, yet data analysis is complex and time-intensive. To accelerate this process, we present a novel two-step approach that integrates physics-informed deep learning to extract key morphological parameters. The approach involves preprocessing GISAXS simulations using different techniques, including intensity thresholding, to generate training data for an artificial neural network (NN). Instead of single-value predictions, the NN is designed to forecast distributions of parameters, such as the average cluster radius, which has shown promising early results. While further development is needed, these initial successes indicate the potential of the method for more efficient GISAXS data analysis.

Speaker: Shachar Dan (Deutsches Elektronen-Synchrotron (DESY))

11 Phase retrieval in the wild: In-situ optimized reconstruction for single-distance X-ray near-field holography

With a holographic setup, full-field projections can be obtained with a single exposure. Reconstruction of the complex refractive index of the sample is typically done in post-processing and requires many manual steps and time. The need for post-processing prevents an immediate optical evaluation of the measurement results during beamtime and complicates in-situ experiments. We propose a novel pre-processing and reconstruction scheme that allows reconstructions from single shot holograms without spatial support constraints. To this end, we investigate the source of reconstruction artifacts that appear in reconstructed projection images and propose countermeasures that are embedded in a gradient-descent-based algorithm. Most of the reconstruction parameters are independent of the measured object, and the reconstruction performance is significantly improved. The Artifact Suppressing Reconstruction Method (ASRM) and an online reconstruction framework are now available to users at the imaging beamline P05 at DESY, Hamburg.

Speaker: Johannes Dora (DESY Photon Science)

12 Beam emittance optimization as an inverse problem

The European XFEL at DESY is a world-leading research infrastructure in Hamburg, enabling scientists to observe and investigate microstructural processes with resolutions on the atomic and femtosecond scale. To improve the performance of the accelerator, it is essential to optimize the EuXFEL for operation in continuous-wave (CW) mode. Despite its advantages, an operation in CW mode requires a reduction of the beam energy and is associated with an increase in the geometric beam emittance. Within the OPAL-FEL project, we pursue a data-driven optimization of the beam emittance to ensure the delivery of high quality beams in CW mode. Central to our approach is the use of deep learning techniques to implement an inverse model, predicting the optimal state of the photoinjector for achieving a desired optimal emittance. We detail our methodology and present initial results from a neural network trained on synthetic data generated using the beam dynamics simulation code ASTRA. Additionally, we explore the theoretical aspects of the forward model's invertibility, drawing on Whitney's embedding theorem within the context of attractor reconstruction.

Speaker: Alexander Klemps (Hamburg University of Technology (TUHH))

13 Phase Retrieval in Audio Processing and Ptychography: From Traditional Methods to Deep Learning

Phase retrieval from amplitude-only measurements is a common task that appears in diverse scientific fields. Recognizing the conceptual similarity between ptychographic measurements and the short-time Fourier transform (STFT), often used in audio processing, our interdisciplinary research focuses on phase retrieval methods in these two domains. We present several recent contributions exhibiting how advances in one domain can be effectively applied to the other, leading to improved phase retrieval algorithms based on both traditional projection-based methods and data-driven deep learning approaches.

Speaker: Tal Peer (University of Hamburg)

14 Near-field Holographic Image Denoising Using Dilated Convolutional Neural Networks

Near-field holography imaging plays a crucial role in various scientific and industrial applications, offering detailed insights into nanostructures and surface properties. However, the acquired images often suffer from significant noise, particularly at lower exposure times, which can severely impact their interpretability and analytical accuracy. In this study, we present a novel approach for enhancing the denoising capability of near-field holography images using dilated convolutional deep neural networks (DnCNN). By leveraging the advantages of dilated convolutions, our method aims to preserve crucial spatial details while effectively reducing noise artifacts inherent in near-field holography at the lower exposure times. To ensure the preservation of structural similarity between the denoised and input images, we introduce a custom loss function combining L1-norm and Structural Similarity Index (SSIM). Our approach involves training a deep neural network architecture comprising multiple layers of dilated convolutions to capture both local and global contextual information from the input images. The hierarchical feature extraction facilitated by dilated convolutions enables the network to learn complex patterns associated with noise and anatomical structures, thus enhancing its ability to discriminate between signal and noise components. Experimental results on a diverse dataset of low-dose holography images demonstrate the effectiveness of our proposed method in significantly reducing noise while preserving important image features. This study highlights the potential of dilated convolutional neural networks as a promising tool for denoising near-field holography images, facilitating improved analysis and interpretation in various scientific and industrial domains.

Speaker: Vahid Rahmani (DESY)

15 Sequential Experimental Design for X-ray CT

In X-ray Computed Tomography (CT), obtaining projections from various angles is crucial for 3D reconstruction. To adapt CT for real-time quality control, it's essential to reduce the scan angles while preserving reconstruction quality. Sparse-angle tomography, which achieves 3D reconstructions with fewer data, necessitates selecting the most informative angles—a challenge equivalent to solving a sequential optimal experimental design (OED) problem. However, OED issues are marked by complexity, including high-dimensional, non-convex optimization that makes adaptive solutions during scanning difficult. To navigate these complexities, we approach the sequential OED problem through a Bayesian framework, modeling it as a partially observable Markov decision process and employing deep reinforcement learning for solutions. The approach learns efficient non-greedy policies to solve a given class of OED problems through extensive offline training rather than solving a given OED problem directly via numerical optimization. Consequently, our policy efficiently identifies the most informative angles for real-time operations, significantly enhancing the practicality of sparse-angle CT in quality control scenarios. This streamlined approach ensures that CT can be efficiently integrated into quality control processes, with the potential to significantly impact the field.

Speaker: Tianyuan Wang (CWI Amsterdam)

16 Application of Machine Learning to X-ray Scattering Data Processing

X-ray scattering is one of the main techniques used to characterise the structure of nanomaterials. Extraction of real-space structures from X-ray scattering patterns needs to be carried out through the use of scattering formulae fitting, which has the disadvantages of being time-consuming, requiring specialised knowledge, and initial parameter estimation. In the face of a large amount of experimental data, especially synchrotron radiation experimental data dealing with increasing luminous flux, the existing frontal analysis methods are not able to tone track the challenges of real-time analysis. We Use machine learning methods, relevant structural information can be quickly obtained from scattering experimental data without the introduction of the relevant scattering knowledge.

Speaker: Yufeng Zhai (DESY)

Poster Session

18:00 Barbecue

Tuesday 24 September

17 Hearing the Walls of a Room: When Signal Processing and Machine Learning Meet Acoustics

Close your eyes, clap your hands. Can you hear the shape of the room? Is the floor made of tiles or carpet? Answering such questions using only audio signals recorded by microphones form a set of multifaceted and open inverse problems, located at the narrow intersection between the fields of mathematics, acoustics and computer science. Progress on it could make the acoustic diagnosis of rooms simpler, cheaper and more accurate, or bring improvements to the fields of sound source localization, enhancement, synthesis or acquisition. This presentation will explore some facets of these questions from the angles of signal processing, machine learning and optimization, covering joint works performed in the MULTISPEECH team of Inria and the UMRAE team of Cerema over the past 4 years.

Speaker: Antoine Deleforge (INRIA Nancy)

10:30 Coffee

18 Diffusion Models for Inverse Problems in Audio: Applications in Music Restoration

The talk will discuss the application of diffusion models to a variety of inverse problems in audio restoration, with a focus on music. These models can be integrated within a posterior sampling framework to offer a probabilistic generative approach for addressing ill-posed problems such as audio inpainting, bandwidth extension, and declipping. The first part of the talk will focus on using pretrained diffusion models as generative priors for several degradation-informed restoration tasks, without requiring retraining. The second part will cover more complex blind restoration tasks, where the degradation process is unknown and inferred iteratively. Examples from historical music restoration will be presented, demonstrating how these techniques improve the quality of degraded audio while maintaining coherence with the original recordings.

Speaker: Eloi Moliner (Aalto University)

19 Unsupervised Blind Joint Dereverberation and Room Acoustics Estimation with Diffusion Models

We present an unsupervised method for single-channel blind dereverberation and room impulse response (RIR) estimation, called BUDDy. The algorithm is rooted in Bayesian posterior sampling: it combines a likelihood model enforcing fidelity to the reverberant measurement, and an ane- choic speech prior implemented by an unconditional diffusion model. We design a parametric filter representing the RIR, with exponential decay for each frequency subband. Room acoustics estimation and speech dereverberation are jointly carried out, as the filter parameters are iteratively estimated and the speech utterance refined along the reverse diffusion trajectory. In a blind scenario where the room impulse response is unknown, BUDDy successfully performs speech dereverberation in various acoustic scenarios, significantly outperforming other blind unsupervised baselines. Unlike supervised methods, which often struggle to generalize, BUDDy seamlessly adapts to different acoustic condi- tions. This paper extends our previous work by offering new ex- perimental results and insights into the algorithm’s performance and versatility. We first investigate the robustness of informed dereverberation methods to RIR estimation errors, to motivate the joint acoustic estimation and dereverberation paradigm. Then, we demonstrate the adaptability of our method to high- resolution singing voice dereverberation, study its performance in RIR estimation, and conduct subjective evaluation experiments to validate the perceptual quality of the results, among other contributions.

Speaker: Jean-Marie Lemercier (University of Hamburg)

12:30 Lunch

20 Score-based Diffusion Models as Principled Priors for Inverse Imaging

Abstract: Score-based diffusion models represent rich image priors, but using them to solve inverse problems in imaging poses challenges. In this talk, I will address two challenges: (1) the intractability of exact posterior sampling with a score-based prior and (2) the fact that diffusion models often violate physical (e.g., PDE) constraints inherent in the training data. For (1), we propose using the exact log-probability function of a score-based diffusion model as the regularizer in variational inference. We apply this method to black-hole imaging and re-imagine the M87* black hole under different assumptions. For (2), we propose neural approximate mirror maps for constrained diffusion models. By learning an approximate mirror map, we can train diffusion models in an unconstrained space that satisfy the constraint through an inverse mirror map. This approach works for general constraints and generative models. We demonstrate applying it to solve constrained inverse problems, such as data assimilation with PDE constraints.

Speaker: Berthy Feng (California Institute of Technology)

21 Reducing the Cost of Posterior Sampling in Linear Inverse Problems via Task-dependent Score Learning

Score-based diffusion models (SDMs) enable efficient posterior sampling in Bayesian inverse problems. Traditional methods require multiple forward mapping evaluations, which are computationally costly. We focus on linear inverse problems, such as medical imaging, where these costs are significant. Our novel approach eliminates the need for forward mapping evaluations during sampling by shifting computation to an offline task. We train the score of a diffusion-like process based on the forward mapping, and then derive the posterior score using affine transformations without error. Our method, applicable to infinite-dimensional models, significantly reduces computational costs, making it ideal for frequent posterior sampling.

Speaker: Fabian Schneider (Lappeenranta-Lahti University of Technology)

22 ForwardNET: Solving the Near-field Phase Retrieval Inverse Problem Using a Forward-Operator Guided Generative Network

The near-field phase retrieval problem is a non-linear, ill-posed inverse problem. It is also an important step in X-ray imaging, a precursor to the tomographic reconstruction stage. We will discuss ForwardNET, a deep learning model for solving this problem, along with other inverse problems that we face at our PETRA III, P05 and P07 experiments. This includes fly-scan tomographic reconstruction, and reconstructing images from Gaussian, motion and rotational blur. We will then continue to discuss how we handle noise and uncertainity.

Speaker: Dawit Hailu (Hereon (Helmholtz-Zentrum Hereon))

15:30 Farewell