Publications
249 results found
Liu S, Dragotti PL, 2023, Sensing Diversity and Sparsity Models for Event Generation and Video Reconstruction from Events., IEEE Trans Pattern Anal Mach Intell, Vol: 45, Pages: 12444-12458
Events-to-video (E2V) reconstruction and video-to-events (V2E) simulation are two fundamental research topics in event-based vision. Current deep neural networks for E2V reconstruction are usually complex and difficult to interpret. Moreover, existing event simulators are designed to generate realistic events, but research on how to improve the event generation process has been so far limited. In this paper, we propose a light, simple model-based deep network for E2V reconstruction, explore the diversity for adjacent pixels in V2E generation, and finally build a video-to-events-to-video (V2E2V) architecture to validate how alternative event generation strategies improve video reconstruction. For the E2V reconstruction, we model the relationship between events and intensity using sparse representation models. A convolutional ISTA network (CISTA) is then designed using the algorithm unfolding strategy. Long short-term temporal consistency (LSTC) constraints are further introduced to enhance the temporal coherence. In the V2E generation, we introduce the idea of having interleaved pixels with different contrast threshold and lowpass bandwidth and conjecture that this can help extract more useful information from intensity. Finally, V2E2V architecture is used to verify the effectiveness of this strategy. Results highlight that our CISTA-LSTC network outperforms state-of-the-art methods and achieves better temporal consistency. Sensing diversity in event generation reveals more fine details and this leads to a significantly improved reconstruction quality.
Liu S, Leung VCH, Dragotti PL, 2023, First-spike coding promotes accurate and efficient spiking neural networks for discrete events with rich temporal structures, Frontiers in Neuroscience, ISSN: 1662-453X
Erdemir E, Tung T-Y, Dragotti PL, et al., 2023, Generative joint source-channel coding for semantic image transmission, IEEE Journal on Selected Areas in Communications, Vol: 41, Pages: 2645-2657, ISSN: 0733-8716
Recent works have shown that joint source-channel coding (JSCC) schemes using deep neural networks (DNNs), called DeepJSCC, provide promising results in wireless image transmission. However, these methods mostly focus on the distortion of the reconstructed signals with respect to the input image, rather than their perception by humans. However, focusing on traditional distortion metrics alone does not necessarily result in high perceptual quality, especially in extreme physical conditions, such as very low bandwidth compression ratio (BCR) and low signal-to-noise ratio (SNR) regimes. In this work, we propose two novel JSCC schemes that leverage the perceptual quality of deep generative models (DGMs) for wireless image transmission, namely InverseJSCC and GenerativeJSCC. While the former is an inverse problem approach to DeepJSCC, the latter is an end-to-end optimized JSCC scheme. In both, we optimize a weighted sum of mean squared error (MSE) and learned perceptual image patch similarity (LPIPS) losses, which capture more semantic similarities than other distortion metrics. InverseJSCC performs denoising on the distorted reconstructions of a DeepJSCC model by solving an inverse optimization problem using the pre-trained style-based generative adversarial network (StyleGAN). Our simulation results show that InverseJSCC significantly improves the state-of-the-art DeepJSCC in terms of perceptual quality in edge cases. In GenerativeJSCC, we carry out end-to-end training of an encoder and a StyleGAN-based decoder, and show that GenerativeJSCC significantly outperforms DeepJSCC both in terms of distortion and perceptual quality.
Verinaz-Jadan H, Howe CL, Song P, et al., 2023, Physics-Based Deep Learning for Imaging Neuronal Activity via Two-Photon and Light Field Microscopy, IEEE Transactions on Computational Imaging, Vol: 9, Pages: 565-580, ISSN: 2573-0436
Light Field Microscopy (LFM) is an imaging technique that offers the opportunity to study fast dynamics in biological systems due to its 3D imaging speed and is particularly attractive for functional neuroimaging. Traditional model-based approaches employed in microscopy for reconstructing 3D images from light-field data are affected by reconstruction artifacts and are computationally demanding. This work introduces a deep neural network for LFM to image neuronal activity under adverse conditions: limited training data, background noise, and scattering mammalian brain tissue. The architecture of the network is obtained by unfolding the ISTA algorithm and is based on the observation that neurons in the tissue are sparse. Our approach is also based on a novel modelling of the imaging system that uses a linear convolutional neural network to fit the physics of the acquisition process. We train the network in a semi-supervised manner based on an adversarial training framework. The small labelled dataset required for training is acquired from a single sample via two-photon microscopy, a point-scanning 3D imaging technique that achieves high spatial resolution and deep tissue penetration but at a lower speed than LFM. We introduce physics knowledge of the system in the design of the network architecture and during training to complete our semi-supervised approach. We experimentally show that in the proposed scenario, our method performs better than typical deep learning and model-based reconstruction strategies for imaging neuronal activity in mammalian brain tissue via LFM, considering reconstruction quality, generalization to functional imaging, and reconstruction speed.
Yan S, Huang JJ, Verinaz-Jadan H, et al., 2023, A Fast Automatic Method for Deconvoluting Macro X-Ray Fluorescence Data Collected from Easel Paintings, IEEE Transactions on Computational Imaging, Vol: 9, Pages: 649-664, ISSN: 2573-0436
Macro X-ray Fluorescence (MA-XRF) scanning is increasingly widely used by researchers in heritage science to analyse easel paintings as one of a suite of non-invasive imaging techniques. The task of processing the resulting MA-XRF datacube generated in order to produce individual chemical element maps is called MA-XRF deconvolution. While there are several existing methods that have been proposed for MA-XRF deconvolution, they require a degree of manual intervention from the user that can affect the final results. The state-of-the-art AFRID approach can automatically deconvolute the datacube without user input, but it has a long processing time and does not exploit spatial dependency. In this paper, we propose two versions of a fast automatic deconvolution (FAD) method for MA-XRF datacubes collected from easel paintings with ADMM (alternating direction method of multipliers) and FISTA (fast iterative shrinkage-thresholding algorithm). The proposed FAD method not only automatically analyses the datacube and produces element distribution maps of high-quality with spatial dependency considered, but also significantly reduces the running time. The results generated on the MA-XRF datacubes collected from two easel paintings from the National Gallery, London, verify the performance of the proposed FAD method.
Erdemir E, Dragotti PL, Gündüz D, 2023, Active privacy-utility trade-off against inference in time-series data sharing, IEEE Journal on Selected Areas in Information Theory, Vol: 4, Pages: 159-173, ISSN: 2641-8770
Internet of things devices have become highly popular thanks to the services they offer. However, they also raise privacy concerns since they share fine-grained time-series user data with untrusted third parties. We model the users personal information as the secret variable, to be kept private from an honest-but-curious service provider, and the useful variable, to be disclosed for utility. We consider an active learning framework, where one out of a finite set of measurement mechanisms is chosen at each time step, each revealing some information about the underlying secret and useful variables, albeit with different statistics. The measurements are taken such that the correct value of useful variable can be detected quickly, while the confidence on the secret variable remains below a predefined level. For privacy measure, we consider both the probability of correctly detecting the secret variable value and the mutual information between the secret and released data. We formulate both problems as partially observable Markov decision processes, and numerically solve by advantage actor-critic deep reinforcement learning. We evaluate the privacy-utility trade-off of the proposed policies on both the synthetic and real-world time-series datasets.
Leung VCH, Huang J-J, Eldar YC, et al., 2023, Learning-based reconstruction of FRI signals, IEEE Transactions on Signal Processing, Vol: 71, Pages: 2564-2578, ISSN: 1053-587X
Finite Rate of Innovation (FRI) sampling theory enables reconstruction of classes of continuous non-bandlimited signals that have a small number of free parameters from their low-rate discrete samples. This task is often translated into a spectral estimation problem that is solved using methods involving estimating signal subspaces, which tend to break down at a certain peak signal-to-noise ratio (PSNR). To avoid this breakdown, we consider alternative approaches that make use of information from labelled data. We propose two model-based learning methods, including deep unfolding the denoising process in spectral estimation, and constructing an encoder-decoder deep neural network that models the acquisition process. Simulation results of both learning algorithms indicate significant improvements of the breakdown PSNR over classical subspace-based methods. While the deep unfolded network achieves similar performance as the classical FRI techniques and outperforms the encoder-decoder network in the low noise regimes, the latter allows to reconstruct the FRI signal even when the sampling kernel is unknown. We also achieve competitive results in detecting pulses from in vivo calcium imaging data in terms of true positive and false positive rate while providing more precise estimations.
Verinaz-Jadan H, Howe CL, Song P, et al., 2022, Physics-based Deep Learning for Imaging Neuronal Activity via Two-photon and Light Field Microscopy
<jats:title>Abstract</jats:title><jats:p>Light Field Microscopy (LFM) is an imaging technique that offers the opportunity to study fast dynamics in biological systems due to its rapid 3D imaging rate. In particular, it is attractive to analyze neuronal activity in the brain. Unlike scanning-based imaging methods, LFM simultaneously encodes the spatial and angular information of light in a single snapshot. However, LFM is limited by a trade-off between spatial and angular resolution and is affected by scattering at deep layers in the brain tissue. In contrast, two-photon (2P) microscopy is a point-scanning 3D imaging technique that achieves higher spatial resolution, deeper tissue penetration, and reduced scattering effects. However, point-scanning acquisition limits the imaging speed in 2P microscopy and cannot be used to simultaneously monitor the activity of a large population of neurons. This work introduces a physics-driven deep neural network to image neuronal activity in scattering volume tissues using LFM. The architecture of the network is obtained by unfolding the ISTA algorithm and is based on the observation that the neurons in the tissue are sparse. The deep-network architecture is also based on a novel imaging system modeling that uses a linear convolutional neural network and fits the physics of the acquisition process. To achieve the high-quality reconstruction of neuronal activity in 3D brain tissues from temporal sequences of light field (LF) images, we train the network in a semi-supervised manner using generative adversarial networks (GANs). We use the TdTomato indicator to obtain static structural information of the tissue with the microscope operating in 2P scanning modality, representing the target reconstruction quality. We also use additional functional data in LF modality with GCaMP indicators to train the network. Our approach is tested under adverse conditions: limited training data, background noise, and scattering sampl
Huang J-J, Dragotti PL, 2022, WINNet: wavelet-inspired invertible network for image denoising, IEEE Transactions on Image Processing, Vol: 31, Pages: 4377-4392, ISSN: 1057-7149
Image denoising aims to restore a clean image from an observed noisy one. Model-based image denoising approaches can achieve good generalization ability over different noise levels and are with high interpretability. Learning-based approaches are able to achieve better results, but usually with weaker generalization ability and interpretability. In this paper, we propose a wavelet-inspired invertible network (WINNet) to combine the merits of the wavelet-based approaches and learning-based approaches. The proposed WINNet consists of K -scale of lifting inspired invertible neural networks (LINNs) and sparsity-driven denoising networks together with a noise estimation network. The network architecture of LINNs is inspired by the lifting scheme in wavelets. LINNs are used to learn a non-linear redundant transform with perfect reconstruction property to facilitate noise removal. The denoising network implements a sparse coding process for denoising. The noise estimation network estimates the noise level from the input image which will be used to adaptively adjust the soft-thresholds in LINNs. The forward transform of LINNs produces a redundant multi-scale representation for denoising. The denoised image is reconstructed using the inverse transform of LINNs with the denoised detail channels and the original coarse channel. The simulation results show that the proposed WINNet method is highly interpretable and has strong generalization ability to unseen noise levels. It also achieves competitive results in the non-blind/blind image denoising and in image deblurring.
Verinaz-Jadan H, Song P, Howe CL, et al., 2022, Shift-invariant-subspace discretization and volume reconstruction for light field microscopy, IEEE Transactions on Computational Imaging, Vol: 8, Pages: 286-301, ISSN: 2573-0436
Light Field Microscopy (LFM) is an imaging technique that captures 3D spatial information with a single 2D image. LFM is attractive because of its relatively simple implementation and fast volume acquisition rate. Capturing volume time series at a camera frame rate can enable the study of the behaviour of many biological systems. For instance, it could provide insights into the communication dynamics of living 3D neural networks. However, conventional 3D reconstruction algorithms for LFM typically suffer from high computational cost, low lateral resolution, and reconstruction artifacts. In this work, we study the origin of these issues and propose novel techniques to improve the performance of the reconstruction process. First, we propose a discretization approach that uses shift-invariant subspaces to generalize the typical discretization framework used in LFM. Then, we study the shift-invariant-subspace assumption as a prior for volume reconstruction under ideal conditions. Furthermore, we present a method to reduce the computational time of the forward model by using singular value decomposition (SVD). Finally, we propose to use iterative approaches that incorporate additional priors to perform artifact-free 3D reconstruction from real light field images. We experimentally show that our approach performs better than Richardson-Lucy-based strategies in computational time, image quality, and artifact reduction.
Howe C, Song P, Verinaz Jadan HI, et al., 2022, Comparing synthetic refocusing to deconvolution for the extraction of neuronal calcium transients from light fields, Neurophotonics, Vol: 9, Pages: 1-17, ISSN: 2329-4248
Significance: Light-field microscopy (LFM) enables fast, light-efficient, volumetric imaging of neuronal activity with calcium indicators. Calcium transients differ in temporal signal-to-noise ratio (tSNR) and spatial confinement when extracted from volumes reconstructed by different algorithms.Aim: We evaluated the capabilities and limitations of two light-field reconstruction algorithms for calcium fluorescence imaging.Approach: We acquired light-field image series from neurons either bulk-labeled or filled intracellularly with the red-emitting calcium dye CaSiR-1 in acute mouse brain slices. We compared the tSNR and spatial onfinement of calcium signals extracted from volumes reconstructed with synthetic refocusing and Richardson-Lucy 3D deconvolution with and without total variation regularization.Results: Both synthetic refocusing and Richardson-Lucy deconvolution resolved calcium signals from single cells and neuronal dendrites in three dimensions. Increasing deconvolution iteration number improved spatial confinement but reduced tSNR compared to synthetic refocusing. Volumetric light-field imaging did not decrease calcium signal tSNR compared to interleaved, widefield image series acquired in matched planes.Conclusions: LFM enables high-volume rate, volumetric imaging of calcium transients in single cells (bulk-labeled), somata and dendrites (intracellular loaded). The trade-offs identified for tSNR, spatial confinement, and computational cost indicate which of synthetic refocusing or deconvolution can better realize the scientific requirements of future LFM calcium imaging applications.
Foust A, Song P, Verinaz Jadan HI, et al., 2022, Light-field microscopy for optical imaging of neuronal activity: when model-based methods meet data-driven approaches, IEEE: Signal Processing Magazine, Vol: 39, ISSN: 1053-5888
Understanding how networks of neurons process information is one of the key challenges in modern neuroscience.A necessary step to achieve this goal is to be able to observe the dynamics of large populations of neurons overa large area of the brain. Light-field microscopy (LFM), a type of scanless microscope, is a particularly attractivecandidate for high-speed three-dimensional (3D) imaging. It captures volumetric information in a single snapshot,allowing volumetric imaging at video frame-rates. Specific features of imaging neuronal activity using LFM callfor the development of novel machine learning approaches that fully exploit priors embedded in physics and opticsmodels. Signal processing theory and wave-optics theory could play a key role in filling this gap, and contributeto novel computational methods with enhanced interpretability and generalization by integrating model-driven anddata-driven approaches. This paper is devoted to a comprehensive survey to state-of-the-art of computational methods for LFM, with a focus on model-based and data-driven approaches
Chen Y, Schönlieb C-B, Liò P, et al., 2022, AI-based reconstruction for fast MRI – a systematic review and meta-analysis, Proceedings of the Institute of Electrical and Electronics Engineers (IEEE), Vol: 110, Pages: 224-245, ISSN: 0018-9219
Compressed sensing (CS) has been playing a key role in accelerating the magnetic resonance imaging (MRI) acquisition process. With the resurgence of artificial intelligence, deep neural networks and CS algorithms are being integrated to redefine the state of the art of fast MRI. The past several years have witnessed substantial growth in the complexity, diversity, and performance of deep learning-based CS techniques that are dedicated to fastMRI. In this meta-analysis, we systematically review the deep learning-based CS techniques for fast MRI, describe key model designs, highlight breakthroughs, and discuss promising directions. We have also introduced a comprehensive analysis framework and a classification system to assess the pivotal role of deep learning in CS-based accelerationfor MRI.
Erdemir E, Dragotti PL, Gunduz D, 2022, PRIVACY-AWARE COMMUNICATION OVER A WIRETAP CHANNEL WITH GENERATIVE NETWORKS, 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Publisher: IEEE, Pages: 2989-2993, ISSN: 1520-6149
Gao F, Deng X, Xu M, et al., 2022, Multi-Modal Convolutional Dictionary Learning, IEEE TRANSACTIONS ON IMAGE PROCESSING, Vol: 31, Pages: 1325-1339, ISSN: 1057-7149
- Author Web Link
- Cite
- Citations: 1
Wang R, Alexandru R, Dragotti PL, 2022, PERFECT RECONSTRUCTION OF CLASSES OF NON-BANDLIMITED SIGNALS FROM PROJECTIONS WITH UNKNOWN ANGLES, 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Publisher: IEEE, Pages: 5877-5881, ISSN: 1520-6149
Pu W, Huang J-J, Sober B, et al., 2022, Mixed X-Ray Image Separation for Artworks With Concealed Designs, IEEE TRANSACTIONS ON IMAGE PROCESSING, Vol: 31, Pages: 4458-4473, ISSN: 1057-7149
Liu S, Alexandru R, Dragotti PL, 2022, CONVOLUTIONAL ISTA NETWORK WITH TEMPORAL CONSISTENCY CONSTRAINTS FOR VIDEO RECONSTRUCTION FROM EVENT CAMERAS, 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Publisher: IEEE, Pages: 1935-1939, ISSN: 1520-6149
Sober B, Bucklow S, Daly N, et al., 2022, Revealing and Reconstructing Hidden or Lost Features in Art Investigation, IEEE BITS the Information Theory Magazine, Pages: 1-16, ISSN: 2692-4080
Perez-Nieves N, Leung VCH, Dragotti PL, et al., 2021, Neural heterogeneity promotes robust learning, Nature Communications, Vol: 12, Pages: 5791-5791
The brain has a hugely diverse, heterogeneous structure. Whether or not heterogeneity at the neural level plays a functional role remains unclear, and has been relatively little explored in models which are often highly homogeneous. We compared the performance of spiking neural networks trained to carry out tasks of real-world difficulty, with varying degrees of heterogeneity, and found that it substantially improved task performance. Learning was more stable and robust, particularly for tasks with a rich temporal structure. In addition, the distribution of neuronal parameters in the trained networks closely matches those observed experimentally. We suggest that the heterogeneity observed in the brain may be more than just the byproduct of noisy processes, but rather may serve an active and important role in allowing animals to learn in changing environments.Summary Neural heterogeneity is metabolically efficient for learning, and optimal parameter distribution matches experimental data.
Wang X, Jiang L, Li L, et al., 2021, Joint Learning of 3D Lesion Segmentation and Classification for Explainable COVID-19 Diagnosis, IEEE TRANSACTIONS ON MEDICAL IMAGING, Vol: 40, Pages: 2463-2476, ISSN: 0278-0062
- Author Web Link
- Cite
- Citations: 15
Yan S, Huang J-J, Daly N, et al., 2021, When de prony met Leonardo: an automatic algorithm for chemical element extraction from macro X-ray fluorescence data, IEEE Transactions on Computational Imaging, Vol: 7, Pages: 908-924, ISSN: 2333-9403
Macro X-ray Fluorescence (MA-XRF) scanning is an increasingly widely used technique for analytical imaging of paintings and other artworks. The datasets acquired must be processed to produce maps showing the distribution of the chemical elements that are present in the painting. Existing approaches require varying degrees of expert user intervention, in particular to select a list of target elements against which to fit the data. In this paper, we propose a novel approach that can automatically extract and identify chemical elements and their distributions from MA-XRF datasets. The proposed approach consists of three parts: 1) pre-processing steps, 2) pulse detection and model order selection based on Finite Rate of Innovation theory, and 3) chemical element estimation based on Cramér-Rao bounding techniques. The performance of our approach is assessed using MA-XRF datasets acquired from paintings in the collection of the National Gallery, London. The results presented show the ability of our approach to detect elements with weak X-ray fluorescence intensity and from noisy XRF spectra, to separate overlapping elemental signals and, excitingly, to aid visualisation of hidden underdrawing in a masterpiece by Leonardo da Vinci.
Leung VCH, Huang J-J, Eldar Y, et al., 2021, Reconstruction Of FRI Signals Using Autoencoders With Fixed Decoders, 2021 29th European Signal Processing Conference (EUSIPCO), Pages: 1496-1500
Yu Q, Huang J-J, Zhu J, et al., 2021, Deep phase retrieval: Analyzing over-parameterization in phase retrieval, SIGNAL PROCESSING, Vol: 180, ISSN: 0165-1684
Hilton M, Alexandru R, Dragotti PL, 2021, GUARANTEED RECONSTRUCTION FROM INTEGRATE-AND-FIRE NEURONS WITH ALPHA SYNAPTIC ACTIVATION, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Publisher: IEEE, Pages: 5474-5478
- Author Web Link
- Cite
- Citations: 1
Huang J-J, Dragotti PL, 2021, LINN: Lifting Inspired Invertible Neural Network for Image Denoising, 29th European Signal Processing Conference (EUSIPCO), Publisher: EUROPEAN ASSOC SIGNAL SPEECH & IMAGE PROCESSING-EURASIP, Pages: 636-640, ISSN: 2076-1465
Alexandru R, Blu T, Dragotti PL, 2021, LOCALISING DIFFUSION SOURCES FROM SAMPLES TAKEN ALONG UNKNOWN PARAMETRIC TRAJECTORIES, 29th European Signal Processing Conference (EUSIPCO), Publisher: EUROPEAN ASSOC SIGNAL SPEECH & IMAGE PROCESSING-EURASIP, Pages: 2199-2203, ISSN: 2076-1465
Pu W, Huang J, Sober B, et al., 2021, A Learning Based Approach to Separate Mixed X-Ray Images Associated with Artwork with Concealed Designs, 29th European Signal Processing Conference (EUSIPCO), Publisher: EUROPEAN ASSOC SIGNAL SPEECH & IMAGE PROCESSING-EURASIP, Pages: 1491-1495, ISSN: 2076-1465
- Author Web Link
- Cite
- Citations: 1
Alexandru R, Blu T, Dragotti PL, 2021, Diffusion SLAM: Localizing Diffusion Sources From Samples Taken by Location-Unaware Mobile Sensors, IEEE TRANSACTIONS ON SIGNAL PROCESSING, Vol: 69, Pages: 5539-5554, ISSN: 1053-587X
Song P, Jadan HV, Howe CL, et al., 2021, MODEL-INSPIRED DEEP LEARNING FOR LIGHT-FIELD MICROSCOPY WITH APPLICATION TO NEURON LOCALIZATION, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Publisher: IEEE, Pages: 8087-8091
- Author Web Link
- Cite
- Citations: 3
This data is extracted from the Web of Science and reproduced under a licence from Thomson Reuters. You may not copy or re-distribute this data in whole or in part without the written consent of the Science business of Thomson Reuters.