Publications
255 results found
Wang R, Blu T, Dragotti PL, 2023, Reconstruction of images with finite rate of innovation from noisy tomographic projections, 2023 31st European Signal Processing Conference (EUSIPCO), Publisher: IEEE, ISSN: 2076-1465
In this paper, we propose a robust 2D tomographic reconstruction algorithm for classes of piecewise constant images whose edges are defined by curves with finite rate innovation (FRI). The curve is known to satisfy an annihilation relation associated with the derivative of the image. Given limited numbers of noisy projections and inaccurately known angles, unlike conventional methods that are directly reconstructing images from projections, we first recover the curve exploiting the frequency domain annihilation relation. Once the curve is retrieved, we leverage the spatial domain interpretation of the annihilation relation, and formulate the image reconstruction as a regularized optimization problem. To enhance the reconstruction quality, we further refine the angles given the reconstructed image and iterate the process. Experiments show that by doing so, we can robustly reconstruct the 2D image even from limited numbers of severely corrupted projections and highly inaccurate projection angles.
Wang R, Dragotti PL, 2023, Perfect reconstruction of classes of 3D non-bandlimited signals from tomographic projections at unknown angles, 2023 31st European Signal Processing Conference (EUSIPCO), Publisher: IEEE, Pages: 1903-1907, ISSN: 2076-1465
In this paper, we consider the problem of reconstructing 3D objects from sampled 2D tomographic projections at unknown angles. We consider 3D polyhedrons and we provide a constructive solution to recover the 3D structure of the polyhedron and the projection directions. The proposed method is based on the observation that the 2D projections are actually signals with finite rate of innovation, and this allows us to retrieve the locations of the projected vertices using Prony's method. We are then able to unveil the 3D geometric information of the projection directions, as well as the structure of the polyhedron with an algebraic approach. The reconstruction method can also be applied to 3D point source models and is resilient to noise.
Liu S, Leung VCH, Dragotti PL, 2023, First-spike coding promotes accurate and efficient spiking neural networks for discrete events with rich temporal structures, Frontiers in Neuroscience, Vol: 17, ISSN: 1662-453X
Spiking neural networks (SNNs) are well-suited to process asynchronous event-based data. Most of the existing SNNs use rate-coding schemes that focus on firing rate (FR), and so they generally ignore the spike timing in events. On the contrary, methods based on temporal coding, particularly time-to-first-spike (TTFS) coding, can be accurate and efficient but they are difficult to train. Currently, there is limited research on applying TTFS coding to real events, since traditional TTFS-based methods impose one-spike constraint, which is not realistic for event-based data. In this study, we present a novel decision-making strategy based on first-spike (FS) coding that encodes FS timings of the output neurons to investigate the role of the first-spike timing in classifying real-world event sequences with complex temporal structures. To achieve FS coding, we propose a novel surrogate gradient learning method for discrete spike trains. In the forward pass, output spikes are encoded into discrete times to generate FS times. In the backpropagation, we develop an error assignment method that propagates error from FS times to spikes through a Gaussian window, and then supervised learning for spikes is implemented through a surrogate gradient approach. Additional strategies are introduced to facilitate the training of FS timings, such as adding empty sequences and employing different parameters for different layers. We make a comprehensive comparison between FS and FR coding in the experiments. Our results show that FS coding achieves comparable accuracy to FR coding while leading to superior energy efficiency and distinct neuronal dynamics on data sequences with very rich temporal structures. Additionally, a longer time delay in the first spike leads to higher accuracy, indicating important information is encoded in the timing of the first spike.
Liu S, Dragotti PL, 2023, Sensing diversity and sparsity models for event generation and video reconstruction from events, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol: 45, Pages: 12444-12458, ISSN: 0162-8828
Events-to-video (E2V) reconstruction and video-to-events (V2E) simulation are two fundamental research topics in event-based vision. Current deep neural networks for E2V reconstruction are usually complex and difficult to interpret. Moreover, existing event simulators are designed to generate realistic events, but research on how to improve the event generation process has been so far limited. In this paper, we propose a light, simple model-based deep network for E2V reconstruction, explore the diversity for adjacent pixels in V2E generation, and finally build a video-to-events-to-video (V2E2V) architecture to validate how alternative event generation strategies improve video reconstruction. For the E2V reconstruction, we model the relationship between events and intensity using sparse representation models. A convolutional ISTA network (CISTA) is then designed using the algorithm unfolding strategy. Long short-term temporal consistency (LSTC) constraints are further introduced to enhance the temporal coherence. In the V2E generation, we introduce the idea of having interleaved pixels with different contrast threshold and lowpass bandwidth and conjecture that this can help extract more useful information from intensity. Finally, V2E2V architecture is used to verify the effectiveness of this strategy. Results highlight that our CISTA-LSTC network outperforms state-of-the-art methods and achieves better temporal consistency. Sensing diversity in event generation reveals more fine details and this leads to a significantly improved reconstruction quality.
Erdemir E, Tung T-Y, Dragotti PL, et al., 2023, Generative joint source-channel coding for semantic image transmission, IEEE Journal on Selected Areas in Communications, Vol: 41, Pages: 2645-2657, ISSN: 0733-8716
Recent works have shown that joint source-channel coding (JSCC) schemes using deep neural networks (DNNs), called DeepJSCC, provide promising results in wireless image transmission. However, these methods mostly focus on the distortion of the reconstructed signals with respect to the input image, rather than their perception by humans. However, focusing on traditional distortion metrics alone does not necessarily result in high perceptual quality, especially in extreme physical conditions, such as very low bandwidth compression ratio (BCR) and low signal-to-noise ratio (SNR) regimes. In this work, we propose two novel JSCC schemes that leverage the perceptual quality of deep generative models (DGMs) for wireless image transmission, namely InverseJSCC and GenerativeJSCC. While the former is an inverse problem approach to DeepJSCC, the latter is an end-to-end optimized JSCC scheme. In both, we optimize a weighted sum of mean squared error (MSE) and learned perceptual image patch similarity (LPIPS) losses, which capture more semantic similarities than other distortion metrics. InverseJSCC performs denoising on the distorted reconstructions of a DeepJSCC model by solving an inverse optimization problem using the pre-trained style-based generative adversarial network (StyleGAN). Our simulation results show that InverseJSCC significantly improves the state-of-the-art DeepJSCC in terms of perceptual quality in edge cases. In GenerativeJSCC, we carry out end-to-end training of an encoder and a StyleGAN-based decoder, and show that GenerativeJSCC significantly outperforms DeepJSCC both in terms of distortion and perceptual quality.
Erdemir E, Dragotti PL, Gündüz D, 2023, Active privacy-utility trade-off against inference in time-series data sharing, IEEE Journal on Selected Areas in Information Theory, Vol: 4, Pages: 159-173, ISSN: 2641-8770
Internet of things devices have become highly popular thanks to the services they offer. However, they also raise privacy concerns since they share fine-grained time-series user data with untrusted third parties. We model the users personal information as the secret variable, to be kept private from an honest-but-curious service provider, and the useful variable, to be disclosed for utility. We consider an active learning framework, where one out of a finite set of measurement mechanisms is chosen at each time step, each revealing some information about the underlying secret and useful variables, albeit with different statistics. The measurements are taken such that the correct value of useful variable can be detected quickly, while the confidence on the secret variable remains below a predefined level. For privacy measure, we consider both the probability of correctly detecting the secret variable value and the mutual information between the secret and released data. We formulate both problems as partially observable Markov decision processes, and numerically solve by advantage actor-critic deep reinforcement learning. We evaluate the privacy-utility trade-off of the proposed policies on both the synthetic and real-world time-series datasets.
Leung VCH, Huang J-J, Eldar YC, et al., 2023, Learning-based reconstruction of FRI signals, IEEE Transactions on Signal Processing, Vol: 71, Pages: 2564-2578, ISSN: 1053-587X
Finite Rate of Innovation (FRI) sampling theory enables reconstruction of classes of continuous non-bandlimited signals that have a small number of free parameters from their low-rate discrete samples. This task is often translated into a spectral estimation problem that is solved using methods involving estimating signal subspaces, which tend to break down at a certain peak signal-to-noise ratio (PSNR). To avoid this breakdown, we consider alternative approaches that make use of information from labelled data. We propose two model-based learning methods, including deep unfolding the denoising process in spectral estimation, and constructing an encoder-decoder deep neural network that models the acquisition process. Simulation results of both learning algorithms indicate significant improvements of the breakdown PSNR over classical subspace-based methods. While the deep unfolded network achieves similar performance as the classical FRI techniques and outperforms the encoder-decoder network in the low noise regimes, the latter allows to reconstruct the FRI signal even when the sampling kernel is unknown. We also achieve competitive results in detecting pulses from in vivo calcium imaging data in terms of true positive and false positive rate while providing more precise estimations.
Verinaz-Jadan H, Howe CL, Song P, et al., 2023, Physics-based deep learning for imaging neuronal activity via two-photon and light field microscopy, IEEE Transactions on Computational Imaging, Vol: 9, Pages: 565-580, ISSN: 2333-9403
Light Field Microscopy (LFM) is an imaging technique that offers the opportunity to study fast dynamics in biological systems due to its 3D imaging speed and is particularly attractive for functional neuroimaging. Traditional model-based approaches employed in microscopy for reconstructing 3D images from light-field data are affected by reconstruction artifacts and are computationally demanding. This work introduces a deep neural network for LFM to image neuronal activity under adverse conditions: limited training data, background noise, and scattering mammalian brain tissue. The architecture of the network is obtained by unfolding the ISTA algorithm and is based on the observation that neurons in the tissue are sparse. Our approach is also based on a novel modelling of the imaging system that uses a linear convolutional neural network to fit the physics of the acquisition process. We train the network in a semi-supervised manner based on an adversarial training framework. The small labelled dataset required for training is acquired from a single sample via two-photon microscopy, a point-scanning 3D imaging technique that achieves high spatial resolution and deep tissue penetration but at a lower speed than LFM. We introduce physics knowledge of the system in the design of the network architecture and during training to complete our semi-supervised approach. We experimentally show that in the proposed scenario, our method performs better than typical deep learning and model-based reconstruction strategies for imaging neuronal activity in mammalian brain tissue via LFM, considering reconstruction quality, generalization to functional imaging, and reconstruction speed.
Yan S, Huang J-J, Verinaz-Jadan H, et al., 2023, A Fast Automatic Method for Deconvoluting Macro X-Ray Fluorescence Data Collected From Easel Paintings, IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, Vol: 9, Pages: 649-664, ISSN: 2573-0436
Huang J-J, Dragotti PL, 2022, WINNet: wavelet-inspired invertible network for image denoising, IEEE Transactions on Image Processing, Vol: 31, Pages: 4377-4392, ISSN: 1057-7149
Image denoising aims to restore a clean image from an observed noisy one. Model-based image denoising approaches can achieve good generalization ability over different noise levels and are with high interpretability. Learning-based approaches are able to achieve better results, but usually with weaker generalization ability and interpretability. In this paper, we propose a wavelet-inspired invertible network (WINNet) to combine the merits of the wavelet-based approaches and learning-based approaches. The proposed WINNet consists of K -scale of lifting inspired invertible neural networks (LINNs) and sparsity-driven denoising networks together with a noise estimation network. The network architecture of LINNs is inspired by the lifting scheme in wavelets. LINNs are used to learn a non-linear redundant transform with perfect reconstruction property to facilitate noise removal. The denoising network implements a sparse coding process for denoising. The noise estimation network estimates the noise level from the input image which will be used to adaptively adjust the soft-thresholds in LINNs. The forward transform of LINNs produces a redundant multi-scale representation for denoising. The denoised image is reconstructed using the inverse transform of LINNs with the denoised detail channels and the original coarse channel. The simulation results show that the proposed WINNet method is highly interpretable and has strong generalization ability to unseen noise levels. It also achieves competitive results in the non-blind/blind image denoising and in image deblurring.
Verinaz-Jadan H, Song P, Howe CL, et al., 2022, Shift-invariant-subspace discretization and volume reconstruction for light field microscopy, IEEE Transactions on Computational Imaging, Vol: 8, Pages: 286-301, ISSN: 2573-0436
Light Field Microscopy (LFM) is an imaging technique that captures 3D spatial information with a single 2D image. LFM is attractive because of its relatively simple implementation and fast volume acquisition rate. Capturing volume time series at a camera frame rate can enable the study of the behaviour of many biological systems. For instance, it could provide insights into the communication dynamics of living 3D neural networks. However, conventional 3D reconstruction algorithms for LFM typically suffer from high computational cost, low lateral resolution, and reconstruction artifacts. In this work, we study the origin of these issues and propose novel techniques to improve the performance of the reconstruction process. First, we propose a discretization approach that uses shift-invariant subspaces to generalize the typical discretization framework used in LFM. Then, we study the shift-invariant-subspace assumption as a prior for volume reconstruction under ideal conditions. Furthermore, we present a method to reduce the computational time of the forward model by using singular value decomposition (SVD). Finally, we propose to use iterative approaches that incorporate additional priors to perform artifact-free 3D reconstruction from real light field images. We experimentally show that our approach performs better than Richardson-Lucy-based strategies in computational time, image quality, and artifact reduction.
Howe C, Song P, Verinaz Jadan HI, et al., 2022, Comparing synthetic refocusing to deconvolution for the extraction of neuronal calcium transients from light fields, Neurophotonics, Vol: 9, Pages: 1-17, ISSN: 2329-4248
Significance: Light-field microscopy (LFM) enables fast, light-efficient, volumetric imaging of neuronal activity with calcium indicators. Calcium transients differ in temporal signal-to-noise ratio (tSNR) and spatial confinement when extracted from volumes reconstructed by different algorithms.Aim: We evaluated the capabilities and limitations of two light-field reconstruction algorithms for calcium fluorescence imaging.Approach: We acquired light-field image series from neurons either bulk-labeled or filled intracellularly with the red-emitting calcium dye CaSiR-1 in acute mouse brain slices. We compared the tSNR and spatial onfinement of calcium signals extracted from volumes reconstructed with synthetic refocusing and Richardson-Lucy 3D deconvolution with and without total variation regularization.Results: Both synthetic refocusing and Richardson-Lucy deconvolution resolved calcium signals from single cells and neuronal dendrites in three dimensions. Increasing deconvolution iteration number improved spatial confinement but reduced tSNR compared to synthetic refocusing. Volumetric light-field imaging did not decrease calcium signal tSNR compared to interleaved, widefield image series acquired in matched planes.Conclusions: LFM enables high-volume rate, volumetric imaging of calcium transients in single cells (bulk-labeled), somata and dendrites (intracellular loaded). The trade-offs identified for tSNR, spatial confinement, and computational cost indicate which of synthetic refocusing or deconvolution can better realize the scientific requirements of future LFM calcium imaging applications.
Foust A, Song P, Verinaz Jadan HI, et al., 2022, Light-field microscopy for optical imaging of neuronal activity: when model-based methods meet data-driven approaches, IEEE: Signal Processing Magazine, Vol: 39, ISSN: 1053-5888
Understanding how networks of neurons process information is one of the key challenges in modern neuroscience.A necessary step to achieve this goal is to be able to observe the dynamics of large populations of neurons overa large area of the brain. Light-field microscopy (LFM), a type of scanless microscope, is a particularly attractivecandidate for high-speed three-dimensional (3D) imaging. It captures volumetric information in a single snapshot,allowing volumetric imaging at video frame-rates. Specific features of imaging neuronal activity using LFM callfor the development of novel machine learning approaches that fully exploit priors embedded in physics and opticsmodels. Signal processing theory and wave-optics theory could play a key role in filling this gap, and contributeto novel computational methods with enhanced interpretability and generalization by integrating model-driven anddata-driven approaches. This paper is devoted to a comprehensive survey to state-of-the-art of computational methods for LFM, with a focus on model-based and data-driven approaches
Chen Y, Schönlieb C-B, Liò P, et al., 2022, AI-based reconstruction for fast MRI – a systematic review and meta-analysis, Proceedings of the Institute of Electrical and Electronics Engineers (IEEE), Vol: 110, Pages: 224-245, ISSN: 0018-9219
Compressed sensing (CS) has been playing a key role in accelerating the magnetic resonance imaging (MRI) acquisition process. With the resurgence of artificial intelligence, deep neural networks and CS algorithms are being integrated to redefine the state of the art of fast MRI. The past several years have witnessed substantial growth in the complexity, diversity, and performance of deep learning-based CS techniques that are dedicated to fastMRI. In this meta-analysis, we systematically review the deep learning-based CS techniques for fast MRI, describe key model designs, highlight breakthroughs, and discuss promising directions. We have also introduced a comprehensive analysis framework and a classification system to assess the pivotal role of deep learning in CS-based accelerationfor MRI.
Wang R, Huangt J-J, Dragotti PL, 2022, FRISPEE: FRI-Based Single Image Super-Resolution with Deep Recursive Residual Network, 30th European Signal Processing Conference (EUSIPCO), Publisher: IEEE, Pages: 479-483, ISSN: 2076-1465
Liu S, Alexandru R, Dragotti PL, 2022, CONVOLUTIONAL ISTA NETWORK WITH TEMPORAL CONSISTENCY CONSTRAINTS FOR VIDEO RECONSTRUCTION FROM EVENT CAMERAS, 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Publisher: IEEE, Pages: 1935-1939, ISSN: 1520-6149
- Author Web Link
- Cite
- Citations: 1
Sober B, Bucklow S, Daly N, et al., 2022, Revealing and Reconstructing Hidden or Lost Features in Art Investigation, IEEE BITS the Information Theory Magazine, Pages: 1-16, ISSN: 2692-4080
Erdemir E, Dragotti PL, Gunduz D, 2022, PRIVACY-AWARE COMMUNICATION OVER A WIRETAP CHANNEL WITH GENERATIVE NETWORKS, 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Publisher: IEEE, Pages: 2989-2993, ISSN: 1520-6149
- Author Web Link
- Cite
- Citations: 1
Wang R, Alexandru R, Dragotti PL, 2022, PERFECT RECONSTRUCTION OF CLASSES OF NON-BANDLIMITED SIGNALS FROM PROJECTIONS WITH UNKNOWN ANGLES, 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Publisher: IEEE, Pages: 5877-5881, ISSN: 1520-6149
Gao F, Deng X, Xu M, et al., 2022, Multi-Modal Convolutional Dictionary Learning, IEEE TRANSACTIONS ON IMAGE PROCESSING, Vol: 31, Pages: 1325-1339, ISSN: 1057-7149
- Author Web Link
- Cite
- Citations: 5
Pu W, Huang J-J, Sober B, et al., 2022, Mixed X-Ray Image Separation for Artworks With Concealed Designs, IEEE TRANSACTIONS ON IMAGE PROCESSING, Vol: 31, Pages: 4458-4473, ISSN: 1057-7149
- Author Web Link
- Cite
- Citations: 1
Perez-Nieves N, Leung VCH, Dragotti PL, et al., 2021, Neural heterogeneity promotes robust learning, Nature Communications, Vol: 12, Pages: 5791-5791
The brain has a hugely diverse, heterogeneous structure. Whether or not heterogeneity at the neural level plays a functional role remains unclear, and has been relatively little explored in models which are often highly homogeneous. We compared the performance of spiking neural networks trained to carry out tasks of real-world difficulty, with varying degrees of heterogeneity, and found that it substantially improved task performance. Learning was more stable and robust, particularly for tasks with a rich temporal structure. In addition, the distribution of neuronal parameters in the trained networks closely matches those observed experimentally. We suggest that the heterogeneity observed in the brain may be more than just the byproduct of noisy processes, but rather may serve an active and important role in allowing animals to learn in changing environments.Summary Neural heterogeneity is metabolically efficient for learning, and optimal parameter distribution matches experimental data.
Wang X, Jiang L, Li L, et al., 2021, Joint Learning of 3D Lesion Segmentation and Classification for Explainable COVID-19 Diagnosis, IEEE TRANSACTIONS ON MEDICAL IMAGING, Vol: 40, Pages: 2463-2476, ISSN: 0278-0062
- Author Web Link
- Cite
- Citations: 29
Yan S, Huang J-J, Daly N, et al., 2021, When de prony met Leonardo: an automatic algorithm for chemical element extraction from macro X-ray fluorescence data, IEEE Transactions on Computational Imaging, Vol: 7, Pages: 908-924, ISSN: 2333-9403
Macro X-ray Fluorescence (MA-XRF) scanning is an increasingly widely used technique for analytical imaging of paintings and other artworks. The datasets acquired must be processed to produce maps showing the distribution of the chemical elements that are present in the painting. Existing approaches require varying degrees of expert user intervention, in particular to select a list of target elements against which to fit the data. In this paper, we propose a novel approach that can automatically extract and identify chemical elements and their distributions from MA-XRF datasets. The proposed approach consists of three parts: 1) pre-processing steps, 2) pulse detection and model order selection based on Finite Rate of Innovation theory, and 3) chemical element estimation based on Cramér-Rao bounding techniques. The performance of our approach is assessed using MA-XRF datasets acquired from paintings in the collection of the National Gallery, London. The results presented show the ability of our approach to detect elements with weak X-ray fluorescence intensity and from noisy XRF spectra, to separate overlapping elemental signals and, excitingly, to aid visualisation of hidden underdrawing in a masterpiece by Leonardo da Vinci.
Leung VCH, Huang J-J, Eldar Y, et al., 2021, Reconstruction Of FRI Signals Using Autoencoders With Fixed Decoders, 2021 29th European Signal Processing Conference (EUSIPCO), Pages: 1496-1500
Yu Q, Huang J-J, Zhu J, et al., 2021, Deep phase retrieval: Analyzing over-parameterization in phase retrieval, SIGNAL PROCESSING, Vol: 180, ISSN: 0165-1684
Hilton M, Alexandru R, Dragotti PL, 2021, GUARANTEED RECONSTRUCTION FROM INTEGRATE-AND-FIRE NEURONS WITH ALPHA SYNAPTIC ACTIVATION, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Publisher: IEEE, Pages: 5474-5478
- Author Web Link
- Cite
- Citations: 1
Huang J-J, Dragotti PL, 2021, LINN: Lifting Inspired Invertible Neural Network for Image Denoising, 29th European Signal Processing Conference (EUSIPCO), Publisher: EUROPEAN ASSOC SIGNAL SPEECH & IMAGE PROCESSING-EURASIP, Pages: 636-640, ISSN: 2076-1465
Alexandru R, Blu T, Dragotti PL, 2021, LOCALISING DIFFUSION SOURCES FROM SAMPLES TAKEN ALONG UNKNOWN PARAMETRIC TRAJECTORIES, 29th European Signal Processing Conference (EUSIPCO), Publisher: EUROPEAN ASSOC SIGNAL SPEECH & IMAGE PROCESSING-EURASIP, Pages: 2199-2203, ISSN: 2076-1465
Pu W, Huang J, Sober B, et al., 2021, A Learning Based Approach to Separate Mixed X-Ray Images Associated with Artwork with Concealed Designs, 29th European Signal Processing Conference (EUSIPCO), Publisher: EUROPEAN ASSOC SIGNAL SPEECH & IMAGE PROCESSING-EURASIP, Pages: 1491-1495, ISSN: 2076-1465
- Author Web Link
- Cite
- Citations: 2
This data is extracted from the Web of Science and reproduced under a licence from Thomson Reuters. You may not copy or re-distribute this data in whole or in part without the written consent of the Science business of Thomson Reuters.