Publications
1021 results found
Hinterwimmer F, Lazic I, Langer S, et al., 2023, Prediction of complications and surgery duration in primary TKA with high accuracy using machine learning with arthroplasty-specific data., Knee Surg Sports Traumatol Arthrosc, Vol: 31, Pages: 1323-1333
PURPOSE: The number of primary total knee arthroplasties (TKA) is expected to rise constantly. For patients and healthcare providers, the early identification of risk factors therefore becomes increasingly fundamental in the context of precision medicine. Others have already investigated the detection of risk factors by conducting literature reviews and applying conventional statistical methods. Since the prediction of events has been moderately accurate, a more comprehensive approach is needed. Machine learning (ML) algorithms have had ample success in many disciplines. However, these methods have not yet had a significant impact in orthopaedic research. The selection of a data source as well as the inclusion of relevant parameters is of utmost importance in this context. In this study, a standardized approach for ML in TKA to predict complications during surgery and an irregular surgery duration using data from two German arthroplasty-specific registries was evaluated. METHODS: The dataset is based on two initiatives of the German Society for Orthopaedics and Orthopaedic Surgery. A problem statement and initial parameters were defined. After screening, cleaning and preparation of these datasets, 864 cases of primary TKA (2016-2019) were gathered. The XGBoost algorithm was chosen and applied with a hyperparameter search, a cross validation and a loss weighting to cope with class imbalance. For final evaluation, several metrics (accuracy, sensitivity, specificity, AUC) were calculated. RESULTS: An accuracy of 92.0%, sensitivity of 34.8%, specificity of 95.8%, and AUC of 78.0% were achieved for predicting complications in primary TKA and 93.4%, 74.0%, 96.3%, and 91.6% for predicting irregular surgery duration, respectively. While traditional statistics (correlation coefficient) could not find any relevant correlation between any two parameters, the feature importance revealed several non-linear outcomes. CONCLUSION: In this study, a feasible ML model to predict outco
Dahan S, Fawaz A, Suliman MA, et al., 2023, The Multiscale Surface Vision Transformer., ArXiv
Surface meshes are a favoured domain for representing structural and functional information on the human cortex, but their complex topology and geometry pose significant challenges for deep learning analysis. While Transformers have excelled as domain-agnostic architectures for sequence-to-sequence learning, notably for structures where the translation of the convolution operation is non-trivial, the quadratic cost of the self-attention operation remains an obstacle for many dense prediction tasks. Inspired by some of the latest advances in hierarchical modelling with vision transformers, we introduce the Multiscale Surface Vision Transformer (MS-SiT) as a backbone architecture for surface deep learning. The self-attention mechanism is applied within local-mesh-windows to allow for high-resolution sampling of the underlying data, while a shifted-window strategy improves the sharing of information between windows. Neighbouring patches are successively merged, allowing the MS-SiT to learn hierarchical representations suitable for any prediction task. Results demonstrate that the MS-SiT outperforms existing surface deep learning methods for neonatal phenotyping prediction tasks using the Developing Human Connectome Project (dHCP) dataset. Furthermore, building the MS-SiT backbone into a U-shaped architecture for surface segmentation demonstrates competitive results on cortical parcellation using the UK Biobank (UKB) and manually-annotated MindBoggle datasets. Code and trained models are publicly available at https://github.com/metrics-lab/surface-vision-transformers.
Menten MJ, Holland R, Leingang O, et al., 2023, Exploring healthy retinal aging with deep learning, Ophthalmology Science, Vol: 3, Pages: 1-10, ISSN: 2666-9145
PurposeTo study the individual course of retinal changes caused by healthy aging using deep learning.DesignRetrospective analysis of a large data set of retinal OCT images.ParticipantsA total of 85 709 adults between the age of 40 and 75 years of whom OCT images were acquired in the scope of the UK Biobank population study.MethodsWe created a counterfactual generative adversarial network (GAN), a type of neural network that learns from cross-sectional, retrospective data. It then synthesizes high-resolution counterfactual OCT images and longitudinal time series. These counterfactuals allow visualization and analysis of hypothetical scenarios in which certain characteristics of the imaged subject, such as age or sex, are altered, whereas other attributes, crucially the subject’s identity and image acquisition settings, remain fixed.Main Outcome MeasuresUsing our counterfactual GAN, we investigated subject-specific changes in the retinal layer structure as a function of age and sex. In particular, we measured changes in the retinal nerve fiber layer (RNFL), combined ganglion cell layer plus inner plexiform layer (GCIPL), inner nuclear layer to the inner boundary of the retinal pigment epithelium (INL-RPE), and retinal pigment epithelium (RPE).ResultsOur counterfactual GAN is able to smoothly visualize the individual course of retinal aging. Across all counterfactual images, the RNFL, GCIPL, INL-RPE, and RPE changed by −0.1 μm ± 0.1 μm, −0.5 μm ± 0.2 μm, −0.2 μm ± 0.1 μm, and 0.1 μm ± 0.1 μm, respectively, per decade of age. These results agree well with previous studies based on the same cohort from the UK Biobank population study. Beyond population-wide average measures, our counterfactual GAN allows us to explore whether the retinal layers of a given eye will increase in thickness, decrease in thickness, or stagnate as a subject ages.ConclusionThis study demonstrates how counterfactual GANs
Hinterwimmer F, Consalvo S, Wilhelm N, et al., 2023, SAM-X: sorting algorithm for musculoskeletal x-ray radiography., Eur Radiol, Vol: 33, Pages: 1537-1544
OBJECTIVE: To develop a two-phased deep learning sorting algorithm for post-X-ray image acquisition in order to facilitate large musculoskeletal image datasets according to their anatomical entity. METHODS: In total, 42,608 unstructured and pseudonymized radiographs were retrieved from the PACS of a musculoskeletal tumor center. In phase 1, imaging data were sorted into 1000 clusters by a self-supervised model. A human-in-the-loop radiologist assigned weak, semantic labels to all clusters and clusters with the same label were merged. Three hundred thirty-two non-musculoskeletal clusters were discarded. In phase 2, the initial model was modified by "injecting" the identified labels into the self-supervised model to train a classifier. To provide statistical significance, data split and cross-validation were applied. The hold-out test set consisted of 50% external data. To gain insight into the model's predictions, Grad-CAMs were calculated. RESULTS: The self-supervised clustering resulted in a high normalized mutual information of 0.930. The expert radiologist identified 28 musculoskeletal clusters. The modified model achieved a classification accuracy of 96.2% and 96.6% for validation and hold-out test data for predicting the top class, respectively. When considering the top two predicted class labels, an accuracy of 99.7% and 99.6% was accomplished. Grad-CAMs as well as final cluster results underlined the robustness of the proposed method by showing that it focused on similar image regions a human would have considered for categorizing images. CONCLUSION: For efficient dataset building, we propose an accurate deep learning sorting algorithm for classifying radiographs according to their anatomical entity in the assessment of musculoskeletal diseases. KEY POINTS: • Classification of large radiograph datasets according to their anatomical entity. • Paramount importance of structuring vast amounts of retrospective data for modern deep learning app
Ma Q, Li L, Robinson EC, et al., 2023, CortexODE: learning cortical surface reconstruction by neural ODEs, IEEE Transactions on Medical Imaging, Vol: 42, Pages: 430-443, ISSN: 0278-0062
We present CortexODE, a deep learning framework for cortical surface reconstruction. CortexODE leverages neural ordinary differential equations (ODEs) to deform an input surface into a target shape by learning a diffeomorphic flow. The trajectories of the points on the surface are modeled as ODEs, where the derivatives of their coordinates are parameterized via a learnable Lipschitz-continuous deformation network. This provides theoretical guarantees for the prevention of self-intersections. CortexODE can be integrated to an automatic learning-based pipeline, which reconstructs cortical surfaces efficiently in less than 5 seconds. The pipeline utilizes a 3D U-Net to predict a white matter segmentation from brain Magnetic Resonance Imaging (MRI) scans, and further generates a signed distance function that represents an initial surface. Fast topology correction is introduced to guarantee homeomorphism to a sphere. Following the isosurface extraction step, two CortexODE models are trained to deform the initial surface to white matter and pial surfaces respectively. The proposed pipeline is evaluated on large-scale neuroimage datasets in various age groups including neonates (25-45 weeks), young adults (22-36 years) and elderly subjects (55-90 years). Our experiments demonstrate that the CortexODE-based pipeline can achieve less than 0.2mm average geometric error while being orders of magnitude faster compared to conventional processing pipelines.
Zimmer VA, Gomez A, Skelton E, et al., 2023, Placenta segmentation in ultrasound imaging: Addressing sources of uncertainty and limited field-of-view, Medical Image Analysis, Vol: 83, ISSN: 1361-8415
Automatic segmentation of the placenta in fetal ultrasound (US) is challenging due to the (i) high diversity of placenta appearance, (ii) the restricted quality in US resulting in highly variable reference annotations, and (iii) the limited field-of-view of US prohibiting whole placenta assessment at late gestation. In this work, we address these three challenges with a multi-task learning approach that combines the classification of placental location (e.g., anterior, posterior) and semantic placenta segmentation in a single convolutional neural network. Through the classification task the model can learn from larger and more diverse datasets while improving the accuracy of the segmentation task in particular in limited training set conditions. With this approach we investigate the variability in annotations from multiple raters and show that our automatic segmentations (Dice of 0.86 for anterior and 0.83 for posterior placentas) achieve human-level performance as compared to intra- and inter-observer variability. Lastly, our approach can deliver whole placenta segmentation using a multi-view US acquisition pipeline consisting of three stages: multi-probe image acquisition, image fusion and image segmentation. This results in high quality segmentation of larger structures such as the placenta in US with reduced image artifacts which are beyond the field-of-view of single probes.
Sideri-Lampretsa V, Zimmer VA, Qiu H, et al., 2023, MAD: Modality Agnostic Distance Measure for Image Registration, Pages: 147-156, ISSN: 0302-9743
Multi-modal image registration is a crucial pre-processing step in many medical applications. However, it is a challenging task due to the complex intensity relationships between different imaging modalities, which can result in large discrepancy in image appearance. The success of multi-modal image registration, whether it is conventional or learning based, is predicated upon the choice of an appropriate distance (or similarity) measure. Particularly, deep learning registration algorithms lack in accuracy or even fail completely when attempting to register data from an “unseen” modality. In this work, we present Modality Agnostic Distance (MAD), a deep image distance measure that utilises random convolutions to learn the inherent geometry of the images while being robust to large appearance changes. Random convolutions are geometry-preserving modules which we use to simulate an infinite number of synthetic modalities alleviating the need for aligned paired data during training. We can therefore train MAD on a mono-modal dataset and successfully apply it to a multi-modal dataset. We demonstrate that not only can MAD affinely register multi-modal images successfully, but it has also a larger capture range than traditional measures such as Mutual Information and Normalised Gradient Fields. Our code is available at: https://github.com/ModalityAgnosticDistance/MAD.
Mueller TT, Zhou S, Starck S, et al., 2023, Body Fat Estimation from Surface Meshes Using Graph Neural Networks, Pages: 105-117, ISSN: 0302-9743
Body fat volume and distribution can be a strong indication for a person’s overall health and the risk for developing diseases like type 2 diabetes and cardiovascular diseases. Frequently used measures for fat estimation are the body mass index (BMI), waist circumference, or the waist-hip-ratio. However, those are rather imprecise measures that do not allow for a discrimination between different types of fat or between fat and muscle tissue. The estimation of visceral (VAT) and abdominal subcutaneous (ASAT) adipose tissue volume has shown to be a more accurate measure for named risk factors. In this work, we show that triangulated body surface meshes can be used to accurately predict VAT and ASAT volumes using graph neural networks. Our methods achieve high performance while reducing training time and required resources compared to state-of-the-art convolutional neural networks in this area. We furthermore envision this method to be applicable to cheaper and easily accessible medical surface scans instead of expensive medical images.
Bani-Harouni D, Mueller TT, Rueckert D, et al., 2023, Gradient Self-alignment in Private Deep Learning, Pages: 89-97, ISSN: 0302-9743
Differential Privacy (DP) has become a gold-standard to preserve privacy in deep learning. Intuitively speaking, DP ensures that the output of a model is approximately invariant to the inclusion or exclusion of a single individual’s data from the training set. There is, however, a trade-off between privacy and utility. DP models tend to perform worse than non-DP models trained on the same data. This is caused by the clipping of per-sample gradients and the addition of noise required for DP guarantees causing an obfuscation of the individual data point’s contribution. In this work, we propose a method to reduce this discrepancy by improving the alignment between the per-sample gradients of each individual training sample with its non-DP gradient by increasing their cosine similarity. Optimizing the alignment in only a relevant subset of gradient dimensions, further improves the performance. We evaluate our method on CIFAR-10 and a pediatric pneumonia chest x-ray dataset.
Menten MJ, Paetzold JC, Zimmer VA, et al., 2023, A skeletonization algorithm for gradient-based optimization, Pages: 21337-21346, ISSN: 1550-5499
The skeleton of a digital image is a compact representation of its topology, geometry, and scale. It has utility in many computer vision applications, such as image description, segmentation, and registration. However, skeletonization has only seen limited use in contemporary deep learning solutions. Most existing skeletonization algorithms are not differentiable, making it impossible to integrate them with gradient-based optimization. Compatible algorithms based on morphological operations and neural networks have been proposed, but their results often deviate from the geometry and topology of the true medial axis. This work introduces the first three-dimensional skeletonization algorithm that is both compatible with gradient-based optimization and preserves an object's topology. Our method is exclusively based on matrix additions and multiplications, convolutional operations, basic non-linear functions, and sampling from a uniform probability distribution, allowing it to be easily implemented in any major deep learning library. In benchmarking experiments, we prove the advantages of our skeletonization algorithm compared to non-differentiable, morphological, and neural-network-based baselines. Finally, we demonstrate the utility of our algorithm by integrating it with two medical image processing applications that use gradient-based optimization: deep-learning-based blood vessel segmentation, and multimodal registration of the mandible in computed tomography and magnetic resonance images.
Prokopenko D, Hammernik K, Roberts T, et al., 2023, The Challenge of Fetal Cardiac MRI Reconstruction Using Deep Learning, Pages: 64-74, ISSN: 0302-9743
Dynamic free-breathing fetal cardiac MRI is one of the most challenging modalities, which requires high temporal and spatial resolution to depict rapid changes in a small fetal heart. The ability of deep learning methods to recover undersampled data could help to optimise the kt-SENSE acquisition strategy and improve non-gated kt-SENSE reconstruction quality. However, their application is limited by the lack of available fetal cardiac data. In this work, we explore supervised deep learning networks for reconstruction of kt-SENSE style acquired data using an extensive in vivo dataset. Having access to fully-sampled low-resolution multi-coil fetal cardiac MRI, we study the performance of the networks to recover fully-sampled data from undersampled data. We consider model architectures together with training strategies taking into account their application in the real clinical setup used to collect the dataset to enable networks to recover prospectively undersampled data. We explore a set of modifications to form a baseline performance evaluation for dynamic fetal cardiac MRI on real data. We systematically evaluate the models on coil-combined data to reveal the effect of the suggested changes to the architecture in the context of fetal heart properties. We show that the best-performing models recover a detailed depiction of the maternal anatomy on a large scale, but the dynamic properties of the fetal heart are under-represented. Training directly on multi-coil data improves the performance of the models, allows their prospective application to undersampled data and makes them outperform CTFNet introduced for adult cardiac cine MRI. However, these models deliver similar qualitative performances recovering the maternal body very well but underestimating the dynamic properties of fetal heart. This dynamic feature of fast change of fetal heart that is highly localised suggests both more targeted training and evaluation methods might be needed for fetal heart application.
Bercea CI, Puyol-Antón E, Wiestler B, et al., 2023, Bias in Unsupervised Anomaly Detection in Brain MRI, Pages: 122-131, ISSN: 0302-9743
Unsupervised anomaly detection methods offer a promising and flexible alternative to supervised approaches, holding the potential to revolutionize medical scan analysis and enhance diagnostic performance. In the current landscape, it is commonly assumed that differences between a test case and the training distribution are attributed solely to pathological conditions, implying that any disparity indicates an anomaly. However, the presence of other potential sources of distributional shift, including scanner, age, sex, or race, is frequently overlooked. These shifts can significantly impact the accuracy of the anomaly detection task. Prominent instances of such failures have sparked concerns regarding the bias, credibility, and fairness of anomaly detection. This work presents a novel analysis of biases in unsupervised anomaly detection. By examining potential non-pathological distributional shifts between the training and testing distributions, we shed light on the extent of these biases and their influence on anomaly detection results. Moreover, this study examines the algorithmic limitations that arise due to biases, providing valuable insights into the challenges encountered by anomaly detection algorithms in accurately capturing the variability in the normative distribution. Here, we specifically investigate Alzheimer’s disease detection from brain MR imaging as a case study, revealing significant biases related to sex, race, and scanner variations that substantially impact the results. These findings align with the broader goal of improving the reliability, fairness, and effectiveness of anomaly detection.
Liu S, Zhang B, Fang R, et al., 2023, Dynamic Graph Neural Representation Based Multi-modal Fusion Model for Cognitive Outcome Prediction in Stroke Cases, Pages: 338-347, ISSN: 0302-9743
The number of stroke patients is growing worldwide and half of them will suffer from cognitive impairment. Therefore, the prediction of Post-Stroke Cognitive Impairment (PSCI) becomes more and more important. However, the determinants and mechanisms of PSCI are still insufficiently understood, making this task challenging. In this paper, we propose a multi-modal graph fusion model to solve this task. First, dynamic graph neural representation is proposed to integrate multi-modal information, such as clinical data and image data, which separates them into node-level and global-level properties rather than processing them uniformly. Second, considering the variability of brain anatomy, a subject-specific undirected graph is constructed based on the connections among 131 brain anatomical regions segmented from image data, while first-order statistical features are extracted from each brain region and internal stroke lesions as node features. Meanwhile, a novel missing information compensation module is proposed to reduce the impact of missing or incomplete clinical data. In the dynamic graph neural representation, two kinds of attention mechanisms are used to encourage the model to automatically localize brain anatomical regions that are highly relevant to PSCI prediction. One is node attention established between global tabular neural representation and nodes, the other is multi-head graph self-attention which changes the static undirected graph into multiple dynamic directed graphs and optimizes the broadcasting process of the graph. The proposed method studies 418 stroke patients and achieves the best overall performance with a balanced accuracy score of 79.6% on PSCI prediction, outperforming the competing models. The code is publicly available at github.com/fightingkitty/MHGSA.
Müller P, Meissen F, Brandt J, et al., 2023, Anatomy-Driven Pathology Detection on Chest X-rays, Pages: 57-66, ISSN: 0302-9743
Pathology detection and delineation enables the automatic interpretation of medical scans such as chest X-rays while providing a high level of explainability to support radiologists in making informed decisions. However, annotating pathology bounding boxes is a time-consuming task such that large public datasets for this purpose are scarce. Current approaches thus use weakly supervised object detection to learn the (rough) localization of pathologies from image-level annotations, which is however limited in performance due to the lack of bounding box supervision. We therefore propose anatomy-driven pathology detection (ADPD), which uses easy-to-annotate bounding boxes of anatomical regions as proxies for pathologies. We study two training approaches: supervised training using anatomy-level pathology labels and multiple instance learning (MIL) with image-level pathology labels. Our results show that our anatomy-level training approach outperforms weakly supervised methods and fully supervised detection with limited training samples, and our MIL approach is competitive with both baseline approaches, therefore demonstrating the potential of our approach.
Bercea CI, Rueckert D, Schnabel JA, 2023, What Do AEs Learn? Challenging Common Assumptions in Unsupervised Anomaly Detection, Pages: 304-314, ISSN: 0302-9743
Detecting abnormal findings in medical images is a critical task that enables timely diagnoses, effective screening, and urgent case prioritization. Autoencoders (AEs) have emerged as a popular choice for anomaly detection and have achieved state-of-the-art (SOTA) performance in detecting pathology. However, their effectiveness is often hindered by the assumption that the learned manifold only contains information that is important for describing samples within the training distribution. In this work, we challenge this assumption and investigate what AEs actually learn when they are posed to solve anomaly detection tasks. We have found that standard, variational, and recent adversarial AEs are generally not well-suited for pathology detection tasks where the distributions of normal and abnormal strongly overlap. In this work, we propose MorphAEus, novel deformable AEs to produce pseudo-healthy reconstructions refined by estimated dense deformation fields. Our approach improves the learned representations, leading to more accurate reconstructions, reduced false positives and precise localization of pathology. We extensively validate our method on two public datasets and demonstrate SOTA performance in detecting pneumonia and COVID-19. Code: https://github.com/ci-ber/MorphAEus.
Pan J, Shit S, Turgut Ö, et al., 2023, Global k-Space Interpolation for Dynamic MRI Reconstruction Using Masked Image Modeling, Pages: 228-238, ISSN: 0302-9743
In dynamic Magnetic Resonance Imaging (MRI), k-space is typically undersampled due to limited scan time, resulting in aliasing artifacts in the image domain. Hence, dynamic MR reconstruction requires not only modeling spatial frequency components in the x and y directions of k-space but also considering temporal redundancy. Most previous works rely on image-domain regularizers (priors) to conduct MR reconstruction. In contrast, we focus on interpolating the undersampled k-space before obtaining images with Fourier transform. In this work, we connect masked image modeling with k-space interpolation and propose a novel Transformer-based k-space Global Interpolation Network, termed k-GIN. Our k-GIN learns global dependencies among low- and high-frequency components of 2D+t k-space and uses it to interpolate unsampled data. Further, we propose a novel k-space Iterative Refinement Module (k-IRM) to enhance the high-frequency components learning. We evaluate our approach on 92 in-house 2D+t cardiac MR subjects and compare it to MR reconstruction methods with image-domain regularizers. Experiments show that our proposed k-space interpolation method quantitatively and qualitatively outperforms baseline methods. Importantly, the proposed approach achieves substantially higher robustness and generalizability in cases of highly-undersampled MR data.
McGinnis J, Shit S, Li HB, et al., 2023, Single-subject Multi-contrast MRI Super-resolution via Implicit Neural Representations, Pages: 173-183, ISSN: 0302-9743
Clinical routine and retrospective cohorts commonly include multi-parametric Magnetic Resonance Imaging; however, they are mostly acquired in different anisotropic 2D views due to signal-to-noise-ratio and scan-time constraints. Thus acquired views suffer from poor out-of-plane resolution and affect downstream volumetric image analysis that typically requires isotropic 3D scans. Combining different views of multi-contrast scans into high-resolution isotropic 3D scans is challenging due to the lack of a large training cohort, which calls for a subject-specific framework. This work proposes a novel solution to this problem leveraging Implicit Neural Representations (INR). Our proposed INR jointly learns two different contrasts of complementary views in a continuous spatial function and benefits from exchanging anatomical information between them. Trained within minutes on a single commodity GPU, our model provides realistic super-resolution across different pairs of contrasts in our experiments with three datasets. Using Mutual Information (MI) as a metric, we find that our model converges to an optimum MI amongst sequences, achieving anatomically faithful reconstruction. (Code is available at: https://github.com/jqmcginnis/multi_contrast_inr/ ).
Prabhakar C, Li HB, Paetzold JC, et al., 2023, Self-pruning Graph Neural Network for Predicting Inflammatory Disease Activity in Multiple Sclerosis from Brain MR Images, Pages: 226-236, ISSN: 0302-9743
Multiple Sclerosis (MS) is a severe neurological disease characterized by inflammatory lesions in the central nervous system. Hence, predicting inflammatory disease activity is crucial for disease assessment and treatment. However, MS lesions can occur throughout the brain and vary in shape, size and total count among patients. The high variance in lesion load and locations makes it challenging for machine learning methods to learn a globally effective representation of whole-brain MRI scans to assess and predict disease. Technically it is non-trivial to incorporate essential biomarkers such as lesion load or spatial proximity. Our work represents the first attempt to utilize graph neural networks (GNN) to aggregate these biomarkers for a novel global representation. We propose a two-stage MS inflammatory disease activity prediction approach. First, a 3D segmentation network detects lesions, and a self-supervised algorithm extracts their image features. Second, the detected lesions are used to build a patient graph. The lesions act as nodes in the graph and are initialized with image features extracted in the first stage. Finally, the lesions are connected based on their spatial proximity and the inflammatory disease activity prediction is formulated as a graph classification task. Furthermore, we propose a self-pruning strategy to auto-select the most critical lesions for prediction. Our proposed method outperforms the existing baseline by a large margin (AUCs of 0.67 vs. 0.61 and 0.66 vs. 0.60 for one-year and two-year inflammatory disease activity, respectively). Finally, our proposed method enjoys inherent explainability by assigning an importance score to each lesion for the overall prediction. Code is available at https://github.com/chinmay5/ms_ida.git.
Sadafi A, Adonkina O, Khakzar A, et al., 2023, Pixel-Level Explanation of Multiple Instance Learning Models in Biomedical Single Cell Images, Pages: 170-182, ISSN: 0302-9743
Explainability is a key requirement for computer-aided diagnosis systems in clinical decision-making. Multiple instance learning with attention pooling provides instance-level explainability, however for many clinical applications a deeper, pixel-level explanation is desirable, but missing so far. In this work, we investigate the use of four attribution methods to explain a multiple instance learning models: GradCAM, Layer-Wise Relevance Propagation (LRP), Information Bottleneck Attribution (IBA), and InputIBA. With this collection of methods, we can derive pixel-level explanations on for the task of diagnosing blood cancer from patients’ blood smears. We study two datasets of acute myeloid leukemia with over 100 000 single cell images and observe how each attribution method performs on the multiple instance learning architecture focusing on different properties of the white blood single cells. Additionally, we compare attribution maps with the annotations of a medical expert to see how the model’s decision-making differs from the human standard. Our study addresses the challenge of implementing pixel-level explainability in multiple instance learning models and provides insights for clinicians to better understand and trust decisions from computer-aided diagnosis systems.
Huang W, Li HB, Pan J, et al., 2023, Neural Implicit k-Space for Binning-Free Non-Cartesian Cardiac MR Imaging, Pages: 548-560, ISSN: 0302-9743
In this work, we propose a novel image reconstruction framework that directly learns a neural implicit representation in k-space for ECG-triggered non-Cartesian Cardiac Magnetic Resonance Imaging (CMR). While existing methods bin acquired data from neighboring time points to reconstruct one phase of the cardiac motion, our framework allows for a continuous, binning-free, and subject-specific k-space representation. We assign a unique coordinate that consists of time, coil index, and frequency domain location to each sampled k-space point. We then learn the subject-specific mapping from these unique coordinates to k-space intensities using a multi-layer perceptron with frequency domain regularization. During inference, we obtain a complete k-space for Cartesian coordinates and an arbitrary temporal resolution. A simple inverse Fourier transform recovers the image, eliminating the need for density compensation and costly non-uniform Fourier transforms for non-Cartesian data. This novel imaging framework was tested on 42 radially sampled datasets from 6 subjects. The proposed method outperforms other techniques qualitatively and quantitatively using data from four and one heartbeat(s) and 30 cardiac phases. Our results for one heartbeat reconstruction of 50 cardiac phases show improved artifact removal and spatio-temporal resolution, leveraging the potential for real-time CMR. (Code available: https://github.com/wenqihuang/NIK_MRI ).
Ezhov I, Giannoni L, Shit S, et al., 2023, Identifying chromophore fingerprints of brain tumor tissue on hyperspectral imaging using principal component analysis, ISSN: 0277-786X
Hyperspectral imaging (HSI) is an optical technique that processes the electromagnetic spectrum at a multitude of monochromatic, adjacent frequency bands. The wide-bandwidth spectral signature of a target object's reflectance allows fingerprinting its physical, biochemical, and physiological properties. HSI has been applied for various applications, such as remote sensing and biological tissue analysis. Recently, HSI was also used to differentiate between healthy and pathological tissue under operative conditions in a surgery room on patients diagnosed with brain tumors. In this article, we perform a statistical analysis of the brain tumor patients' HSI scans from the HELICoiD dataset with the aim of identifying the correlation between reflectance spectra and absorption spectra of tissue chromophores. By using the principal component analysis (PCA), we determine the most relevant spectral features for intra- and inter-tissue class differentiation. Furthermore, we demonstrate that such spectral features are correlated with the spectra of cytochrome, i.e., the chromophore highly involved in (hyper) metabolic processes. Identifying such fingerprints of chromophores in reflectance spectra is a key step for automated molecular profiling and, eventually, expert-free biomarker discovery.
Meissen F, Paetzold J, Kaissis G, et al., 2023, Unsupervised Anomaly Localization with Structural Feature-Autoencoders, Pages: 14-24, ISSN: 0302-9743
Unsupervised Anomaly Detection has become a popular method to detect pathologies in medical images as it does not require supervision or labels for training. Most commonly, the anomaly detection model generates a “normal” version of an input image, and the pixel-wise lp -difference of the two is used to localize anomalies. However, large residuals often occur due to imperfect reconstruction of the complex anatomical structures present in most medical images. This method also fails to detect anomalies that are not characterized by large intensity differences to the surrounding tissue. We propose to tackle this problem using a feature-mapping function that transforms the input intensity images into a space with multiple channels where anomalies can be detected along different discriminative feature maps extracted from the original image. We then train an Autoencoder model in this space using structural similarity loss that does not only consider differences in intensity but also in contrast and structure. Our method significantly increases performance on two medical data sets for brain MRI. Code and experiments are available at https://github.com/FeliMe/feature-autoencoder.
Stolt-Ansó N, McGinnis J, Pan J, et al., 2023, NISF: Neural Implicit Segmentation Functions, Pages: 734-744, ISBN: 9783031439001
Segmentation of anatomical shapes from medical images has taken an important role in the automation of clinical measurements. While typical deep-learning segmentation approaches are performed on discrete voxels, the underlying objects being analysed exist in a real-valued continuous space. Approaches that rely on convolutional neural networks (CNNs) are limited to grid-like inputs and not easily applicable to sparse or partial measurements. We propose a novel family of image segmentation models that tackle many of CNNs’ shortcomings: Neural Implicit Segmentation Functions (NISF). Our framework takes inspiration from the field of neural implicit functions where a network learns a mapping from a real-valued coordinate-space to a shape representation. NISFs have the ability to segment anatomical shapes in high-dimensional continuous spaces. Training is not limited to voxelized grids, and covers applications with sparse and partial data. Interpolation between observations is learnt naturally in the training procedure and requires no post-processing. Furthermore, NISFs allow the leveraging of learnt shape priors to make predictions for regions outside of the original image plane. We go on to show the framework achieves dice scores of $$0.87 \pm 0.045$$ on a (3D+t) short-axis cardiac segmentation task using the UK Biobank dataset. We also provide a qualitative analysis on our frameworks ability to perform segmentation and image interpolation on unseen regions of an image volume at arbitrary resolutions.
Kreitner L, Ezhov I, Rueckert D, et al., 2023, Automated Analysis of Diabetic Retinopathy Using Vessel Segmentation Maps as Inductive Bias, Pages: 16-25, ISSN: 0302-9743
Recent studies suggest that early stages of diabetic retinopathy (DR) can be diagnosed by monitoring vascular changes in the deep vascular complex. In this work, we investigate a novel method for automated DR grading based on ultra-wide optical coherence tomography angiography (UW-OCTA) images. Our work combines OCTA scans with their vessel segmentations, which then serve as inputs to task specific networks for lesion segmentation, image quality assessment and DR grading. For this, we generate synthetic OCTA images to train a segmentation network that can be directly applied on real OCTA data. We test our approach on MICCAI 2022’s DR analysis challenge (DRAC). In our experiments, the proposed method performs equally well as the baseline model.
Emre T, Oghbaie M, Chakravarty A, et al., 2023, Pretrained Deep 2.5D Models for Efficient Predictive Modeling from Retinal OCT: A PINNACLE Study Report, Pages: 132-141, ISSN: 0302-9743
In the field of medical imaging, 3D deep learning models play a crucial role in building powerful predictive models of disease progression. However, the size of these models presents significant challenges, both in terms of computational resources and data requirements. Moreover, achieving high-quality pretraining of 3D models proves to be even more challenging. To address these issues, hybrid 2.5D approaches provide an effective solution for utilizing 3D volumetric data efficiently using 2D models. Combining 2D and 3D techniques offers a promising avenue for optimizing performance while minimizing memory requirements. In this paper, we explore 2.5D architectures based on a combination of convolutional neural networks (CNNs), long short-term memory (LSTM), and Transformers. In addition, leveraging the benefits of recent non-contrastive pretraining approaches in 2D, we enhanced the performance and data efficiency of 2.5D techniques even further. We demonstrate the effectiveness of architectures and associated pretraining on a task of predicting progression to wet age-related macular degeneration (AMD) within a six-month period on two large longitudinal OCT datasets.
Rueckert D, Knolle M, Duchateau N, et al., 2023, Diagnosis, AI and Big Data in Cardiology: a Practical Guide, Pages: 85-103, ISBN: 9783031050701
This chapter covers the clinical application of diagnosis of cardiovascular disease. A clinical opinion piece discusses the current clinical standard for diagnosis tasks and its limitations. The technical review summarizes the classical machine learning pipeline for medical diagnosis as well as some common types of traditional machine learning models that have been used for this application. Following this, some relevant deep learning architectures for computer-aided diagnosis are discussed. Some example applications of artificial intelligence based automated diagnosis are introduced and the key challenges highlighted. The practical tutorial deals with a simple diagnosis task based on characteristics derived from cardiac MR segmentations and other patient characteristics. The chapter closes with a clinical opinion piece that speculates on the future role of AI in cardiac diagnosis.
Li L, Ma Q, Ouyang C, et al., 2023, Robust Segmentation via Topology Violation Detection and Feature Synthesis, Pages: 67-77, ISSN: 0302-9743
Despite recent progress of deep learning-based medical image segmentation techniques, fully automatic results often fail to meet clinically acceptable accuracy, especially when topological constraints should be observed, e.g., closed surfaces. Although modern image segmentation methods show promising results when evaluated based on conventional metrics such as the Dice score or Intersection-over-Union, these metrics do not reflect the correctness of a segmentation in terms of a required topological genus. Existing approaches estimate and constrain the topological structure via persistent homology (PH). However, these methods are not computationally efficient as calculating PH is not differentiable. To overcome this problem, we propose a novel approach for topological constraints based on the multi-scale Euler Characteristic (EC). To mitigate computational complexity, we propose a fast formulation for the EC that can inform the learning process of arbitrary segmentation networks via topological violation maps. Topological performance is further facilitated through a corrective convolutional network block. Our experiments on two datasets show that our method can significantly improve topological correctness.
Ma Q, Li L, Kyriakopoulou V, et al., 2023, Conditional Temporal Attention Networks for Neonatal Cortical Surface Reconstruction, Pages: 312-322, ISSN: 0302-9743
Cortical surface reconstruction plays a fundamental role in modeling the rapid brain development during the perinatal period. In this work, we propose Conditional Temporal Attention Network (CoTAN), a fast end-to-end framework for diffeomorphic neonatal cortical surface reconstruction. CoTAN predicts multi-resolution stationary velocity fields (SVF) from neonatal brain magnetic resonance images (MRI). Instead of integrating multiple SVFs, CoTAN introduces attention mechanisms to learn a conditional time-varying velocity field (CTVF) by computing the weighted sum of all SVFs at each integration step. The importance of each SVF, which is estimated by learned attention maps, is conditioned on the age of the neonates and varies with the time step of integration. The proposed CTVF defines a diffeomorphic surface deformation, which reduces mesh self-intersection errors effectively. It only requires 0.21 s to deform an initial template mesh to cortical white matter and pial surfaces for each brain hemisphere. CoTAN is validated on the Developing Human Connectome Project (dHCP) dataset with 877 3D brain MR images acquired from preterm and term born neonates. Compared to state-of-the-art baselines, CoTAN achieves superior performance with only 0.12 ± 0.03 mm geometric error and 0.07 ± 0.03% self-intersecting faces. The visualization of our attention maps illustrates that CoTAN indeed learns coarse-to-fine surface deformations automatically without intermediate supervision.
Holland R, Leingang O, Holmes C, et al., 2023, Clustering Disease Trajectories in Contrastive Feature Space for Biomarker Proposal in Age-Related Macular Degeneration, Pages: 724-734, ISSN: 0302-9743
Age-related macular degeneration (AMD) is the leading cause of blindness in the elderly. Current grading systems based on imaging biomarkers only coarsely group disease stages into broad categories that lack prognostic value for future disease progression. It is widely believed that this is due to their focus on a single point in time, disregarding the dynamic nature of the disease. In this work, we present the first method to automatically propose biomarkers that capture temporal dynamics of disease progression. Our method represents patient time series as trajectories in a latent feature space built with contrastive learning. Then, individual trajectories are partitioned into atomic sub-sequences that encode transitions between disease states. These are clustered using a newly introduced distance metric. In quantitative experiments we found our method yields temporal biomarkers that are predictive of conversion to late AMD. Furthermore, these clusters were highly interpretable to ophthalmologists who confirmed that many of the clusters represent dynamics that have previously been linked to the progression of AMD, even though they are currently not included in any clinical grading system.
Dima AF, Zimmer VA, Menten MJ, et al., 2023, 3D Arterial Segmentation via Single 2D Projections and Depth Supervision in Contrast-Enhanced CT Images, Pages: 141-151, ISSN: 0302-9743
Automated segmentation of the blood vessels in 3D volumes is an essential step for the quantitative diagnosis and treatment of many vascular diseases. 3D vessel segmentation is being actively investigated in existing works, mostly in deep learning approaches. However, training 3D deep networks requires large amounts of manual 3D annotations from experts, which are laborious to obtain. This is especially the case for 3D vessel segmentation, as vessels are sparse yet spread out over many slices and disconnected when visualized in 2D slices. In this work, we propose a novel method to segment the 3D peripancreatic arteries solely from one annotated 2D projection per training image with depth supervision. We perform extensive experiments on the segmentation of peripancreatic arteries on 3D contrast-enhanced CT images and demonstrate how well we capture the rich depth information from 2D projections. We demonstrate that by annotating a single, randomly chosen projection for each training sample, we obtain comparable performance to annotating multiple 2D projections, thereby reducing the annotation effort. Furthermore, by mapping the 2D labels to the 3D space using depth information and incorporating this into training, we almost close the performance gap between 3D supervision and 2D supervision. Our code is available at: https://github.com/alinafdima/3Dseg-mip-depth.
This data is extracted from the Web of Science and reproduced under a licence from Thomson Reuters. You may not copy or re-distribute this data in whole or in part without the written consent of the Science business of Thomson Reuters.