Biomedical Image Computing

Introduction: Open vocabulary video semantic segmentation (OV-VSS) aims to assign a semantic label to each pixel of each frame of the video given an arbitrary set of open-vocabulary category names. There are a number of attempts on open vocabulary image semantic segmentation (OV-ISS). However, OV-VSS does not get enough attention due to the difficulty of video understanding tasks in modeling local redundancy and global correlation. In this master thesis project, we plan to fill the gap by extending existing OV-ISS methods to OV-VSS. Specifically, we aim to develop a OV-VSS method which achieves high accuracy by using temporal information and keeps high efficiency.

Goal:

1. Familiarize with OV-ISS and OV-VSS.
2. Adopt existing video semantic segmentation methods to OV-ISS.
3. Propose an algorithm for OV-VSS.
4. Possibility of a submission to top AI conferences such as NeurIPS 2024 and ICLR 2024.
   

Requirement: Familiar with Python and Pytorch. Prior experience with computer vision, e.g. take computer vision courses at ETH Zurich. Knowledge in image/video semantic segmentation is a plus.


Supervisor: Dr. Guolei Sun, .ch;
Dr. Yawei Li,

Professor: Ender Konukoglu

References:
[1] Guolei Sun, et. al., “Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation”, ECCV 2022
[2] Guolei Sun, et. al., “Coarse-to-Fine Feature Mining for Video Semantic Segmentation”, ECCV 2022
[3] Feng Liang, et. al., “Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP”, CVPR 2023

Introduction:
In statistical learning, understanding the models' bias-variance trade-off is crucial, particularly under specific assumptions. This concept is vital from a domain generalization standpoint, as it relates to the divergence between source and target distributions. In the field of medical imaging, Castro et al. [1] have highlighted this by using a causal diagram (see Fig. 5) to illustrate medical image generation, which informs the divergence and consequently the bias-variance trade-off for an optimal statistical model.
Building on Castro et al.'s framework and the principles of the Shepp-Logan phantom [2], our project aims to develop a mechanism for generating toy medical image data. This mechanism will allow us to freely define and manipulate certain assumptions, thereby enabling fast and effective assessment of our models. Familiarity with Python and PyTorch will be beneficial, as they form the basis of our development and assessment platform.

References:
[1] Castro, D.C., Walker, I. & Glocker, B. Causality matters in medical imaging. Nat Commun 11, 3673 (2020). https://doi.org/10.1038/s41467-020-17478-w

[2] L. A. Shepp and B. F. Logan, "The Fourier reconstruction of a head section," in IEEE Transactions on Nuclear Science, vol. 21, no. 3, pp. 21-43, June 1974, doi: 10.1109/TNS.1974.6499235.  

Supervisor:
Güney Tombak,
 

Professor: Ender Konukoglu,

Master Thesis in Machine Learning for NMR Spectroscopy:  

Background and Motivation:
Nuclear Magnetic Resonance (NMR) spectroscopy plays a vital role in molecular structure analysis, particularly for proteins. The technique's effectiveness largely depends on the strength of the external magnetic field. High-field NMR spectrometers, often operating above 600 MHz, offer better data clarity but come with high technological and financial costs. Our initial work contributed to this area by exploring deep learning to enhance lower-field NMR spectra, aiming to approach the quality of high-field data. This approach has the potential to make high-field NMR more accessible and cost-effective. The following research phase seeks to refine these methods and explore their practical applications, hoping to broaden the toolset available for scientific investigation.

Project Description:
This master thesis, "Virtual Magnetic Field Enhancement in NMR Spectroscopy," invites you to engage in this evolving research area. The project focuses on several key objectives:

1. Model Development: Adapting, training, and testing machine learning models for various NMR pulse sequences, including NOESY, and scaling these models to handle larger datasets.
2. Bridging Theory and Practice: Transferring insights from synthetic to experimental data is a crucial step in testing the feasibility and accuracy of our approach.
3. Practical Application Testing: Demonstrating how the developed algorithm can be applied in real-world scenarios to enhance the utility of NMR spectroscopy potentially. Practical examples could involve testing enhanced spectra with established NMR data analysis workflows, such as ARTINA and NMRtist (nmrtist.org).
4. Prediction Quality Optimization: Investigating the best compromise in prediction quality between low and high-field NMR spectra, which is key to ensuring the practicality of our enhancements.
5. Exploration of Diffusion Models: Experiment with emerging diffusion models to explore new possibilities in NMR spectrum analysis.
   

Your Role and Impact:
By participating in this project, you will contribute to an ongoing effort to advance NMR spectroscopy. Your work will help explore how machine learning can open new doors in scientific research, potentially making high-field NMR data more accessible. This project is an opportunity to challenge current technological limits and contribute to a field with broad implications for molecular biology and drug development.

We invite you to join this journey in advancing NMR spectroscopy and contributing to a meaningful scientific endeavour.

Supervisor:
Nicolas Schmid,

Professor: Ender Konukoglu, ETF E113,
 

Introduction:
Recent developments in the field of computer vision have highlighted the growing prominence of foundation models, particularly those like DINOv2 [1] and Segment-Anything [2], which have achieved impressive outcomes in processing natural images. Yet, the effectiveness of these models in medical imaging remains somewhat ambiguous. This project intends to bridge this gap by rigorously examining various training methodologies for these models. Our goal is to explore the most effective approaches to adapt these advanced foundation models for medical imaging, thereby enhancing their utility and potential impact in healthcare and medical research.

References:

[1] external pagehttps://dinov2.metademolab.com/call_made
[2] external pagehttps://segment-anything.com/call_made  

Supervisors:
Anna Susmelj,
Ertunc Erdil, ertunc.erdil@vision.ee.ethz.ch
 

Professor: Ender Konukoglu,

Introduction:
In contemporary computer vision applications, neural networks have demonstrated exceptional proficiency in semantic segmentation tasks across diverse application domains. However, these models are susceptible to significant performance degradation in real-world scenarios due to distributional disparities. This remains a prevalent concern in safety-critical domains such as autonomous driving and medical imaging. Recent research emphasizes the critical role of network architecture selection in addressing challenges related to domain generalization[1,2]. Specifically, transformer architectures have exhibited notably superior generalization capabilities, particularly in autonomous driving contexts. Conversely, classical convolutional networks like the widely adopted UNet continue to dominate the landscape in medical imaging applications [3]. This study aims to delve into the impact of network architecture in medical imaging contexts. Our objective is to identify a versatile architectural paradigm applicable to a wide spectrum of computer vision tasks, ranging from autonomous driving to medical imaging.

References:

[1] external pagehttps://openaccess.thecvf.com/content/CVPR2022/html/Hoyer_DAFormer_Improving_Network_Architectures_and_Training_Strategies_for_Domain-Adaptive_Semantic_CVPR_2022_paper.htmlcall_made

[2] external pagehttps://openaccess.thecvf.com/content/CVPR2023/papers/Hoyer_MIC_Masked_Image_Consistency_for_Context-Enhanced_Domain_Adaptation_CVPR_2023_paper.pdfcall_made

[3] external pagehttps://arxiv.org/abs/2004.04668call_made
 

Supervisors:
Anna Susmelj,
Lukas Hoyer,
 

Professor: Ender Konukoglu,

Introduction:
The application of prior knowledge, in the form of exemplary shapes extracted from segmented volumetric medical images, has emerged as an appealing approach for reconstructing anatomical shapes from limited or incomplete measurements [1]. However, in certain medical contexts, essential anatomical structures might be absent in these segmentations, despite their critical role in clinical diagnostics. In this project, we aim to utilize a synthetic dataset from a rigid anatomical atlas in order to use it as a strong prior on anatomical shape variations via a combination of conditional diffusion models [2, 3] as a generative model and an implicit function as smoothness prior [4].

Requirements:

• Programming knowledge of Python.
• Familiarity with PyTorch
• Good mathematical background

References:

[1] external pagehttps://openreview.net/forum?id=UuHtdwRXkzwcall_made
[2] external pagehttps://arxiv.org/abs/2111.05826call_made
[3] external pagehttps://arxiv.org/abs/2302.05543call_made
[4] external pagehttps://arxiv.org/pdf/2303.12865.pdfcall_made
 

Supervisors:
Anna Susmelj,
Kyriakos Flouris,
 

Professor: Ender Konukoglu,

This project aims to integrate the principles of classical generative adversarial networks (GANs) with those of quantum generative adversarial networks (QGANs) to generate realistic, high-dimensional image data. Drawing inspiration from the foundational studies in [1], [2], and [3], the objective is to assess the potential of synthesizing images using near-term quantum devices, for example the noisy intermediate-scale quantum (NISQ) devices. The main aspect of the project will be deploying the methodologies from [1], [2], and [3] to train models on high-quality open-source datasets and/or investicating the capability of QGANs equipped with quantum circuit generators to produce images without resorting to dimensionality reduction or classical data processing techniques.

References:
[1] H.-L. Huang, Y. Du, M. Gong, Y. Zhao, Y. Wu, C. Wang, S. Li, F. Liang, J. Lin, Y. Xu, R. Yang, T. Liu, M.-H. Hsieh, H. Deng, H. Rong, C.-Z. Peng, C.-Y. Lu, Y.-A. Chen, D. Tao, X. Zhu, and J.-W. Pan, “Experimental quantum generative adversarial networks for image generation,” Phys. Rev. Appl., vol. 16, p. 024051, 2021.

[2] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved training of Wasserstein GANs,” 2017, arXiv:1704.00028v3.

[3] S. Lok Tsang, M. T. West, S. M. Erfani and M. Usman, “Hybrid Quantum-Classical Generative Adversarial Network for High Resolution Image Generation,” 2023, arXiv:2212:11614v2.

Supervisor:
Kyriakos Flouris,

Professor: Ender Konukoglu, ETF E113,
 

This project aims to design an adaptable encoder model that leverages various medical imaging datasets. The model will be trained utilizing self-supervised and contrastive learning methods such as SimCLR [1] and masked autoencoders [2], emphasizing versatility and high performance to serve multiple applications in the medical imaging sector.

The project offers an opportunity to experiment with different machine learning paradigms, improve model performance, and tackle unique challenges presented by medical image datasets. The objective is to create a robust encoder model that can effectively serve as a backbone for a variety of tasks in medical imaging. Prerequisites for this project include a solid understanding of deep learning and prior experience with PyTorch framework.

References:

[1] Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020, November). A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.

[2] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16000-16009).

Supervisors:
Güney Tombak ()
Ertunc Erdil ()

Professor: Ender Konukoglu, ETF E113,

The standard approach for estimating hemodynamics parameters involves running CFD simulations on patient-specific models extracted from medical images. Personalization of these models can be performed by integrating further data from MRA and Flow MRI. In this project we aim to estimate hemodynamics parameters from flow and anatomical MRI, which can be routinely acquired in clinical practice. The flow information and the geometry will be combined together in a computational mesh and will be processed using Graph Convolutional Neural Networks (GCNN) or other deep learning method. A motivated student will explore integration of the CFD model into the deep network construction. The proposed models will be trained using existing synthetic MRI datasets.

Supervisors:
Kyriakos Flouris ()

Professor: Ender Konukoglu, ETF E113,

Reducing the time of magnetic resonance imaging (MRI) data acquisition is a long standing goal. Shorter acquisition times would come with many benefits, e.g. higher patient comfort or enabling dynamic imaging (e.g. the moving heart). Ultimately it can lead to higher clinical throughput, which reduces the cost of MRI for one individual and will make MRI more widely accessible.
One possible avenue towards this goal is to under-sample the acquision and incorporate prior knowledge to solve the resulting ill-posed recosntruction problem. This strategy has received much attention and many different methods have been proposed.
In this project we aim to understand performance differences between the different methods and analyse which components make them work. We will implement State-of-the-art reconstruction methods and perform experiments to judge their performance and robustness properties.

Depending on student's interests the project can have a different focus:

• Supervised Methods [1]
• Unsupervised Methods [2], [3], [4]
• Untrained Methods [5]

References:

[1]: external pagehttps://onlinelibrary.wiley.com/doi/10.1002/mrm.28827call_made
[2]: external pagehttps://ieeexplore.ieee.org/document/8579232call_made
[3]: external pagehttps://ieeexplore.ieee.org/document/9695412call_made
[4]: external pagehttps://link.springer.com/chapter/10.1007/978-3-031-16446-0_62call_made
[5]: external pagehttps://arxiv.org/abs/2111.10892call_made

Supervisors:
Georg Brunner ()
Emiljo Mehillaj ()

Professor: Ender Konukoglu, ETF E113,

Domain shift poses a significant challenge in medical image analysis, as the performance of deep learning models can degrade when encountering out-of-distribution (OOD) samples. Domain generalization aims to address this issue by training models that are robust to domain shifts. Recent research, such as the SIMPLE [1] method introduced in ICLR 2023, has explored the idea of using a pool of pretrained models to achieve higher domain generalization without fine-tuning. In this project, the student will investigate the potential of ensemble methods and pretrained model pools for domain generalization in medical image analysis. The primary goal is to build upon the ideas presented in the SIMPLE [1] paper and the Modular Deep Learning review [2], extending their applications to medical image datasets.

References:

[1] Li, Z., Ren, K., Jiang, X., Shen, Y., Zhang, H., & Li, D. (2023). SIMPLE: Specialized Model-Sample Matching for Domain Generalization. In The Eleventh International Conference on Learning Representations.

[2] Pfeiffer, J., Ruder, S., Vulić, I., & Ponti, E. M. (2023). Modular deep learning. arXiv preprint arXiv:2302.11529.

Supervisors:
Guney Tombak ()
Ertunc Erdil ()

Professor: Ender Konukoglu, ETF E113,