Biomedical Image Computing

This project aims to create a benchmark of current detection networks for anatomy detection/segmentation and potentially improve these for intracranial surgery.

Supervisor: Gary Sarwin,

Professor: Ender Konukoglu

This project aims to use large language models to predict the acceptance of academic papers by combining textual and contextual information. Leveraging text and figures from manuscripts alongside metadata (e.g., venue/journal), the project seeks to train a model that can evaluate the likelihood of acceptance based on established criteria such as relevance, novelty, clarity, and methodological soundness.

Supervisor: Gary Sarwin,

Professor: Ender Konukoglu

This project explores the potential of Vision-Language Models (VLMs) to predict surgical phases solely based on visual inputs from surgical videos, leveraging pre-existing knowledge without the need for labeled data or additional training. By inputting surgical procedure videos into a VLM, the project aims to assess the model’s ability to recognize and sequence phases of surgery based on its general visual-linguistic understanding of typical surgical steps and objects.

The project hypothesizes that VLMs can inherently identify relevant surgical contexts and transitions due to their extensive pre-training on diverse multimodal datasets. Through this work, we seek to demonstrate the feasibility of using VLMs as zero-shot predictors in specialized medical tasks, with potential applications in intraoperative decision support.

Supervisor: Gary Sarwin,

Professor: Ender Konukoglu

Description for the Master project

One promising method for cardiovascylar disease prediction involves analyzing retinal images, as the retinal vasculature provides insights into cardiovascular health, including stroke risk. Researchers can use retinal images to assess overall health, and in previous work, a framework was developed combining graph-based retinal image representations with clinical data to enhance stroke prediction using a contrastive self-supervised model. The aim of this project is to extend our previous work. In particular, we aim to make predictions interpretable, which is crucial in the context of clinically relevant machine learning models. Further, we aim to cope with incomplete data, improve our model’s architecture by proposing state-of-the-art graph encoders, and potentially fine-tune ro build upon recent foundation models. Lastly, we aim to explore additional downstream tasks for cardiovascular diseases with potentially new modalities like brain images that can be paired with retinal fundus image graph representations. To summarize, the focus lies on exploring the use of retinal fundus image graph representations in a contrastive learning framework. In-depth literature research will be expected. The aim should also be to contribute to a scientific publication.

Your qualifications / what we are looking for

1. Knowledge of common machine learning paradigms and architectures (graph neural networks, self-supervised learning, etc.)
2. Knowledge of explainable and interpretable AI is advantageous
3. Excellent programming skills in Python as well as familiarity with PyTorch (and PyTorch geometric)
4. Full time commitment towards the completion of your project
5. Ability to work independently on challenging projects
6. Ability to understand scientific papers and conduct literature research
   
7. Prior medical knowledge is advantageous

How to apply
Please send your CV and transcript to Neda Davoudi () and Bastian Wittmann ().
Links to previous work (e.g., your GitHub profile) are highly appreciated.

Co-Supervisors: Neda Davoudi, Bastian Wittmann, and Bjoern Menze

Professor: Ender Konukoglu
 

References:

[1] “Retinal vasculature of different diameters and plexuses exhibit distinct vulnerability in varying severity of diabetic retinopathy” (external page https://www.nature.com/articles/s41433-024-03021-4)
[2] “Geometric deep learning for disease classification in OCTA images” (external page https://iovs.arvojournals.org/article.aspx?articleid=2790851)
[3] “A foundation model for generalizable disease detection from retinal images”(external page https://www.nature.com/articles/s41586-023-06555-x)
[4] “Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data” (external page https://arxiv.org/abs/2303.14080)
[5] “GNNExplainer: Generating Explanations for Graph Neural Networks” (external page https://arxiv.org/abs/1903.03894)
[6] “TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data” (external page https://arxiv.org/abs/2407.07582)
[7] “Link prediction for flow-driven spatial networks” (external page https://arxiv.org/abs/2303.14501)
[8] "The emerging role of combined brain/heart magnetic resonance imaging for the evaluation of brain/heart interaction in heart failure" (external page https://www.mdpi.com/2077-0383/11/14/4009)
[9] Ferrando SB et al. Stroke and Retinal microvascular changes: Neuroimaging markers of brain damage and association with retinal Optical Coherence Tomography Angiography parameters.

Background: Foundation models are a breakthrough in the field of artificial intelligence. These models are characterized by massive size, reaching billions (even trillions) of parameters, and by the ability to be adapted to a wide variety of tasks without needing to be trained from scratch. The development of these models marks a pivotal shift in AI research and application, pushing the boundaries of what machines can understand and do. However, due to the huge size of foundation model, they are very demanding in terms of computation, memory footprint and bandwidth. For this reason, foundation models face significant computational challenges. These models are typically trained on massive clusters equipped with thousands of advanced GPUs. Moreover, they require cloud services for inference as well.

Aim: Quantization is an effective technique to reduce the stored model size of foundation models and accelerate their inference. For example, a 70B Llama model approximately needs 150GB GPU memory with 16-bit floating point weights, which requires two A100 80G GPUs for inference. If the model is quantized to 4 bits, the required GPU memory can be reduced to 35GB and allows the model to fit on a single GPU with less memory. Thus, in this project, we aim to further release the power of quantization, shrinking the vision and language foundation model size and accelerating their inference. We will consider vision and language foundation models such as SAM2 and LLAVA.

Methods: The project target mixed-precision quantization methods within a quantization-aware training framework, specifically adapted to the BitNet training and QLoRA finetuning process. Mixed-precision quantization allows for different parts of the model to be quantized at different levels, enabling a tailored approach where critical components of the model can retain higher precision to pre-serve essential information and model integrity, while less critical components can be quantized more aggressively to achieve greater reductions in memory usage and computational demand. The motivation behind using mixed-precision quantization lies in its potential to find Pareto-optimal points in the trade-offs between model size, computational efficiency, and performance accuracy. By selectively applying different quantization strategies across the model, it becomes possible to maintain or even enhance the model's effectiveness while still benefiting from the efficiency gains of lower-bit quantization.

Materials and Resources: The candidate will join the research team with extensive experience in machine learning, computer vision. The candidate will have the opportunity to work with active researchers and to be supervised by world-leading professors and senior researchers. Access to high-performance supercomputers equipped with rich GPU resource will be possible.

Nature of the Thesis:
review: 10%; Model building: 70%; Model validation: 10%; Results analysis: 10%

Requirements:
Familiar with Python, Pytorch;
Knowledge in machine learning, deep learning;
Experience with training deep learning models;
Knowledge with Transformers and Mamba is a bonus;
Knowledge with Pytorch-Lightning is a bonus.

Co-Supervisors:
Dr. Yawei Li ()
Dr. Guolei Sun ()

Professors:
Prof. Luca Benini (), main supervisor
Prof. Ender Konukoglu ()

Institutes:
Integrated System Lab & Computer Vision Lab, D-ITET, ETH Zurich
 

References:

[1] Pandey, Nilesh Prasad, et al. "A practical mixed precision algorithm for post-training quantization." arXiv preprint arXiv:2302.05397 (2023).

[2] Van Baalen, Mart, et al. "Bayesian bits: Unifying quantization and pruning." Advances in neural information processing systems 33 (2020): 5741-5752.

[3] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).

[4] Wang, Hongyu, et al. "Bitnet: Scaling 1-bit transformers for large language models." arXiv preprint arXiv:2310.11453 (2023).

[5] Ma, Shuming, et al. "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits." arXiv preprint arXiv:2402.17764 (2024).  

Introduction: In recent years, large models trained on extensive datasets with substantial GPU resources have demonstrated remarkable generalizability. These models excel across various domains, including novel ones not represented in their training data. Within the realm of computer vision, there are two primary types of large models: vision-language models and vision foundation models. Vision-language models are trained on paired vision and language inputs, while vision foundation models are primarily trained using visual information such as images and videos. Although these models are designed for general purposes, they may not perform optimally for specific tasks like few-shot segmentation.

Few-shot segmentation aims to segment arbitrary categories given a few support samples with ground-truth masks for those classes. This approach significantly reduces annotation efforts and has wide-ranging applications, including medical image analysis and autonomous driving. While some studies suggest that large models can segment novel classes without additional support samples, this ability has not been systematically studied. Moreover, the visual information from support samples can complement the knowledge embedded in current large models. Consequently, adapting large models for few-shot semantic segmentation by incorporating additional visual information holds great promise.

The goal of the project is to explore large models for few-shot semantic segmentation. First, a rigorous study on existing models’ ability under current few-shot segmentation settings will be conducted. Second, novel algorithms built upon those models will be developed to further enhance segmentation performance on new classes.

Requirement: Familiar with Python and Pytorch. Prior experience with computer vision, e.g. take computer vision courses at ETH Zurich. Knowledge in image/video semantic segmentation is a plus.

References:
[1] Lanyun Zhu, et. al., “LLaFS: When Large Language Models Meet Few-Shot Segmentation”, CVPR 2024
[2] Alec Radford, et. al., “Learning Transferable Visual Models From Natural Language Supervision”, ICML 2021

Supervisors: Dr. Guolei Sun, ;
Dr. Yawei Li, .ch

Professor: Ender Konukoglu

Introduction: Digital twins of transportation networks require accurate and detailed information, such as the number of lanes and turning relations, which are crucial for effective transport modeling. This project aims to enhance road network data from OpenStreetMap (OSM) using satellite images to fill in missing or incomplete information. Specifically, the goal is to develop a machine learning (ML) program to infer road network features from satellite images. The project will follow these steps:

Define Data Requirements: Determine the necessary resolution and detail of OSM data, focusing on critical features like the number of lanes and turning relations.

Overlap and Georeference Data: Align OSM data with satellite images, ensuring consistent georeferencing for accurate data integration. Extract Labels from OSM: Identify and extract labels (number of lanes, turning relations) from OSM data to use as training data the ML model.

Train the ML Model: Develop and train an ML model using satellite images and extracted OSM labels to predict road features.
Validate the Model: Implement a validation framework to evaluate the ML model's performance against the ground truth and refine the model to improve accuracy.

Notes: This project is in collaboration with ETH spin-off Transcality (https://transcality.com/). Please find more information on their website for what they are doing.

Requirement: Familiar with Python and Pytorch. Prior experience with computer vision, e.g. take computer vision courses at ETH Zurich. Knowledge in image classification is a plus.

Supervisor: Dr. Guolei Sun, ;
Dr. Lukas Ambühl,

References:
[1] Syed Waqas Zamir, et. al., Isaid: A large-scale dataset for instance segmentation in aerial images, CVPRW 2019
[2] Yun Liu, et. al., Transformer in convolutional neural networks, MIR 2024

Professor: Ender Konukoglu, ETF E113,

Introduction: Video semantic segmentation (VSS) is a pivotal task within the domain of computer vision, aiming to assign predefined categories to each frame of a video sequence. The challenge lies not only in achieving accuracy but also in maintaining temporal consistency, minimizing unwanted flickering artifacts across consecutive frames. This pursuit of stability and precision is further compounded by the contemporary landscape of computational constraints and the burgeoning demand for real-time processing capabilities, particularly on edge devices. Against this backdrop, binarization emerges as a compelling avenue for optimization, offering a pathway to significant reductions in both computational complexity and memory footprint by harnessing the power of 1-bit parameters and bitwise operations. In the forthcoming master's thesis project, our ambition is to explore the adaptability of binarization techniques within the realm of VSS, with a view towards improving efficiency without compromising segmentation performance.

Goal:

• Familiarize with VSS and binarization.
• Adopt binarization to existing VSS methods.
• Design novel algorithms for efficient VSS.
• Possibility of a submission to top AI conferences such as ICLR 2024 and CVPR 2024.
  

Requirement: Familiar with Python and Pytorch. Prior experience with computer vision, e.g. take computer vision courses at ETH Zurich. Knowledge in image/video semantic segmentation is a plus.

Supervisors: Dr. Guolei Sun, ;
Dr. Yawei Li, .ch

Professor: Ender Konukoglu

References:
[1] Guolei Sun, et. al., “Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation”, ECCV 2022
[2] Haotong Qin, et. al., “BiMatting: Efficient Video Matting via Binarization”, NeurIPS 2023
[3] Amir Gholami, et. al., “A Survey of Quantization Methods for Efficient Neural Network Inference”

This project aims to revolutionize the analysis of electroencephalography (EEG) data by developing a specialized foundational model utilizing the principles of artificial intelligence. Despite the critical role of EEG in diagnosing and treating neurological disorders, challenges such as low signal-to-noise ratios and complex signal patterns hinder practical analysis. By adapting strategies from successful domains like natural language processing and computer vision, this project will build a machine learning model tailored for EEG signals. The model will undergo extensive pre-training on diverse EEG datasets to establish a robust understanding of neural activities, followed by fine-tuning for specific clinical tasks such as seizure detection and sleep stage classification. Our approach promises to enhance the accuracy, efficiency, and accessibility of EEG diagnostics, paving the way for improved patient outcomes. Validation and testing using standard performance metrics will measure the model's efficacy, setting a new standard in EEG analysis.

Keywords: EEG Analysis, Foundational Models, Large Language Models, Machine Learning, Deep Learning, Transfer Learning, Signal Processing

Description
Electroencephalography (EEG) is a fundamental tool in neuroscience, allowing us to monitor the brain's electrical activity non-invasively. It is beneficial for diagnosing and treating neurological disorders, but interpreting EEG data can be tricky due to many factors, such as low signal-to-noise ratio and the inherent complexity of EEG signals.
Large Language Models (LLMs), such as OpenAI’s GPT series, are a specific type of Foundational Model designed to understand, generate, and manipulate human language. LLMs are trained on extensive collections of text data, allowing them to learn a wide range of language patterns and nuances. This training enables them to perform various language-related tasks, from simple text generation to more complex applications like summarization, translation, and answering questions across multiple domains.

Foundational Models, a broader category, include LLMs and models designed for other data types, such as images, audio, and time-series. The common thread among all Foundational Models is their training approach: they are typically pre-trained on a large, diverse set of data to develop a broad understanding of a particular type of input, whether text, visual content, or EEG signals. After this extensive pre-training phase, these models are fine-tuned on more specific datasets or tasks to adapt their capabilities to more specialized applications.
Foundation models, known for their extensive pre-training on large datasets before being fine-tuned for specific tasks, have dramatically altered the landscape in natural language processing and computer vision. Nevertheless, their application in interpreting the complexities inherent in EEG data is still developing. This project proposes to develop a sophisticated foundational model specifically for EEG analysis, harnessing the capabilities of deep learning and AI.

Our project aims to make EEG analysis more accessible and accurate using the power of artificial intelligence. We plan to build a specialized foundational model for EEG data in this project.

Here is what we are going to do:

1. Dataset Compilation: We will gather various open-source EEG datasets. We will focus on ensuring the data is diverse, covering different demographics, conditions, and ways the data was collected.
2. Model Architecture Design: Next, we will develop a neural network architecture tailored for EEG data. We’ll take inspiration from successful models in other areas and adapt them to meet our needs.
3. Pretraining: We will train our model using the datasets we've compiled. We'll use transfer and semi-supervised learning to make the most of our data.
4. Fine-Tuning: We will fine-tune our model using a smaller, more specific set of data for tasks like detecting seizures or classifying sleep stages.
5. Validation and Testing: Finally, we will test our model to see how well it performs. We'll use metrics like accuracy, precision, recall, and the F1-score to evaluate and compare it to existing methods.

Requirements:

• Strong programming skills in Python
• Machine Learning background (courses etc.)

Related literature

• Jiang, Wei-Bang, Li-Ming Zhao, and Bao-Liang "Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI" ICLR 2024
• Wang, Christopher, et al. "BrainBERT: Self-supervised representation learning for intracranial recordings." ICLR 2023.
• Cui, Wenhui, et al. "Neuro-gpt: Developing a foundation model for eeg." arXiv preprint arXiv:2311.03764 (2023)
• Chen, Yuqi, et al. "EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model." arXiv preprint arXiv:2401.10278 (2024).

Goal

Our goal is to make EEG analysis not only better but also easier for researchers and clinicians. This project is about bringing practical, effective AI solutions into the world of neuroscience to help improve how we understand and treat the human brain.

Contact Details

Please include your CV and transcript in the submission.

Thorir Mar Ingolfsson: external page https://thorirmar.com;
Yawei Li: https://yaweili.bitbucket.ioexternal page /;
Xiaying Wang: external page https://xiaywang.github.io/;

Professors
Luca Benini, Ender Konukoglu
 

Introduction: Open vocabulary video semantic segmentation (OV-VSS) aims to assign a semantic label to each pixel of each frame of the video given an arbitrary set of open-vocabulary category names. There are a number of attempts on open vocabulary image semantic segmentation (OV-ISS). However, OV-VSS does not get enough attention due to the difficulty of video understanding tasks in modeling local redundancy and global correlation. In this master thesis project, we plan to fill the gap by extending existing OV-ISS methods to OV-VSS. Specifically, we aim to develop a OV-VSS method which achieves high accuracy by using temporal information and keeps high efficiency.

Goal:

1. Familiarize with OV-ISS and OV-VSS.
2. Adopt existing video semantic segmentation methods to OV-ISS.
3. Propose an algorithm for OV-VSS.
4. Possibility of a submission to top AI conferences such as NeurIPS 2024 and ICLR 2024.
   

Requirement: Familiar with Python and Pytorch. Prior experience with computer vision, e.g. take computer vision courses at ETH Zurich. Knowledge in image/video semantic segmentation is a plus.


Supervisor: Dr. Guolei Sun, .ch;
Dr. Yawei Li,

Professor: Ender Konukoglu

References:
[1] Guolei Sun, et. al., “Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation”, ECCV 2022
[2] Guolei Sun, et. al., “Coarse-to-Fine Feature Mining for Video Semantic Segmentation”, ECCV 2022
[3] Feng Liang, et. al., “Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP”, CVPR 2023

Introduction:
In statistical learning, understanding the models' bias-variance trade-off is crucial, particularly under specific assumptions. This concept is vital from a domain generalization standpoint, as it relates to the divergence between source and target distributions. In the field of medical imaging, Castro et al. [1] have highlighted this by using a causal diagram (see Fig. 5) to illustrate medical image generation, which informs the divergence and consequently the bias-variance trade-off for an optimal statistical model.
Building on Castro et al.'s framework and the principles of the Shepp-Logan phantom [2], our project aims to develop a mechanism for generating toy medical image data. This mechanism will allow us to freely define and manipulate certain assumptions, thereby enabling fast and effective assessment of our models. Familiarity with Python and PyTorch will be beneficial, as they form the basis of our development and assessment platform.

References:
[1] Castro, D.C., Walker, I. & Glocker, B. Causality matters in medical imaging. Nat Commun 11, 3673 (2020). external page https://doi.org/10.1038/s41467-020-17478-w

[2] L. A. Shepp and B. F. Logan, "The Fourier reconstruction of a head section," in IEEE Transactions on Nuclear Science, vol. 21, no. 3, pp. 21-43, June 1974, doi: 10.1109/TNS.1974.6499235.  

Supervisor:
Güney Tombak,
 

Professor: Ender Konukoglu,

Introduction:
Recent developments in the field of computer vision have highlighted the growing prominence of foundation models, particularly those like DINOv2 [1] and Segment-Anything [2], which have achieved impressive outcomes in processing natural images. Yet, the effectiveness of these models in medical imaging remains somewhat ambiguous. This project intends to bridge this gap by rigorously examining various training methodologies for these models. Our goal is to explore the most effective approaches to adapt these advanced foundation models for medical imaging, thereby enhancing their utility and potential impact in healthcare and medical research.

References:

[1] https://dinov2.metademolab.com/
[2] https://segment-anything.com/  

Supervisors:
Ertunc Erdil,
 

Professor: Ender Konukoglu,

This project aims to design an adaptable encoder model that leverages various medical imaging datasets. The model will be trained utilizing self-supervised and contrastive learning methods such as SimCLR [1] and masked autoencoders [2], emphasizing versatility and high performance to serve multiple applications in the medical imaging sector.

The project offers an opportunity to experiment with different machine learning paradigms, improve model performance, and tackle unique challenges presented by medical image datasets. The objective is to create a robust encoder model that can effectively serve as a backbone for a variety of tasks in medical imaging. Prerequisites for this project include a solid understanding of deep learning and prior experience with PyTorch framework.

References:

[1] Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020, November). A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.

[2] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16000-16009).

Supervisors:
Güney Tombak ()
Ertunc Erdil ()

Professor: Ender Konukoglu, ETF E113,

Reducing the time of magnetic resonance imaging (MRI) data acquisition is a long standing goal. Shorter acquisition times would come with many benefits, e.g. higher patient comfort or enabling dynamic imaging (e.g. the moving heart). Ultimately it can lead to higher clinical throughput, which reduces the cost of MRI for one individual and will make MRI more widely accessible.
One possible avenue towards this goal is to under-sample the acquision and incorporate prior knowledge to solve the resulting ill-posed recosntruction problem. This strategy has received much attention and many different methods have been proposed.
In this project we aim to understand performance differences between the different methods and analyse which components make them work. We will implement State-of-the-art reconstruction methods and perform experiments to judge their performance and robustness properties.

Depending on student's interests the project can have a different focus:

• Supervised Methods [1]
• Unsupervised Methods [2], [3], [4]
• Untrained Methods [5]

References:

[1]: external page https://onlinelibrary.wiley.com/doi/10.1002/mrm.28827
[2]: external page https://ieeexplore.ieee.org/document/8579232
[3]: external page https://ieeexplore.ieee.org/document/9695412
[4]: external page https://link.springer.com/chapter/10.1007/978-3-031-16446-0_62
[5]: external page https://arxiv.org/abs/2111.10892

Supervisors:
Georg Brunner ()
Emiljo Mehillaj ()

Professor: Ender Konukoglu, ETF E113,