Multimodal Learning for Contextually-Aware Longitudinal PET/CT image analysis

PROJECT SUMMARY
18F-Fluorodeoxyglucose (FDG) PET/CT imaging has become an essential tool for guiding and adapting
treatments for lymphoma. However, the PET evaluation criteria currently used for assessing lymphoma, which
consists of subjective visual scoring on a 5-point scale, is suboptimal. The visual scores suffer from high inter-
observer variability and have low prognostic power for new emerging biological therapies. Quantitative PET
metrics have been shown to be more predictive of clinical outcomes than visual scores, but quantitative analysis
of whole-body PET/CT images is prohibitively time-consuming and impractical in routine clinical care.
Deep learning (DL) has shown promise in automating the quantitative analysis of baseline FDG PET/CT images,
but comprehensive evaluation of interim-therapy and post-therapy images using DL has proven difficult. Residual
lymphoma has low-level uptake, which can be hard to differentiate from physiologic or treatment-related uptake,
and reading physicians must use clinical histories and baseline PET images (i.e., sites of initial disease) to make
reliable diagnoses. DL algorithms, on the other hand, only operate on cross-sectional images and are unable to
account for historical context.
Our objective is to develop DL algorithms that operate on PET/CT images from more than one time point so that
algorithms can learn longitudinal dependencies for contextually-aware predictions. We also aim to develop
multimodal vision-language models that can simultaneously interpret radiology text reports while performing
PET/CT image analysis. These models can leverage critical information about patient history and physician
interpretation when processing retrospective images. Furthermore, we will use semi-supervised learning to
leverage both unlabeled datasets and labeled datasets. Our overall goal is to develop contextually-aware
algorithms for automated longitudinal analysis of whole-body PET/CT images in lymphoma. These tools will be
developed using diverse datasets from multiple institutions. PET metrics measured by DL will be validated as
predictive markers of outcome using data from a Phase 3 clinical trial.

Grant Number: 5R01EB033782-02
NIH Institute/Center: NIH
Principal Investigator: Tyler Bradshaw