PI: Kemar Green (School of Medicine)
Co-I: Vishal Patel
Eye movement pathways are affected in various conditions owing to the widespread brain areas represented by these neural connections. Changes in the function of various eye movement types can provide accurate information about the location of the brain abnormality, and in the right context, the neurologic diagnosis. Eye movements are therefore beneficial in providing physiologic data for various neurologic illnesses.
While the pattern of eye movements in various neurologic and psychiatric disorders is well described, some of these diseases are very rare, and the existence of robust eye movement disease datasets are equally rare or non-existent – making it difficult to develop reliable deep learning classification systems. In addition to data scarcity, data security also poses a challenge in the use of eye movement video data, as ocular features as well as eye movements bear biometric information. A possible solution to the data scarcity issues is the use of synthetic data. Synthetic eye tracking videos has been simulated for non-medical purposes, and have been shown to mimic human ocular motor dynamics.
We have shown in previous studies that image recognition models can successfully detect torsional eye movements from artificially generated retinal photos. It has also been shown that subtle abnormal eye movements can identified via clinician-guided telemedicine evaluation and with deep learning video classifiers from videooculography recordings of varying qualities. Even though the technology exists for robust video generation, and synthetic medical data usage in artificial intelligence (AI) have been shown to increase model performance metrics.
We hypothesize that novel generative AI techniques can be adopted for generating synthetic neurologic disease specific eye movement datasets that can then be used to develop deep learning-based brain disease phenotypic system.
During the funding cycle, we will focus on eye movement signatures of myasthenia gravis (MG), a rare neurologic disease, that causes muscle fatigue with repetitive muscle use. Several eye movement types (smooth pursuit, saccades, OKN, etc.) have been implicated in MG4. During the funding cycle, we will focus on developing of optokinetic nystagmus (OKN) eye movement signatures based on a recently published method. First, however, due to the rarity the disease and available eye movement datasets, coupled with privacy concerns surrounding the use data containing biometric information, it is necessary to augment the available clinical data (being collected simultaneously from neurologic patients) by generating realistic synthetic eye movement datasets of myasthenia gravis using a pose-guided video generation framework. In this framework, the pose will be obtained from 1-5% of the collected patient waveform data, and the videos will be generated from validated open-source eye movement databanks. Our methodology leverages the efficacy of video diffusion models, traditionally employed for generating high quality, short videos (Fig. 1). Our initial phase employs a model proficient in generating videos from segmented eye movement masks. At the heart of this process is a latent video diffusion mechanism that translates segmented mask inputs into visual sequences.
We will then develop multimodal myasthenia gravis deep learning classifiers using imaging and waveform data (Fig.1). Multimodal data (raw videos, filtered images and extracted OKN time series data) will be used to build a fusion model by adopting existing methods. The approach is based on long short-term memory (LSTM) network that is capable of processing video clips and time-series data representing OKN. The multi-modal representation will be generated as a sequence of unimodal representations (or tokens), such that the fusion module aggregates these representations through the recurrence mechanism of LSTM. The use of multimodal data in our case will allow for better understanding of the ocular motor biomarkers of OKN from videos and waveforms. The data being extracted from the OKN fatigue tasks (video, filtered image, waveform, etc.,) have different signatures for MG that individually might not provide the most accurate diagnosis; therefore, the combination of different data types using the multimodal deep learning approach will allow for the development of the most robust model. Furthermore, the multimodal data approach will address the potential noise that can occur in the eye movement data, as well as the variability of the eye movement signs in MG. The models will be validated on the remaining 95-99% of the real MG data as well as a commensurate value of normal patients to assess the generalizability of the synthetic model. Various explainable AI methods will be applied to assess the model’s prediction and unveil any novel “non”physiologic. If successful, we will apply similar methods to curate similar datasets for various brain diseases such as Parkinson’s disease, Alzheimer’s disease, Huntington’s disease, progressive supranuclear palsy (PSP), multiple system atrophy (MSA), etc.
The IDIES-supported project has made significant strides in overcoming the challenges of data scarcity and privacy concerns in eye movement research, particularly around spontaneous eye movement classification and Myasthenia Gravis diagnosis from optokinetic nystagmus (OKN) eye movement (Figure 1). Developing deep learning models for detecting and characterizing these specific eye movements has long been hindered by the limited availability of publicly accessible data, as eye movement patterns can uniquely identify individuals.To address this, we leverage a diffusion probabilistic framework for generating synthetic, patient-independent, pose-guided eye movement videos.
This framework simulates clinically relevant eye movement patterns by generating long, realistic videos that mimic various waveforms (Figure 2). These synthetic videos are produced without the use of real patient data, utilizing publicly available datasets as a foundation. This approach allows us to create extensive datasets that maintain privacy while providing a rich source of data for training and validating AI models.
The effectiveness of these generated videos has been validated through their application in downstream tasks, using real patient datasets for comparison. Results demonstrate that models trained on these synthetic videos perform comparably to those trained on real data, proving the viability of synthetic datasets for eye movement research. This breakthrough enables us to scale our research efforts while ensuring patient privacy is uncompromised.
In addition to analyzing existing eye movement patterns, we are expanding the range of eye movement types covered in our research. Our aim is to develop disease-specific biomarkers for each type, leveraging synthetic data to create targeted classifiers for a range of neurologic conditions (Figure 3).
Looking ahead, our goal is to make these synthetic eye movement videos, disease-specific biomarkers, and classifiers available to the public research community, encouraging further exploration and analysis in the field. Additionally, our next steps involve integrating these synthetic datasets into a wearable device capable of real-time monitoring and therapy adjustments, bridging the gap between research and clinical application.
