Multimodal learning is one of the central tasks of artificial intelligence (AI), which aims to effectively
fuse and model multimodal data to gain a better understanding of the world around us. Many multimodal fusion
strategies have been proposed, ranging from manually designed policies to advanced automated machine learning
(AutoML)-based approaches. Although AutoML-based solutions outperform handcrafted ones, they are still far
from optimal due to their lack of generalizability in model design and failure to account for the unique
characteristics of multimodal data. This project takes the multimodal healthcare predictive modeling task
as a representative example, aiming to discover and identify the optimal way to fuse multimodal data via a
new learning paradigm, i.e., automated multimodal learning, with minimal human interventions.
The success of this project will yield new fundamental knowledge in various fields, including automated
machine learning, multimodal deep learning, and healthcare predictive modeling. The new automated multimodal
learning paradigm will revolutionize multimodal data mining by automatically searching for new and complex
yet optimal fusion strategies from the data, potentially motivating researchers and domain experts to
understand the multimodal data better. In addition, recognizing unique research challenges posed by
the unique nature of multimodal data in the healthcare domain and providing customized solutions will
advance the research of healthcare predictive modeling significantly.
To meet these goals, the investigator proposes to equip automated multimodal learning with the ability
to model the unique challenges of multimodal health data, including data size variety, noise, and
missing modalities. The investigator also proposes to validate the proposed research for different
multimodal fusion tasks in healthcare informatics and beyond and gather feedback from experts to
refine the proposed research. The results of this project will provide a needed paradigm shift
toward automated multimodal data fusion, impacting a broad range of research fields, including
machine learning, data mining, and healthcare informatics. The proposed research will also make
an enduring contribution to multimodal predictive modeling in clinical practice and other domains.
The generated data, source codes, and software tools will be made available to researchers worldwide.
The open platform will expedite research, enhance global collaborations in this field, and provide
longstanding value for academia, healthcare organizations, and health industries. The proposed education
plan will help to ensure that graduates are well equipped to design and evaluate machine learning
solutions and cultivate K-12 students' interest in computer science and informatics. It will also lead
to a more diverse population of undergraduate research assistants and enhance collaboration and networking
among graduate students.
Students
Publications
Softwares