National Science Foundation (NSF)

CAREER: Automated Multimodal Learning for Healthcare

Multimodal learning is one of the central tasks of artificial intelligence (AI), which aims to effectively fuse and model multimodal data to gain a better understanding of the world around us. Many multimodal fusion strategies have been proposed, ranging from manually designed policies to advanced automated machine learning (AutoML)-based approaches. Although AutoML-based solutions outperform handcrafted ones, they are still far from optimal due to their lack of generalizability in model design and failure to account for the unique characteristics of multimodal data. This project takes the multimodal healthcare predictive modeling task as a representative example, aiming to discover and identify the optimal way to fuse multimodal data via a new learning paradigm, i.e., automated multimodal learning, with minimal human interventions. The success of this project will yield new fundamental knowledge in various fields, including automated machine learning, multimodal deep learning, and healthcare predictive modeling. The new automated multimodal learning paradigm will revolutionize multimodal data mining by automatically searching for new and complex yet optimal fusion strategies from the data, potentially motivating researchers and domain experts to understand the multimodal data better. In addition, recognizing unique research challenges posed by the unique nature of multimodal data in the healthcare domain and providing customized solutions will advance the research of healthcare predictive modeling significantly.
To meet these goals, the investigator proposes to equip automated multimodal learning with the ability to model the unique challenges of multimodal health data, including data size variety, noise, and missing modalities. The investigator also proposes to validate the proposed research for different multimodal fusion tasks in healthcare informatics and beyond and gather feedback from experts to refine the proposed research. The results of this project will provide a needed paradigm shift toward automated multimodal data fusion, impacting a broad range of research fields, including machine learning, data mining, and healthcare informatics. The proposed research will also make an enduring contribution to multimodal predictive modeling in clinical practice and other domains. The generated data, source codes, and software tools will be made available to researchers worldwide. The open platform will expedite research, enhance global collaborations in this field, and provide longstanding value for academia, healthcare organizations, and health industries. The proposed education plan will help to ensure that graduates are well equipped to design and evaluate machine learning solutions and cultivate K-12 students' interest in computer science and informatics. It will also lead to a more diverse population of undergraduate research assistants and enhance collaboration and networking among graduate students.

Students

  • Aofei Chang (Penn State)
  • Xiaochen Wang (Penn State)
  • Yuan Zhong (Penn State)

  • Publications

  • Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictions
    Suhan Cui, Jiaqi Wang, Yuan Zhong, Han Liu, Ting Wang and Fenglong Ma
    Proceedings of the 24th SIAM International Conference on Data Mining (SDM 2024), accepted. (acceptance rate: 29.2%)
  • MedDiffusion: Boosting Health Risk Prediction via Diffusion-based Data Augmentation
    Yuan Zhong, Suhan Cui, Jiaqi Wang, Ziyi Yin, Yaqing Wang, Houping Xiao, Mengdi Huai, Ting Wang and Fenglong Ma
    Proceedings of the 24th SIAM International Conference on Data Mining (SDM 2024), accepted. (acceptance rate: 29.2%)
  • Hierarchical Pretraining on Multimodal Electronic Health Records
    Xiaochen Wang, Junyu Luo, Jiaqi Wang, Ziyi Yin, Suhan Cui, Yuan Zhong, Yaqing Wang and Fenglong Ma
    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), December 6-10, 2023, Singapore, accepted.
  • ClinicalRisk: A New Therapy-related Clinical Trial Dataset for Predicting Risk Levels and Risk Factors
    Junyu Luo, Zhi Qiao, Lucas Glass, Cao Xiao and Fenglong Ma
    Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM 2023), Oct. 21-25, 2023, Birmingham, UK, accepted.
  • pADR: Towards Personalized Adverse Drug Reaction Prediction
    Junyu Luo, Cheng Qian, Xiaochen Wang, Lucas Glass and Fenglong Ma
    Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM 2023), Oct. 21-25, 2023, Birmingham, UK, accepted.

  • Softwares

  • AutoFM (SDM 2024)
  • MedDiffusion (SDM 2024)
  • MedNMP (EMNLP 2023)