National Science Foundation (NSF)

CAREER: Automated Multimodal Learning for Healthcare

Multimodal learning is one of the central tasks of artificial intelligence (AI), which aims to effectively fuse and model multimodal data to gain a better understanding of the world around us. Many multimodal fusion strategies have been proposed, ranging from manually designed policies to advanced automated machine learning (AutoML)-based approaches. Although AutoML-based solutions outperform handcrafted ones, they are still far from optimal due to their lack of generalizability in model design and failure to account for the unique characteristics of multimodal data. This project takes the multimodal healthcare predictive modeling task as a representative example, aiming to discover and identify the optimal way to fuse multimodal data via a new learning paradigm, i.e., automated multimodal learning, with minimal human interventions. The success of this project will yield new fundamental knowledge in various fields, including automated machine learning, multimodal deep learning, and healthcare predictive modeling. The new automated multimodal learning paradigm will revolutionize multimodal data mining by automatically searching for new and complex yet optimal fusion strategies from the data, potentially motivating researchers and domain experts to understand the multimodal data better. In addition, recognizing unique research challenges posed by the unique nature of multimodal data in the healthcare domain and providing customized solutions will advance the research of healthcare predictive modeling significantly.
To meet these goals, the investigator proposes to equip automated multimodal learning with the ability to model the unique challenges of multimodal health data, including data size variety, noise, and missing modalities. The investigator also proposes to validate the proposed research for different multimodal fusion tasks in healthcare informatics and beyond and gather feedback from experts to refine the proposed research. The results of this project will provide a needed paradigm shift toward automated multimodal data fusion, impacting a broad range of research fields, including machine learning, data mining, and healthcare informatics. The proposed research will also make an enduring contribution to multimodal predictive modeling in clinical practice and other domains. The generated data, source codes, and software tools will be made available to researchers worldwide. The open platform will expedite research, enhance global collaborations in this field, and provide longstanding value for academia, healthcare organizations, and health industries. The proposed education plan will help to ensure that graduates are well equipped to design and evaluate machine learning solutions and cultivate K-12 students' interest in computer science and informatics. It will also lead to a more diverse population of undergraduate research assistants and enhance collaboration and networking among graduate students.

Students

  • Aofei Chang (Penn State)
  • Xiaochen Wang (Penn State)
  • Yuan Zhong (Penn State)

    • Publications

  • Asymmetrical Reciprocity-based Federated Learning for Resolving Disparities in Medical Diagnosis
    Jiaqi Wang*, Ziyi Yin*, Quanzeng You, Lingjuan Lyu, and Fenglong Ma
    Proceedings of the 31st SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2025), August 3 - 7, 2025, Toronto, ON, Canada. (* equal contribution, August cycle research track acceptance rate: 19%)
  • Multimodal Artificial Intelligence in Healthcare
    Jiaqi Wang, Xiaochen Wang, Yuan Zhong, Ziyi Yin, Aofei Chang, Cao Xiao, and Fenglong Ma
    Conference Tutorial at the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI 2025), February 25 - March 4, 2025, Philadelphia, Pennsylvania, USA.
  • FedMeKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injection
    Jiaqi Wang*, Xiaochen Wang*, Lingjuan Lyu, Jinghui Chen and Fenglong Ma
    Proceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurPIS 2024), December 9-15, 2024, Vancouver, Canada. (Spotlight, * equal contribution, datasets and benchmarks track acceptance rate: 25.3%)
  • Asymmetric Mutual Learning for Decentralized Federated Medical Imaging
    Jiaqi Wang, Houping Xiao and Fenglong Ma
    Proceedings of the 15th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB 2024), November 22-25, 2024, Shenzhen, China. (Oral acceptance rate: 17.6%)
  • FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models
    Xiaochen Wang*, Jiaqi Wang*, Houping Xiao, Jinghui Chen and Fenglong Ma
    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024 Main), November 12 –16, Miami, Florida. (* equal contribution)
  • BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models
    Aofei Chang, Jiaqi Wang, Han Liu, Parminder Bhatia, Cao Xiao, Ting Wang and Fenglong Ma
    Findings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024 Findings), November 12 –16, Miami, Florida.
  • Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models
    Yuan Zhong, Xiaochen Wang, Jiaqi Wang, Xiaokun Zhang, Yaqing Wang, Mengdi Huai, Cao Xiao and Fenglong Ma
    Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2024), Aug. 25-29, 2024, Barcelona, Spain.
  • Unity in Diversity: Collaborative Pre-training Across Multimodal Medical Sources
    Xiaochen Wang, Junyu Luo, Jiaqi Wang, Yuan Zhong, Xiaokun Zhang, Yaqing Wang, Parminder Bhatia, Cao Xiao and Fenglong Ma
    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), August 11-16, 2024, Bangkok, Thailand.
  • Recent Advances in Predictive Modeling with Electronic Health Records[Paper][Slides]
    Jiaqi Wang, Junyu Luo, Muchao Ye, Xiaochen Wang, Yuan Zhong, Aofei Chang, Guanjie Huang, Ziyi Yin, Cao Xiao, Jimeng Sun and Fenglong Ma
    Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI 2024), Aug. 3-9, 2024, Jeju, South Korea. (Survey Track, acceptance rate: 20.7%)
  • Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictions
    Suhan Cui, Jiaqi Wang, Yuan Zhong, Han Liu, Ting Wang and Fenglong Ma
    Proceedings of the 24th SIAM International Conference on Data Mining (SDM 2024), accepted. (acceptance rate: 29.2%)
  • MedDiffusion: Boosting Health Risk Prediction via Diffusion-based Data Augmentation
    Yuan Zhong, Suhan Cui, Jiaqi Wang, Ziyi Yin, Yaqing Wang, Houping Xiao, Mengdi Huai, Ting Wang and Fenglong Ma
    Proceedings of the 24th SIAM International Conference on Data Mining (SDM 2024), accepted. (acceptance rate: 29.2%)
  • Hierarchical Pretraining on Multimodal Electronic Health Records
    Xiaochen Wang, Junyu Luo, Jiaqi Wang, Ziyi Yin, Suhan Cui, Yuan Zhong, Yaqing Wang and Fenglong Ma
    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), December 6-10, 2023, Singapore, accepted.
  • ClinicalRisk: A New Therapy-related Clinical Trial Dataset for Predicting Risk Levels and Risk Factors
    Junyu Luo, Zhi Qiao, Lucas Glass, Cao Xiao and Fenglong Ma
    Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM 2023), Oct. 21-25, 2023, Birmingham, UK, accepted.
  • pADR: Towards Personalized Adverse Drug Reaction Prediction
    Junyu Luo, Cheng Qian, Xiaochen Wang, Lucas Glass and Fenglong Ma
    Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM 2023), Oct. 21-25, 2023, Birmingham, UK, accepted.

    • Dissemination

  • AAAI 2025 Tutorial [Link]
  • IJCAI 2024 AI4TS Workshop Keynote [Link]
  • ICPR 2024 PRHA Workshop Keynote [Link]

    • Softwares

  • AutoFM (SDM 2024)
  • MedDiffusion (SDM 2024)
  • MedNMP (EMNLP 2023)