Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction Multimodal Medical Data Analysis

Terveydenhuollon suurten tietomassojen kasvu ja monimodaalisen lääketieteellisen datan jatkuva kertyminen ovat tehneet erilaisten tietolähteiden tehokkaasta yhdistämisestä keskeisen haasteen älykkäässä terveydenhuollossa. Monimodaalinen data – kuten kliiniset potilastiedot, lääketieteelliset kuvanta...

Täydet tiedot

Bibliografiset tiedot
Päätekijä: Hao, Xinyu
Muut tekijät: Informaatioteknologian tiedekunta, Faculty of Information Technology
Aineistotyyppi: Väitöskirja
Julkaistu: 2025
Linkit: https://jyx.jyu.fi/handle/123456789/102937
_version_ 1833860119095410688
author Hao, Xinyu
author2 Informaatioteknologian tiedekunta Faculty of Information Technology
author_facet Hao, Xinyu Informaatioteknologian tiedekunta Faculty of Information Technology Hao, Xinyu Informaatioteknologian tiedekunta Faculty of Information Technology
author_sort Hao, Xinyu
datasource_str_mv jyx
description Terveydenhuollon suurten tietomassojen kasvu ja monimodaalisen lääketieteellisen datan jatkuva kertyminen ovat tehneet erilaisten tietolähteiden tehokkaasta yhdistämisestä keskeisen haasteen älykkäässä terveydenhuollossa. Monimodaalinen data – kuten kliiniset potilastiedot, lääketieteelliset kuvantamiset, genomiinformaatio ja tekstimuotoiset lausunnot – tarjoaa toisiaan täydentäviä näkökulmia potilaan terveydentilasta. Näiden yhdistäminen analyysissä voi merkittävästi parantaa diagnostiikan ja hoitotulosten tarkkuutta ja selitettävyyttä. Haasteita aiheuttavat kuitenkin edelleen esimerkiksi rajoitettu määrä merkittyä dataa, puutteelliset tietueet, sairauden ajallinen dynamiikka ja vaikeus mallintaa kudosleikkeiden alueellista yhteistoimintaa. Tässä väitöskirjassa tutkitaan syväoppimismenetelmiä monimodaalisen lääketieteellisen datan analysointiin, tavoitteenaan parantaa sairauden diagnosointia ja hoitovasteen ennustamista. Artikkeleissa I ja II kehitettiin vaiheittaisia koneoppimismalleja raskauden aikaisen riskin reaaliaikaiseen ennustamiseen pitkittäisten kliinisten tietojen perusteella. Artikkelit III ja IV esittivät hybridikehyksen, joka yhdistää heikosti ja osittain valvotun oppimisen rintasyövän patologisen hoitovasteen ennustamiseksi moniparametrisestä MRI-datasta, vähentäen pikselitason annotaatioiden tarvetta. Artikkelissa V esiteltiin ajalliseen kontrastioppimiseen perustuva malli kasvaimen dynamiikan mallintamiseksi pitkittäisen MRI:n avulla. Artikkelissa VI kehitettiin monimodaalinen syklinen piirreluontiverkko, joka oppii patologian ja genomiikan yhteisiä esityksiä koulutuksessa, mahdollistaen ennustamisen pelkän patologian perusteella käyttöhetkellä. Lopuksi artikkelissa VII parannettiin eturauhassyövän Gleason-luokitusta tuomalla esiin mallin, joka hyödyntää valikoivaa piirrekoostamista ja alueiden välistä yhteistyötä digitaalisissa kudosleikkeissä. Yhteenvetona väitöskirja esittelee innovatiivisia menetelmiä, jotka liittyvät monimodaaliseen tietofuusioon, heikosti valvottuun oppimiseen, ajalliseen mallintamiseen ja asiantuntijatietoon perustuvaan analyysiin. Menetelmiä on validoitu laajasti julkisissa ja kliinisissä aineistoissa, ja ne ovat saavuttaneet erinomaisia tuloksia syöpätutkimuksen eri tehtävissä. Tulokset tukevat tarkkuuslääketieteen edistämistä älykkäiden, dataohjattujen ratkaisujen avulla. With the rapid advancement of healthcare big data and the increasing availability of multimodal medical data, the effective integration and analysis of heterogeneous data sources have emerged as critical challenges in intelligent healthcare. Multimodal medical data, such as structured clinical records, medical imaging including MRI and digital pathology, genomic information, and textual medical reports, provide complementary perspectives on a patient’s health status. Joint analysis of these modalities can substantially enhance the accuracy, robustness, and interpretability of disease diagnosis and treatment outcome prediction. However, several technical challenges persist, including limited labeled data, incomplete records, the temporal complexity of disease progression, and the absence of collaborative modeling across spatial regions in digital pathology slides. This dissertation explores deep learning methods for multimodal medical data analysis to improve disease diagnosis and treatment response prediction. In Articles I & II, we developed stage-wise machine learning models for realtime pregnancy risk prediction using longitudinal clinical data. Articles III & IV proposed a hybrid weakly and semi-supervised framework for predicting pathological complete response in breast cancer using multi-parametric MRI, reducing reliance on pixel-level annotations. In Article V, we introduced a temporal contrastive learning model to capture dynamic tumor evolution using longitudinal MRI. Article VI presents a multimodal cyclic generation network that learns joint pathology-genomics representations during training, enabling single-pathology input for prognosis prediction at inference. Finally, Article VII enhances prostate cancer Gleason grading by introducing a pattern-aware feature aggregation strategy to model collaborative regional features in whole slide images. Collectively, this dissertation contributes a series of innovations across multimodal data fusion, weak/semi-supervised learning, temporal modeling, and domain-informed feature aggregation. The proposed methods have been extensively validated on public datasets and clinical cohorts, achieving state-of-the-art performance in multiple cancer-related tasks. The findings offer a solid foundation for advancing precision medicine through intelligent data-driven solutions.
first_indexed 2025-06-02T20:00:55Z
format Väitöskirja
free_online_boolean 1
fullrecord [{"key": "dc.contributor.author", "value": "Hao, Xinyu", "language": null, "element": "contributor", "qualifier": "author", "schema": "dc"}, {"key": "dc.date.accessioned", "value": "2025-06-02T11:00:13Z", "language": null, "element": "date", "qualifier": "accessioned", "schema": "dc"}, {"key": "dc.date.available", "value": "2025-06-02T11:00:13Z", "language": null, "element": "date", "qualifier": "available", "schema": "dc"}, {"key": "dc.date.issued", "value": "2025", "language": null, "element": "date", "qualifier": "issued", "schema": "dc"}, {"key": "dc.identifier.isbn", "value": "978-952-86-0734-2", "language": null, "element": "identifier", "qualifier": "isbn", "schema": "dc"}, {"key": "dc.identifier.uri", "value": "https://jyx.jyu.fi/handle/123456789/102937", "language": null, "element": "identifier", "qualifier": "uri", "schema": "dc"}, {"key": "dc.description.abstract", "value": "Terveydenhuollon suurten tietomassojen kasvu ja monimodaalisen l\u00e4\u00e4ketieteellisen\ndatan jatkuva kertyminen ovat tehneet erilaisten tietol\u00e4hteiden tehokkaasta\nyhdist\u00e4misest\u00e4 keskeisen haasteen \u00e4lykk\u00e4\u00e4ss\u00e4 terveydenhuollossa. Monimodaalinen\ndata \u2013 kuten kliiniset potilastiedot, l\u00e4\u00e4ketieteelliset kuvantamiset, genomiinformaatio\nja tekstimuotoiset lausunnot \u2013 tarjoaa toisiaan t\u00e4ydent\u00e4vi\u00e4 n\u00e4k\u00f6kulmia\npotilaan terveydentilasta. N\u00e4iden yhdist\u00e4minen analyysiss\u00e4 voi merkitt\u00e4v\u00e4sti\nparantaa diagnostiikan ja hoitotulosten tarkkuutta ja selitett\u00e4vyytt\u00e4. Haasteita\naiheuttavat kuitenkin edelleen esimerkiksi rajoitettu m\u00e4\u00e4r\u00e4 merkitty\u00e4 dataa,\npuutteelliset tietueet, sairauden ajallinen dynamiikka ja vaikeus mallintaa kudosleikkeiden\nalueellista yhteistoimintaa.\nT\u00e4ss\u00e4 v\u00e4it\u00f6skirjassa tutkitaan syv\u00e4oppimismenetelmi\u00e4 monimodaalisen l\u00e4\u00e4ketieteellisen\ndatan analysointiin, tavoitteenaan parantaa sairauden diagnosointia\nja hoitovasteen ennustamista. Artikkeleissa I ja II kehitettiin vaiheittaisia koneoppimismalleja\nraskauden aikaisen riskin reaaliaikaiseen ennustamiseen pitkitt\u00e4isten\nkliinisten tietojen perusteella. Artikkelit III ja IV esittiv\u00e4t hybridikehyksen,\njoka yhdist\u00e4\u00e4 heikosti ja osittain valvotun oppimisen rintasy\u00f6v\u00e4n patologisen\nhoitovasteen ennustamiseksi moniparametrisest\u00e4 MRI-datasta, v\u00e4hent\u00e4en\npikselitason annotaatioiden tarvetta. Artikkelissa V esiteltiin ajalliseen kontrastioppimiseen\nperustuva malli kasvaimen dynamiikan mallintamiseksi pitkitt\u00e4isen\nMRI:n avulla. Artikkelissa VI kehitettiin monimodaalinen syklinen piirreluontiverkko,\njoka oppii patologian ja genomiikan yhteisi\u00e4 esityksi\u00e4 koulutuksessa,\nmahdollistaen ennustamisen pelk\u00e4n patologian perusteella k\u00e4ytt\u00f6hetkell\u00e4.\nLopuksi artikkelissa VII parannettiin eturauhassy\u00f6v\u00e4n Gleason-luokitusta tuomalla\nesiin mallin, joka hy\u00f6dynt\u00e4\u00e4 valikoivaa piirrekoostamista ja alueiden v\u00e4list\u00e4\nyhteisty\u00f6t\u00e4 digitaalisissa kudosleikkeiss\u00e4.\nYhteenvetona v\u00e4it\u00f6skirja esittelee innovatiivisia menetelmi\u00e4, jotka liittyv\u00e4t\nmonimodaaliseen tietofuusioon, heikosti valvottuun oppimiseen, ajalliseen mallintamiseen\nja asiantuntijatietoon perustuvaan analyysiin. Menetelmi\u00e4 on validoitu\nlaajasti julkisissa ja kliinisiss\u00e4 aineistoissa, ja ne ovat saavuttaneet erinomaisia\ntuloksia sy\u00f6p\u00e4tutkimuksen eri teht\u00e4viss\u00e4. Tulokset tukevat tarkkuusl\u00e4\u00e4ketieteen\nedist\u00e4mist\u00e4 \u00e4lykk\u00e4iden, dataohjattujen ratkaisujen avulla.", "language": "fin", "element": "description", "qualifier": "abstract", "schema": "dc"}, {"key": "dc.description.abstract", "value": "With the rapid advancement of healthcare big data and the increasing availability\nof multimodal medical data, the effective integration and analysis of heterogeneous\ndata sources have emerged as critical challenges in intelligent healthcare.\nMultimodal medical data, such as structured clinical records, medical imaging\nincluding MRI and digital pathology, genomic information, and textual medical\nreports, provide complementary perspectives on a patient\u2019s health status.\nJoint analysis of these modalities can substantially enhance the accuracy, robustness,\nand interpretability of disease diagnosis and treatment outcome prediction.\nHowever, several technical challenges persist, including limited labeled data, incomplete\nrecords, the temporal complexity of disease progression, and the absence\nof collaborative modeling across spatial regions in digital pathology slides.\nThis dissertation explores deep learning methods for multimodal medical\ndata analysis to improve disease diagnosis and treatment response prediction.\nIn Articles I & II, we developed stage-wise machine learning models for realtime\npregnancy risk prediction using longitudinal clinical data. Articles III & IV\nproposed a hybrid weakly and semi-supervised framework for predicting pathological\ncomplete response in breast cancer using multi-parametric MRI, reducing\nreliance on pixel-level annotations. In Article V, we introduced a temporal contrastive\nlearning model to capture dynamic tumor evolution using longitudinal\nMRI. Article VI presents a multimodal cyclic generation network that learns joint\npathology-genomics representations during training, enabling single-pathology\ninput for prognosis prediction at inference. Finally, Article VII enhances prostate\ncancer Gleason grading by introducing a pattern-aware feature aggregation strategy\nto model collaborative regional features in whole slide images.\nCollectively, this dissertation contributes a series of innovations across multimodal\ndata fusion, weak/semi-supervised learning, temporal modeling, and\ndomain-informed feature aggregation. The proposed methods have been extensively\nvalidated on public datasets and clinical cohorts, achieving state-of-the-art\nperformance in multiple cancer-related tasks. The findings offer a solid foundation\nfor advancing precision medicine through intelligent data-driven solutions.", "language": "en", "element": "description", "qualifier": "abstract", "schema": "dc"}, {"key": "dc.description.provenance", "value": "Submitted by Paivi Vuorio (paelvuor@jyu.fi) on 2025-06-02T11:00:13Z\nNo. of bitstreams: 0", "language": "en", "element": "description", "qualifier": "provenance", "schema": "dc"}, {"key": "dc.description.provenance", "value": "Made available in DSpace on 2025-06-02T11:00:13Z (GMT). No. of bitstreams: 0\n Previous issue date: 2025", "language": "en", "element": "description", "qualifier": "provenance", "schema": "dc"}, {"key": "dc.format.mimetype", "value": "application/pdf", "language": null, "element": "format", "qualifier": "mimetype", "schema": "dc"}, {"key": "dc.publisher", "value": "Jyv\u00e4skyl\u00e4n yliopisto", "language": null, "element": "publisher", "qualifier": null, "schema": "dc"}, {"key": "dc.relation.ispartofseries", "value": "JYU Dissertations", "language": null, "element": "relation", "qualifier": "ispartofseries", "schema": "dc"}, {"key": "dc.relation.haspart", "value": "<b>Artikkeli I:</b> Hao, X., Zheng, D., Khan, M., Wang, L., H\u00e4m\u00e4l\u00e4inen, T., Cong, F., Xu, H., Song, K. (2023). Machine learning models for predicting adverse pregnancy outcomes in pregnant women with systemic lupus erythematosus. <i>Diagnostics, 13(4), 612.</i> DOI: <a href=\"https://doi.org/10.3390/diagnostics13040612\"target=\"_blank\">10.3390/diagnostics13040612</a>", "language": null, "element": "relation", "qualifier": "haspart", "schema": "dc"}, {"key": "dc.relation.haspart", "value": "<b>Artikkeli II:</b> Zheng, D., Hao, X., Khan, M., Wang, L., Li, F., Xiang, N., Kang, F., Hamalainen, T., Cong, F., Song, K., & Qiao, C. (2022). Comparison of machine learning and logistic regression as predictive models for adverse maternal and neonatal outcomes of preeclampsia : A retrospective study. <i>Frontiers in Cardiovascular Medicine, 9, Article 959649.</i> DOI: <a href=\"https://doi.org/10.3389/fcvm.2022.959649\"target=\"_blank\">10.3389/fcvm.2022.959649</a>", "language": null, "element": "relation", "qualifier": "haspart", "schema": "dc"}, {"key": "dc.relation.haspart", "value": "<b>Artikkeli III:</b> Hao, X., Xu, H., Zhao, N., Yu, T., H\u00e4m\u00e4l\u00e4inen, T., & Cong, F. (2024). Predicting pathological complete response based on weakly and semi-supervised joint learning in breast cancer multi-parametric MRI. <i>Biomedical Signal Processing and Control, 93, Article 106164.</i> DOI: <a href=\"https://doi.org/10.1016/j.bspc.2024.106164\"target=\"_blank\">10.1016/j.bspc.2024.106164</a>", "language": null, "element": "relation", "qualifier": "haspart", "schema": "dc"}, {"key": "dc.relation.haspart", "value": "<b>Artikkeli IV:</b> Hao, X., Xu, H., Zhao, N., Yu, T., H\u00e4m\u00e4l\u00e4inen, T., \nCong, F. (2023). Predicting pathological complete response\nbased on weakly and semi-supervised joint learning from\nbreast cancer MRI. <i>2023 45th Annual International Conference of\nthe IEEE Engineering in Medicine & Biology Society (EMBC), 1-4.</i> DOI: <a href=\"https://doi.org/10.1109/EMBC40787.2023.10340081\"target=\"_blank\">10.1109/EMBC40787.2023.10340081</a>", "language": null, "element": "relation", "qualifier": "haspart", "schema": "dc"}, {"key": "dc.relation.haspart", "value": "<b>Artikkeli V:</b> Hao, X., Xu, H., Zhang, Q., Xu, Q., P\u00f6l\u00f6nen, I., Cong; F.(2025). Predicting radiation therapy response based on dynamic temporal\nfeature difference fusion from longitudinal MRI. <i>Submitted to MICCAI\n2025, the 28th International Conference on Medical Image Computing and\nComputer-Assisted Intervention.</i>", "language": null, "element": "relation", "qualifier": "haspart", "schema": "dc"}, {"key": "dc.relation.haspart", "value": "<b>Artikkeli VI:</b> Hao, X., Xu, H., Wang, X., Wang, T., H\u00e4m\u00e4l\u00e4inen, T., \nCong, F. (2025). Cyclic translations between pathomics and genomics\nimprove automatic cancer diagnosis from whole slide images. <i>Submitted to\nEngineering Applications of Artificial Intelligence.</i>", "language": null, "element": "relation", "qualifier": "haspart", "schema": "dc"}, {"key": "dc.relation.haspart", "value": "<b>Artikkeli VII:</b> Hao, X., Xu, H., Zhang, Q., Xu, Q., P\u00f6l\u00f6nen, I., Cong, F. (2025). Dual selective Gleason pattern-aware multiple instance learning\nfor grade group prediction in histopathology images. <i>Early Accept by\nMICCAI 2025, the 28th International Conference on Medical Image Computing\nand Computer-Assisted Intervention.</i>", "language": null, "element": "relation", "qualifier": "haspart", "schema": "dc"}, {"key": "dc.title", "value": "Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction : Multimodal Medical Data Analysis", "language": null, "element": "title", "qualifier": null, "schema": "dc"}, {"key": "dc.type", "value": "doctoral thesis", "language": null, "element": "type", "qualifier": null, "schema": "dc"}, {"key": "dc.identifier.urn", "value": "URN:ISBN:978-952-86-0734-2", "language": null, "element": "identifier", "qualifier": "urn", "schema": "dc"}, {"key": "dc.contributor.faculty", "value": "Informaatioteknologian tiedekunta", "language": null, "element": "contributor", "qualifier": "faculty", "schema": "dc"}, {"key": "dc.contributor.faculty", "value": "Faculty of Information Technology", "language": null, "element": "contributor", "qualifier": "faculty", "schema": "dc"}, {"key": "dc.type.coar", "value": "http://purl.org/coar/resource_type/c_db06", "language": null, "element": "type", "qualifier": "coar", "schema": "dc"}, {"key": "dc.relation.issn", "value": "2489-9003", "language": null, "element": "relation", "qualifier": "issn", "schema": "dc"}, {"key": "dc.rights.copyright", "value": "\u00a9 The Author & University of Jyv\u00e4skyl\u00e4", "language": null, "element": "rights", "qualifier": "copyright", "schema": "dc"}, {"key": "dc.rights.accesslevel", "value": "openAccess", "language": null, "element": "rights", "qualifier": "accesslevel", "schema": "dc"}, {"key": "dc.type.publication", "value": "doctoralThesis", "language": null, "element": "type", "qualifier": "publication", "schema": "dc"}, {"key": "dc.format.content", "value": "fulltext", "language": null, "element": "format", "qualifier": "content", "schema": "dc"}, {"key": "dc.description.accessibilityfeature", "value": "navigointi mahdollista", "language": "fi", "element": "description", "qualifier": "accessibilityfeature", "schema": "dc"}, {"key": "dc.description.accessibilityfeature", "value": "structural navigation", "language": "en", "element": "description", "qualifier": "accessibilityfeature", "schema": "dc"}, {"key": "dc.description.accessibilitysummary", "value": "LaTexilla tuotettu pdf.", "language": "fi", "element": "description", "qualifier": "accessibilitysummary", "schema": "dc"}, {"key": "dc.description.accessibilitysummary", "value": "PDF generated using LaTeX.", "language": "en", "element": "description", "qualifier": "accessibilitysummary", "schema": "dc"}]
id jyx.123456789_102937
last_indexed 2025-06-02T20:03:25Z
main_date 2025-01-01T00:00:00Z
main_date_str 2025
online_boolean 1
online_urls_str_mv {"url":"https:\/\/jyx.jyu.fi\/bitstreams\/c0b1cd68-0192-454f-af6f-07de69224e0b\/download","text":"978-952-86-0734-2_vaitos11062025.pdf","source":"jyx","mediaType":"application\/pdf"}
publishDate 2025
record_format qdc
source_str_mv jyx
spellingShingle Hao, Xinyu Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction : Multimodal Medical Data Analysis
title Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction : Multimodal Medical Data Analysis
title_full Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction : Multimodal Medical Data Analysis
title_fullStr Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction : Multimodal Medical Data Analysis Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction : Multimodal Medical Data Analysis
title_full_unstemmed Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction : Multimodal Medical Data Analysis Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction : Multimodal Medical Data Analysis
title_short Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction
title_sort multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction multimodal medical data analysis
title_sub Multimodal Medical Data Analysis
title_txtP Multimodal medical data analysis for improved disease diagnosis and treatment outcome prediction : Multimodal Medical Data Analysis
url https://jyx.jyu.fi/handle/123456789/102937 http://www.urn.fi/URN:ISBN:978-952-86-0734-2
work_keys_str_mv AT haoxinyu multimodalmedicaldataanalysisforimproveddiseasediagnosisandtreatmentoutcomepredictionmu