Background and Motivation

Data-driven modeling is increasingly being incorporated into healthcare applications such as diagnostics, trial emulation, or decision making [1]. Clinical data used to train such models, however, is subject to two major challenges:

• Clinical data often does not adhere to standard naming conventions and needs to be manually harmonized by domain experts to make data usable for multi-cohort analyses.
• Patient-level data is especially worthy of protection, therefore data-sharing of such sensitive data is massively complicated by European data protection guidelines (GDPR).

Both challenges result in bottlenecks in research; the following post summarizes the findings of our four recent publications and presents scientific software to address them.
 

Different Naming Conventions for Clinical Data

Clinical data usually does not come in a unified format that enables cross-cohort studies - different data holders may often employ different naming conventions for the same concepts.

For example: The same test to measure cognitive decline in Alzheimer’s Disease may be referred to as “Mini-Mental State Examination (MMSE)” or “Folstein Test,” after its inventor, Marshal Folstein. While both refer to the same concept, they are lexically very different.

As studies do not always adhere to the exact same naming convention for equal concepts, concepts of different cohorts need to be mapped to a common reference term, which is, in practice, still mostly done through manual curation by a domain expert.

Automated Cohort Harmonization using Text-Embeddings

We do, however, know methods that can potentially facilitate this manual mapping process: Semantic similarities of different concepts can be matched regardless of their lexical similarities by computing their cosine similarities in embedding space. We evaluated the performance of language model embeddings for the pairwise harmonization of Alzheimer's Disease (AD) and Parkinson's Disease (PD) in a recent publication [2]. A two-dimensional projection of different variables in embedding space is shown in Figure 1. In the left plot (a), we see that both AD and PD share semantically common concepts. In the right plot, we can see that semantically similar variables of different cohorts of the same disease cluster closely together, which enables an automated matching approach in the embedding space. For the PD use case, >80% of variables could be matched fully automatically using this approach. We published a Python library [3, 4] that can be used to automate the pairwise harmonization of any clinical data using text embeddings.

Benchmarking Text-Embedding Models for Clinical Data Harmonization

Given that new and improved text-embedding models are published frequently [5], choosing the right model for the right task can be difficult, given the volume of potential model candidates. To identify the best-suited model for embedding-based variable harmonization of AD cohorts, we created a benchmark [6] specifically for matching AD-related metadata.

Benchmarking models for this very specific task showed surprising results: Model performance differed from big, general-purpose benchmarks, and model size did not directly correlate with model performance, with a small model (MiniLM [7]) in some cases outperforming bigger models (Figure 2). The benchmark is available as an open-source Python library [8].

Synthetic Data for Privacy-Preserving Data Sharing

Synthetic Data refers to artificially generated data that should mimic real data in terms of their structural statistical properties. It is being widely discussed as a potential alternative to traditional sensitive data-sharing approaches, such as anonymization, where it may provide better trade-offs in terms of privacy vs. utility [9]. In the generative process, evaluating synthetic data is crucial for model tuning and guaranteeing that the generated data is actually usable in practice.

Evaluating the quality of synthetic data is, however, non-trivial; it is usually evaluated based on three different dimensions:

• Fidelity: How closely does it resemble the real data in terms of their statistical properties?
• Privacy: Does the synthetic data contain residual risks that would enable an attacker to make assumptions on the real data or re-identify real individuals?
• Utility: Can synthetic data be used for real-world tasks with a similar performance as the real data would?
   

Synthetic Data Fidelity & Privacy

Synthetic data fidelity can be measured based on several different metrics; there is, however, no scientific consensus on which metrics to use [10]. As a general evaluation framework, we propose three different dimensions for synthetic data fidelity:

• Distribution Similarity: How closely do the marginal synthetic feature distributions match those of the real data?
• Correlation Preservation: How well are pairwise feature correlations preserved in synthetic data when compared with the real data?
• Discrimination Complexity: How difficult is it for a trained ML model to differentiate between the real and synthetic data?

Similarly, synthetic data privacy can also be evaluated based on different risk types, as in the proposed Anonymeter framework of Giomi et al. [11]:

• Singling Out Risk: Can an attacker use the synthetic data to isolate a single individual from the real data, for example, by identifying a unique combination of features that points to exactly one person?
• Linkability Risk: Can an attacker link records belonging to the same individual (or group) across different datasets by using the synthetic data as a bridge?
• Inference Risk: Can an attacker deduce previously unknown attributes of a real individual from the synthetic data, for example, infer a sensitive value from other known features?

Computing individual data fidelity metrics and risks for each feature after every model training iteration is tedious and time-consuming. To facilitate this synthetic data evaluation process, we created a framework [12] for automated evaluation that computes fidelity and risk scores in a user-friendly manner. In a Dashboard application, we compile relevant metrics and visualizations for synthetic patient-level data (Figure 3).

We also provide a Python package [13] for easy integration into training or evaluation workflows.

Synthetic Data Utility Trade-Offs

While synthetic data has shown promising results in several studies [9, 10], many researchers still advocate towards anonymized data for a better trade-off in utility vs privacy [14]. Frameworks such as Differential Privacy (DP) aim to provide formal mathematical privacy guarantees for synthetic data, thereby improving its credibility. While aiming to improve privacy, DP can, however, have a strong negative impact on data utility [15].

We conducted a study to systematically investigate trade-offs between synthetic data fidelity, privacy, and utility by comparing 5 different models (with and without DP) and one anonymization method for 3 different clinical datasets [16]. Our main findings were:

• DP introduced noise to feature correlations to a degree that the synthetic data utility is completely lost
• Synthetic data without DP can perform similarly well as real data on downstream tasks (Figure 4), without strong evidence of privacy breaches
• Anonymization, while preserving fidelity well, showed significant Singling Out risks

Future work should focus on methods that effectively balance privacy, fidelity, and utility in synthetic patient data generation, as DP does not appear to solve the problem of privacy-preserving synthetic data sharing with a favorable utility trade-off.

References

[1] Dash, S., Shakyawar, S. K., Sharma, M., & Kaushik, S. (2019). Big data in healthcare: Management, analysis and future prospects. Journal of Big Data, 6, Article 54. https://doi.org/10.1186/s40537-019-0217-0

[2] Salimi, Y., Adams, T., Ay, M. C., et al. (2025). Evaluating language model embeddings for Parkinson’s disease cohort harmonization using a novel manually curated variable mapping schema. Scientific Reports, 15, Article 20210. https://doi.org/10.1038/s41598-025-06447-2

[3] Adams, T., & Ay, M. C. (2026). datastew (Version 1.0.3) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.19386376

[4] Adams, T., Aborageh, M., Salimi, Y., Fröhlich, H. F., & Jacobs, M. (2024). INDEX: the Intelligent Data Steward Toolbox Utilizing Large Language Model Embeddings for Automated Data Harmonization. 15th International Conference on Semantic Web Applications and Tools for Health Care and Life Sciences, SWAT4HCLS 2024. https://publica.fraunhofer.de/handle/publica/509865

[5] Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023). MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (pp. 2014–2037). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.eacl-main.148

[6] Adams, T., et al. (2026). A benchmark of text embedding models for semantic harmonization of Alzheimer’s disease cohorts. The Journal of Prevention of Alzheimer’s Disease, 13(1), Article 100420. https://doi.org/10.1016/j.tjpad.2025.100420

[7] Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020). MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33, 5776–5788.

[8] Adams, T., Ay, M. C., & Salimi, Y. (2025). ADHTEB (Version 0.0.6) [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.17938949

[9] Sarmin, F. J., et al. (2024). Synthetic data: Revisiting the privacy-utility trade-off. arXiv. https://doi.org/10.48550/arXiv.2407.07926

[10] Hernandez, M., Epelde, G., Alberdi, A., Cilla, R., & Rankin, D. (2022). Synthetic data generation for tabular health records: A systematic review. Neurocomputing, 493, 28–45. https://doi.org/10.1016/j.neucom.2022.04.053

[11] Giomi, M., Boenisch, F., Wehmeyer, C., & Ganev, G. (2022). A unified framework for quantifying privacy risk in synthetic data. arXiv. https://doi.org/10.48550/arXiv.2211.10459

[12] Moazemi, S., et al. (2024). NFDI4Health workflow and service for synthetic data generation, assessment and risk management. arXiv. https://doi.org/10.48550/arXiv.2408.04478

[13] Adams, T., & Fröhlich, H. (2026). syndat (Version 0.13.8) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.18926005

[14] Stadler, T., Oprisanu, B., & Troncoso, C. (2022). Synthetic data—Anonymisation groundhog day. In 31st USENIX Security Symposium (USENIX Security 22) (pp. 1451–1468). USENIX Association.

[15] Ganev, G., Oprisanu, B., & De Cristofaro, E. (2022). Robin Hood and Matthew effects: Differential privacy has disparate impact on synthetic data. In Proceedings of the 39th International Conference on Machine Learning (pp. 6944–6958). PMLR.

[16] Adams, T., et al. (2025). On the fidelity versus privacy and utility trade-off of synthetic patient data. iScience, 28(5), Article 112663. https://doi.org/10.1016/j.isci.2025.112382