Background and Motivation

The potential of Artificial Intelligence (AI) in healthcare is significant, especially in supporting clinical diagnoses [1]. However, the performance of these models is directly dependent on the representativeness of the underlying data. Clinical datasets, such as Electronic Health Records (EHRs), are often highly imbalanced, leading to patient groups who are underrepresented and therefore disadvantaged by AI models [2]. This bias can reinforce existing health disparities.

When real data for these minority groups is lacking, the generation of synthetic data offers a potential solution. In my Bachelor Thesis I investigated the central research question of whether generative approaches can expand an existing dataset of a minority group so that downstream models can learn reliably. For this purpose, two central strategies were compared: Transfer Learning and Domain Adaptation.

Methods

Dataset

The study is based on the publicly available dataset “Diabetes 130-US Hospitals (1999-2008)” [3]. The minority class was defined as the population with the race attribute “Asian” (0.62% of the dataset). The predictive task was performed on hospital readmissions within 30 days of discharge (attribute readmitted).

Generative Frameworks

Both approaches were based on Generative Adversarial Networks (GANs) [4]. A GAN uses two neural networks – a Generator (G) and a Discriminator (D) – which are trained in a minimax game. The Generator attempts to create data that the Discriminator can no longer distinguish from real data.

Approach 1 (Transfer Learning): A Conditional Tabular GAN (CTGAN), designed for tabular data, was used [5] with a modified transfer learning strategy: The CTGAN was pre-trained on the majority cohort and subsequently fine-tuned on a combined dataset of majority and minority data to transfer the learned knowledge.

Approach 2 (Domain Adaptation): The RadialGAN [6], a framework for translation between datasets, was used and methodically adapted to fit the use case: Every single value in the race column was defined as its own “pseudo-domain” to facilitate knowledge transfer from the data-rich domains to the minority “Asian” domain.

Results

The answer to the research question was clear: The Transfer Learning approach (CTGAN) was superior to the adapted Domain Adaptation approach (RadialGAN) in the essential evaluation dimensions.

Fidelity

Both models met the baseline requirement for statistical fidelity. Their generated data showed a lower Maximum Mean Discrepancy (MMD) compared to the real minority data than the baseline MMD found between the real minority and majority cohorts. However, qualitative analyses (t-SNE) showed clear differences, as shown in Figure 1: The CTGAN data produced a high structural overlap with the real clusters of the minority data (a). The RadialGAN data showed a better visual separation from the majority data (d), but at the cost of incomplete coverage of the real minority distribution (b).

Utility

A classification model trained with the “Train-on-Synth, Test-on-Real” (TSTR) protocol, and evaluated on a holdout test set of real minority data established via a patient-grouped cross-validation, revealed the greatest differences, as visualized in Figure 2. The CTGAN model achieved a stable and useful predictive performance (AUROC) that was equivalent to that of a baseline model trained on the real minority data. It therefore proved to be a valid substitute.

The RadialGAN, in contrast, showed no predictive performance on the same level as the real data. The AUROC scores were partly below the random chance value of 0.5. This result is attributable to an architecture mismatch: The model, designed for heterogeneous domains, was used on a homogeneous dataset and thus over-interpreted the differences between the internal cohorts.

Fairness

The models trained on CTGAN data showed a consistent tendency to reduce algorithmic bias compared to the models trained on the real minority data. The bias scores were measured using probabilistic variants of the Equal Opportunity Gap (EOG) and Demographic Parity Gap (DPG). The results from the RadialGAN data were inconsistent and not meaningful due to the lack of utility.

Conclusion

The central finding of this work is that for improving AI in fairness for rare, internal groups, an isolated view of these groups is not effective. The key lies in the transfer of robust knowledge from other populations. The Transfer Learning approach of the CTGAN proved to be a successful, efficient, and robust strategy for this purpose. The unreflective adaptation of a Domain Adaptation architecture like RadialGAN to this specific problem led to unstable results. These methods represent an important step towards improving the data foundation for medical AI models.

Citations

[1] I. Y. Chen, E. Pierson, S. Rose, S. Joshi, K. Ferryman and M. Ghassemi, "Ethical Machine Learning in Healthcare," Annual Review of Biomedical Data Science, vol. 4, pp. 123-144, 2021.

[2] Z. Obermeyer, B. Powers, C. Vogeli and S. Mullainathan, "Dissecting racial bias in an algorithm used to manage the health of populations," Science, vol. 366, no. 6464, pp. 447-453, 2019.

[3] B. Strack, J. P. DeShazo, C. Gennings, J. L. Olmo, S. Ventura, K. J. Cios and J. N. Clore, "Impact of HbA1c Measurement on Hospital Readmission Rates: Analysis of 70,000 Clinical Database Patient Records," BioMed Research International, vol. 2014, 2014.

[4] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville and Y. Bengio, "Generative adversarial nets," in Advances in neural information processing systems 27 (NIPS 2014), 2014.

[5] L. Xu, M. Skoularidou, A. Cuesta-Infante and K. Veeramachaneni, "Modeling tabular data using conditional GAN," in Advances in neural information processing systems, 2019.

[6] J. Yoon, J. Jordon and M. van der Schaar, "RadialGAN: Leveraging multiple datasets to improve target-specific predictive models using Generative Adversarial Networks," in Proceedings of the 35th International conference on machine learning, 2018.