ABSTRACT
Background Open science is a movement seeking to make scientific research accessible to all, including publication of code and data. Publishing patient-level data may, however, compromise the confidentiality of that data if there is any significant risk that data may later be associated with individuals. Use of synthetic data offers the potential to be able to release data that may be used to evaluate methods or perform preliminary research without risk to patient confidentiality.
Methods We have tested five synthetic data methods:
A technique based on Principal Component Analysis (PCA) which samples data from distributions derived from the transformed data.
Synthetic Minority Oversampling Technique, SMOTE which is based on interpolation between near neighbours.
Generative Adversarial Network, GAN, an artificial neural network approach with competing networks - a discriminator network trained to distinguish between synthetic and real data., and a generator network trained to produce data that can fool the discriminator network.
CT-GAN, a refinement of GANs specifically for the production of structured tabular synthetic data.
Variational Auto Encoders, VAE, a method of encoding data in a reduced number of dimensions, and sampling from distributions based on the encoded dimensions.
Two data sets are used to evaluate the methods:
The Wisconsin Breast Cancer data set, a histology data set where all features are continuous variables.
A stroke thrombolysis pathway data set, a data set describing characteristics for patients where a decision is made whether to treat with clot-busting medication. Features are mostly categorical, binary, or integers.
Methods are evaluated in three ways:
The ability of synthetic data to train a logistic regression classification model.
A comparison of means and standard deviations between original and synthetic data.
A comparison of covariance between features in the original and synthetic data.
Results Using the Wisconsin Breast Cancer data set, the original data gave 98% accuracy in a logistic regression classification model. Synthetic data sets gave between 93% and 99% accuracy. Performance (best to worst) was SMOTE > PCA > GAN > CT-GAN = VAE. All methods produced a high accuracy in reproducing original data means and stabdard deviations (all R-square > 0.96 for all methods and data classes). CT-GAN and VAE suffered a significant loss of covariance between features in the synthetic data sets.
Using the Stroke Pathway data set, the original data gave 82% accuracy in a logistic regression classification model. Synthetic data sets gave between 66% and 82% accuracy. Performance (best to worst) was SMOTE > PCA > CT-GAN > GAN > VAE. CT-GAN and VAE suffered loss of covariance between features in the synthetic data sets, though less pronounced than with the Wisconsin Breast Cancer data set.
Conclusions The pilot work described here shows, as proof of concept, that synthetic data may be produced, which is of sufficient quality to publish with open methodology, to allow people to better understand and test methodology. The quality of the synthetic data also gives promise of data sets that may be used for screening of ideas, or for research project (perhaps especially in an education setting).
More work is required to further refine and test methods across a broader range of patient-level data sets.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This article presents independent research funded by the National Institute for Health Research (NIHR) Applied Research Collaboration South West Peninsula.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
No identifiable patient data accessed (public domain anonymous data used). No need to seek approval or exemption.
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
The code and data used in this study is available at https://github.com/MichaelAllen1966/synthetic_data_pilot/releases/tag/1.0.0
https://github.com/MichaelAllen1966/synthetic_data_pilot/releases/tag/1.0.0