Convening standard banner for intro slide

Have Your Cake & Synthesize it Too

Billy Buchanan, Ph.D.

Senior Research Scientist, SAG Corporation

Service Disabled Veteran Owned Small Business Emblem

https://wbuchanan.github.io/sdpConvening2023

Get Ready for a Game

Get a blank index card and something to draw with
Draw your best example of the object listed on the index card in the middle of the table (or in your group)
Once everyone has created their example collect all the examples and put the topic index card on top

Key Objectives

Distinguish between user- and model-based synthesis
Gain familiarity with some current synthesis methods
Understand the applications of synthesis methods

But first, some questions about the problems we face...

What are the biggest barriers you face to share data with external stakeholders?

How do you currently analyze data for your smallest demographic groups?

Synthetic Data: The Origin Stories

In the beginning, there was imputation...

Statistical agencies want to share data broadly while protecting privacy
First references to synthetic data come from Statistical Disclosure Control
Initially, synthetic data are a form of multiply imputed data
Brought to you by the makers of Multiple Imputation: Donald Rubin

and then computer scientists came along...

Computer vision researchers find they need more image data
Early methods are based on lossy compression techniques
In 2014, Goodfellow and his colleagues develop Generative Adversarial Networks (GAN)
Now in addition to images, GANs are synthesizing tabular data

What is synthetic data?

Data created by a computer
Can be created by:

User instructions/specifications
Models fitted to observed data

It is not synthetic control

Model-Defined Approaches

Imputation
Shallow Learning
SMOTE - Synthetic Minority Oversampling TEchnique
Deep Learning

Focus on Shallow/Deep Learning, Imputation Difficult/Costly, SMOTE alternatives

Imputation is still used to generate synthetic data, particularly the Census Bureau. It is fairly labor intensive as it requires specifying a model for each variable that will be synthesized. It also requires end-users to have some familiarity with analyzing multiply imputed data. Importantly, it can also include ML modeling techniques.
Shallow learning methods have become the defacto in the statistical disclosure control world. It is similar to imputation because models are used for all variables, but requires less work because the models are determined algorithmically. The code example I included in the GitHub repository uses this approach and highlights some stuff to consider.
I gave SMOTE it's own call out because it has more of a niche application. SMOTE is really useful to increase the sample size of small cell groups and interpolates from the sample space of the observed data to fill in the gaps, so to speak.
Deep learning approaches provide some additional flexibility that doesn't directly exist in the other methods, albeit at the cost of increased computational time. While it is possible to use DL approaches with CPU only, the time it will take is likely to be exponentially longer compared to using GPU/TPU.
We'll talk a bit more about the shallow and deep learning methods. Some deep learning methods can also be used to solve the same problem SMOTE attempts to solve.

Shallow Learning

Most common/popular - Classification & Regression Trees (CART)
First variable is sampled from the observed data; the rest is predicted from models
CART is prone to overfitting
As the number of variables increases, performance can be worse
Requires reshaping longitudinal data to wide format

Good for cross-sectional data, not so great for longitudinal or specific groups.

The heading on this slide is a link to the most popular software package for this type of data synthesis
The first variable in the visit sequence is sampled from the data, so make sure the first variable is not a student identifier and or specify the visit sequence to ensure the first variable isn't going to be one that will release PII.
Once the first variable is sampled, the model building begins. A model is built predicting the second variable to be synethesized from the first variable. The first "synthetic" variable is then passed to the model to predict the values of the second synthetic variable. This continues until you get to the last variable in the data set.
Make sure you specify model hyperparameters for the minimum cell size to stop at to help prevent overfitting.
May not perform well for longitudinal data due to a combination of error propagation and/or the process of modeling a single variable at a time.

Diagram to illustrate how the shallow learning methods generate synthetic data.

Deep Learning

Two primary - Conditional Tabular GAN (CTGAN) and Conditional Probabilistic AutoRegressive (CPAR)
CTGAN can be used like SMOTE
CPAR is designed for longitudinal data
Both are slow on CPU and benefit from GPU or TPU
No observed data is sampled into the synthetic data

Can provide alternative to SMOTE and better for longitudinal, but can take a while/use a lot of resources

The heading on this slide is a link to the software package I would recommend for this type of data synthesis
All output from these models is completely synthetic.
CTGAN is conditional because instead of starting from completely random noise it starts with random noise and a feature/variable that also appears in the real data. So you might have an indicator for biological sex in the data and that would be passed to the generator to help it "learn" how to generate samples of males and females. The conditioning can be on multiple variables as well. This is why it can work as a replacement for SMOTE, since you can generate more samples for specific groups to increase sample size and balance in the dataset.
CTGAN allows you to provide information about the underlying architecture in terms of the number of layers and nodes in each layer. Giving the generator more capacity than the discriminator will help with convergence. Additionally, as you use more layers you may want to consider decreasing the number of nodes. CTGAN will be good for cross-sectional data.
CPAR is impressive to say the least. DL typically does not do well with missingness and DL for timeseries does not do well with irregular time intervals or unbalanced panels. CPAR is fully capable of synthesizing balanced panel data and unbalanced panel data. However, the computations under the hood are much slower due to the autoregressive nature of the architecture.

Considerations

Privacy Protection vs Data Utility
Cross-Sectional vs Longitudinal Data
Time Sensitivity/Computational Resources
Validation Server

Address each bullet individually and provide recommendations.

In short, the more utility the synthetic data retains, the lower the privacy protection. To see why, if you maximized the utility of data synthesized from a source it would reproduce the original source. Additionally, both synthpop and the synthetic data vault include tools to assess the utility of your synthetic data and have varying capabilities for addressing privacy metrics.
For cross-sectional data synthesis, I would definitely recommend using the shallow learning approach since it is highly computationally efficient and does a good job. If you are trying to improve a predictive model that isn't working so well for small demographic groups, you could use SMOTE or CTGAN to generate synthetic records for those groups which should improve the predictive accuracy. For longitudinal data I would recommend using CPAR when possible. It will synthesize unbalanced panels and balanced panels, while accomplishing the same with other approaches is significantly more challenging and time consuming.
If you need to generate synthetic data quickly, shallow learning methods are likely to be the only viable solution. The deep learning methods can compete in clock time, but to do so requires substantial hardware. If the synthesis doesn't need to be performed as quickly as possible, you can leverage the power/flexibility of deep learning approaches.
If providing synthetic data to outside organizations, you may want to consider setting up some type of validation server. It might just be analysts running code that was developed by external partners and then screening results for privacy policy requirements. While blending synthetic data with actual observations helps predictive models become more robust, you shouldn't rely on inference from synthetic data (particularly since you know it will differ from the protected data in some way).

Who is using synthetic data?

Agency	Synthetic Product
Census Bureau	SIPP Synthetic Beta
Census Bureau	Synthetic Longitudinal Business Database
CMS	Medicare Claims SynPUF
IRS	Low-Income Info Returns SynPUF
IRS	Individual Tax Payer SynPUF
Veterans' Affairs	PseudoVet
Veterans' Health Administration	MDClone
Maryland SLDS	See Bonnéry et al. (2019)

If synthesis is allow with more restricted data, it shouldn't be a problem for less restricted data.

In addition to these, there have also been some experiments in the education sector as well.
The importance of Census and IRS, however, is that unlike FERPA, the laws governing privacy in the Census - Title 13 of the US Code - and IRS - Title 26 of the US Code - have criminal and financial penalties associated with disclosure. If synthesis is good enough for these agencies to satisfy the higher standards of privacy protections they implement, it is good enough to satisfy the requirements of FERPA.
Similarly, the project implemented by the Veterans' Health Administration needs to satisfy HIPAA requirements for privacy which are also more stringent than FERPA.
While some of the projects were intended for public use/consumption, others are intended solely for internal or researcher use, at least currently.

Want to play a game?

The phases of the game

Draw phase
Evaluate phase
Telephone phase

Explain how the game works

In the draw phase, the critic needs to keep their eyes closed. The rest of the group will take turns contributing to drawing a synthetic example of the image from your prompt.
Each "artist" will only have a short amount of time to draw before passing things on to the next artist.
Once the last "artist" in the group has added their drawing they will place the team's example and an example from the other group in front of the critic
During the evaluate phase, The critic's job is to determine which drawing is the real example (from the other team/group) and which is the synthetic drawing created by their table/group.
During the telephone phase, the critic whispers to the last artist their decision and explains why they made that decision. That artist then whispers the info to the artist that contributed before them, until it gets back to the first artist.
Once the telephone phase is complete, we start over again.

How GANs work

Explain how the architecture works while walking through the diagram

Two neural nets: a Generator and a Discriminator
Samples are synthesized by the generator
The discriminator evaluates which samples are real/fake
The goal of the discriminator is to get better at identifying synthetic data
The goal of the generator is to produce synthetic data that appears real
Each layer of the discriminator contributes a bit to the synthesis of the data. The final layer of the generator produces a sample with the same structure as the real data, but the properties are going to be bad at first.
The discriminator is basically a logistic regression model that is trying to correctly predict whether the samples are real/synthetic. The Goal is for the generator to create samples that are indistinguishable from real data, while the discriminator is trying to get better and better at identifying the synthetic samples.
After the discriminator does it's work, the error from the model is pushed back through the layers in reverse order to improve the quality of the synthetic samples.

Code Examples

CART-Based Synthesis

CTGAN-Based Synthesis

CPAR-Based Synthesis

You've Got Questions,

I've (hopefully) Got The Answers

Thank You

wbuchanan@sagcorp.com

Slides & Materials Available at:
github.com/wbuchanan/sdpConvening2023