Membership Inference and Model Inversion: Keeping Training Data Private

A membership inference attack answers a question that should be private: was this person’s record used to train the model? Its close relative, model inversion, goes further and reconstructs the training data itself. Neither needs a break-in. Both work through the model’s ordinary interface, by asking it questions and studying the answers.

This guide explains how the two attacks work, why they turn a model into a privacy liability, how to spot them, and the controls ISO/IEC 27090 recommends, including an honest look at what differential privacy costs.

What this guide covers

• How a membership inference attack works
• Model inversion and sensitive output
• Why a membership inference attack is a privacy incident
• Detecting a membership inference attack
• Controls against membership inference and model inversion
• Assessing an existing model for membership inference attack risk
• Mitigations have to line up across the life cycle
• Zero trust measures
• Frequently asked questions
• Treating the model as a data store

How a membership inference attack works

Machine learning models tend to behave slightly differently on data they were trained on than on data they have never seen: more confident, lower error. A membership inference attack exploits that gap.

The classic approach described in ISO 27090 has three steps. The attacker trains shadow models that imitate the target, using data whose membership they already know. They collect the shadow models’ outputs, labelled member or non-member. Then they train a classifier on those outputs that, given the target model’s response to a record, predicts whether that record was in its training set.

Query access is enough. If the attacker also has the model itself, the attack gets easier. A variant, property inference, uses the same technique to learn a general property of the training population, such as whether one group is over-represented.

Model inversion and sensitive output

Three related threats sit around the membership inference attack in ISO 27090.

Model inversion is particularly effective when a model returns confidence scores with its predictions, and when an attacker already holds partial information about a person and wants the rest. Reconstructions are sometimes only an average of a class rather than one individual, but that cannot be relied on.

Why a membership inference attack is a privacy incident

Knowing that a record was in a training set can itself be a disclosure. If a model was trained only on patients with a condition, customers who defaulted, or employees under investigation, membership reveals the attribute. ISO 27090 notes that where training data includes personal data, a successful attack can be a privacy incident, and in some jurisdictions that means legal assessment and notification to regulators and the people affected. ISO/IEC 27701 provides controls for handling those breaches.

Model inversion makes the loss permanent: once training data has been reconstructed, it cannot be recalled. And generative models add a further route, because they can reproduce memorised identifiers, text or images in normal output.

If you run data protection impact assessments for AI, these attacks belong in them. Our guide to DPIAs for AI systems shows where.

Detecting a membership inference attack

Detection is hard because each query can look legitimate. ISO 27090 points to analysis of query patterns per caller instead of per query, looking at:

• how often the caller queries and at what times;
• the order of queries and how they evolve;
• which features the caller varies, and how diverse the queries are.

For model inversion, a caller exploring the whole prediction space is a strong signal. These analyses double as feedback: they can show where training should be strengthened, for example by removing highly correlated features from the dataset.

Controls against membership inference and model inversion

We group the ISO 27090 mitigations by cost.

Baseline: cheap and effective

1. Return less. Give external callers the label or answer they need, not the full probability vector or confidence scores. This removes much of the signal both attacks depend on.
2. Restrict and throttle queries. Limit who can query the model and how often, per user. ISO 27090 distinguishes this from ordinary rate limiting: the aim is to slow experimentation, not to protect capacity.
3. Minimise training data. Remove records and fields the model does not need, and drop highly correlated features. Data that was never in the training set cannot be inferred from it.
4. Reduce overfitting. Regularisation such as dropout, and a model sized to its task, narrow the gap between member and non-member behaviour that the attack exploits.
5. Filter output. Detect and redact recognisable sensitive data, such as identifiers and contact details, before output leaves the system.

Risk-driven

• Shuffle training data and de-identify it.
• Train on synthetic data where it supports the use case.
• Run a privacy audit tool before release to estimate how much membership the model leaks.

Specialist: differential privacy

Differential privacy adds calibrated noise so that the presence or absence of any single record does not significantly change the model’s outputs. It is the only mitigation here that comes with a formal guarantee. ISO 27090 is honest about the catch: for complex learning tasks, current differentially private training rarely offers an acceptable balance. Settings that keep accuracy give little real privacy, and settings that give strong privacy can leave the model useless. Decide on measured results, not on the label.

Assessing an existing model for membership inference attack risk

Many teams meet this threat with models already in production. A short assessment tells you how exposed each one is:

1. Was it trained or fine-tuned on personal or confidential data? If not, the privacy impact of a membership inference attack is low, though intellectual property may still be at stake.
2. Who can query it? A public endpoint is reachable by anyone; an internal one only by staff and compromised accounts.
3. What does it return? Labels only, or scores and probabilities? Full confidence vectors are the attacker’s best friend.
4. Is it overfitted? A large gap between training and validation accuracy suggests it behaves differently on members, which is exactly what the attack measures.
5. Is query volume per user limited and monitored? If not, an attacker can run thousands of probes unnoticed.
6. Has leakage been measured? A privacy audit tool gives a baseline to compare against after each retraining.

Record the answers in the system’s risk assessment. A model trained on sensitive data, publicly reachable, returning scores and unthrottled is a high risk regardless of how good its accuracy is.

Mitigations have to line up across the life cycle

The membership inference attack is ISO 27090’s own example of a threat that needs related mitigations at several stages: audit checks on training data, privacy-preserving training, and input and output filtering at inference. The standard asks that these be checked to work together. Concretely, the checks applied to training data should look for the same things the output filter catches, and the test set should be representative enough to show whether they do.

The same clause warns that mitigations decay. Re-measure membership leakage after retraining, not just at first release. More on this in AI security controls.

Zero trust measures

ISO 27090 attaches three measures to membership inference and model inversion:

• Layered input checks: input passes more than one audited check before it reaches the model.
• Verify outputs explicitly: inputs and outputs are scrutinised so that information about the model and its training data does not leak.
• Least privilege: access to the model is just in time and just enough.

Frequently asked questions

Does a membership inference attack need access to the model’s internals?

No. Query access is enough for the standard attack. Access to the model makes it easier, which is one reason to protect model weights as well as the interface.

Are large language models affected?

Yes. Language models can memorise and repeat parts of their training data, including personal data, and membership-style attacks apply to them as well as to predictive models.

Is differential privacy required?

No. ISO 27090 presents it as one technique with significant trade-offs. Output minimisation, throttling, data minimisation and reduced overfitting should be the first line for most systems.

Which other frameworks cover these attacks?

EU AI Act Article 15 lists confidentiality attacks among the threats high-risk systems must resist. NIST’s AI 100-2e2025 taxonomy classifies membership inference, reconstruction and property inference as privacy attacks. The OWASP list covers the generative side under sensitive information disclosure.

Treating the model as a data store

The practical shift is to treat a trained model as a store of the data it learned from, and protect it accordingly: minimise what goes in, limit what comes out, and watch who is asking. For the integrity side of training data, see our guide to the data poisoning attack; for the full threat list, the ISO 27090 overview.

Our ISO 27090 Toolkit includes dedicated Membership Inference, Model Inversion and Sensitive Model Output standards, an AI Data Limitation Standard, an AI Inference API Protection Standard for throttling, and detection use cases for query-pattern monitoring, all mapped to EU AI Act Article 15 and NIST AI 100-2.