AI privacy budgets: Ask for the calculation, not the claim

Writing a specification requires a precision that a contract does not. This year, I focused on an internet draft for privacy-preserving federated learning in multi-tenant agent systems. The privacy budget could not go in as a number. It had to be presented with the privacy unit, the accounting method, the noise and clipping parameters, and the round cap, because a specification that leaves those implicit cannot be implemented by anyone.

Enterprise contracts quote the number alone.

It usually comes with a reassuring sentence: your data will remain in your tenant, local systems will train on the data, updates to the model will be the only data that is shared, those updates will be aggregated and protected with differential privacy, and the resulting guarantee will have a budget expressed as epsilon and delta.

It can be technically correct. The issue is that the number in the contract is not, on its own, evidence that the model was trained within that number.

Buyers are getting more curious regarding how models were trained and what happens to their data, and finding that most vendors are not prepared to give good answers. For many, the budget expressed is the end of the conversation because a number seems like an answer.

The privacy target can be selected before the training begins. The privacy loss the accountant attributes to the run depends on the entity being protected, the sampling of participants, the number of rounds, the clipping of updates, the amount of noise used, the accounting method and which version of the model is subsequently released.

If these things are not tied together, a vendor can have a real privacy number and a real privacy mechanism that do not relate to the same training run.

The number needs a referent

Start with the privacy unit.

An epsilon of 3.0 does not help you understand if your protected unit is a single record, user, session, device or an entire tenant. That distinction is highly important when dealing with an enterprise system. For instance, record-level privacy can be an inadequate response when one client creates millions of records, and another creates just a few thousand records. A client who requests tenant isolation wants an entirely different answer than a user who is looking to see if the system can distinguish a particular user’s activity.

Then there is composition.

Let’s look at a basic tenant-level configuration example. For this configuration, we have 250 eligible tenants, with 25 drawn per round and 100 training rounds. Let’s adjust one of the parameters, the noise multiplier, and see what happens to the budget.

At 1.1, the run composes to an epsilon around 7.5. If we increase it to 1.5, the epsilon composes to around 4.4. And at 2.0, it’s around 2.9.

Let’s put a stated budget of epsilon 3.0 next to the first configuration, and take a look. The clipping bound, delta and the accountant that composed those numbers are in that worked example too, which is where a buyer should expect to find them. They are illustrative, not intended settings for deployment.

Each value is legitimate on its own, but the problem is that both cannot describe that 100-round run under those assumptions.

The failure mode that enterprise buyers should care about is an internally incoherent privacy claim rather than fake mathematics.

This is where the industry needs to continue the conversation, not where it has to begin it from scratch. Researchers have advocated for quite some time now for the publication of deployed privacy parameters, as well as disclosure of the rationale behind their selection. There is also significant research focused on verifying if an implementation fulfills the epsilon that it claims to guarantee. The translation that has not happened is the translation from research to procurement. Enterprise buyers rarely ask for the artifact that research has been asking for.

The accountant belongs in the release path

Differential privacy systems have accountants that track cumulative privacy loss. The architectural concern is that of the accountant’s jurisdiction.

If after training it writes a value to a dashboard and proceeds to let the release pipeline progress, it is reporting a control failure.

If, in the next training round, the budget would be exceeded and the system prevents that round from starting, the accountant is part of the control plane.

This should also be the case when new models are released. Model promotion should not be an allowed consequence of a file configuration that states the budget is 3.0. The accounting state after the rounds that built the model should determine the release decision.

Here is a simple artifact I would require of each vendor: a privacy release record.

Field

What it proves

Privacy unitWho or what the guarantee protectsTarget epsilon and deltaThe policy promiseAccountant and methodHow composition was computedSampling model and rateHow participants entered each roundNoise and clipping parametersMechanism configurationRounds actually completedThe real run, not the planned oneCumulative epsilon and deltaComposed privacy bound at releaseRelease decisionWhether the model was permitted to shipModel version or hashWhich artifact the claim applies to

The privacy release record: One artifact linking the stated budget to the training run that produced the model.

In a high-assurance system, that record is generated from the training and accounting state and not assembled later for a customer’s request form.

Secure aggregation does not answer who was in the room

The second place these systems become harder is participant selection.

Secure aggregation can prevent the aggregation service from reading individual updates directly, which is valuable. Secure aggregation still doesn’t guarantee that the cohort is secure.

An entity (sometimes a coordinator) decides which tenants are included in each round. If that coordinator is compromised or behaving maliciously, it can impact a victim participant’s surrounding group. This has been studied and published as an attack for reconstructing federated learning with distributed differential privacy and secure aggregation. A malicious server intentionally introduces Sybil participants and decides who trains together. It is an established attack class, not something I discovered. What has not happened is its translation into evidence a buyer can ask for.

That means the cohort is part of the privacy boundary. A CIO contributor recently stated that identity governance was not designed for AI agents, and that governance, rather than authentication, is where programs stall. The same is true one layer below. There is no ownership of the gap between the promise and the fulfillment.

For an enterprise buyer, the more important question is not, “Do you perform secure aggregation?” The more important question is: how is the participant set selected, what is the minimum cohort, and can a subsequent auditor determine who a candidate for the round was without unnecessarily exposing everyone?

Minimum cohort size and selection rule should be included in the training task at the very least. Higher-assurance systems can incorporate committed randomness or verifiable participant-set records.

The idea is not to make procurement teams learn cryptography. The goal is to stop separating scheduling and privacy.

Deleting data does not eliminate its influence

The third gap occurs when a tenant moves out.

Removing data or logs won’t remove the influence an update already had on a deployed model. That is why federated unlearning has become an active research area rather than a retention-policy feature.

There are some valid positions. A provider can retrain from a checkpoint that predates the tenant’s contribution. It can use an approximate unlearning technique. In its risk argument, it may use tenant-level differential privacy bounds. The legal language can state that contributions to models that have been released cannot be removed, and that the tenant withdrawal would only impact future training.

There are different legal meanings to each of these positions, and which, if any, of these positions satisfies a particular deletion obligation is something your legal department must decide. The engineering position is simpler. For the provider, a position must be taken before a deletion request is made, and it must retain enough model lineage to identify which releases contain the departing tenant’s rounds.

Therefore, “We delete your data” is an incomplete answer. The next question is what happens to the models that have learned from it.

Ask for the receipt

These mechanisms are not exotic. Differential privacy has reached a point of maturity. Privacy accountants exist. Secure aggregation exists. Federated unlearning has a growing body of research. Contract guidance already tells you to restrict how a vendor uses your data, and this advice is a good start. Federated learning is the case where the vendor has agreed to all of it, and the exposure can survive anyway, because the protection is arithmetic rather than contractual.

The weak point is the seam between the mechanisms.

Legal is able to negotiate a privacy assurance. A target budget can be set by a privacy team. Sampling can be handled by infrastructure. The number of rounds can change during the process of tuning. A model can be promoted by a release pipeline that another team owns.

The customer sees one number.

So, ask for one record that links a number to a run.

1. Which unit was protected.

   It could be a record, user, session, device or tenant. Without it, the budget is meaningless.

2. At what budget, and computed by which accountant.

   The target epsilon and delta, and the method that composed them.

3. From what parameters, over how many actual rounds.

   Noise, clipping and sampling, against the rounds that actually ran, versus the rounds that were scheduled.

4. At what cumulative loss, and would the release have been blocked beyond it?

   This is the one that distinguishes an accountant that gates from an accountant that reports.

Answered from a single artifact tied to a single model version, the privacy guarantee is operational.

If the budget is from a contract, the training parameters from an architecture deck and the accountant is from a dashboard that nobody can connect to the released model, then all mechanisms may exist, and the claim is still unverifiable.

A privacy budget cannot be simply a value to be negotiated. It must be a runtime invariant that leaves evidence behind.