Add double mixture privacy losses

[jan-schuchardt]

Hi,

this is a draft pull request related to issue #274.

I have implemented instances of AdditiveNoisePrivacyLoss whose privacy loss distribution is dominated by a pair of two mixture distributions. The class hierarchy looks as follows:

DoubleMixturePrivacyLoss
├── DoubleMixtureLaplacePrivacyLoss
├── DoubleMixtureGaussianPrivacyLoss
│   ├── MixtureGaussianPrivacyLoss

I have kept the existing MixtureGaussianPrivacyLoss as a special case of DoubleMixtureGaussianPrivacyLoss for backwards compatibility and because it has certain optimizations that are not implemented in its super classes.

Specifically, the DoubleMixtureXYZ classes are less optimized in the following sense:

• No caching of constant terms in the privacy loss
• No specialized heuristic for finding binary search bounds

These optimization cannot be trivially generalized to the super classes, because they rely on the (inverse) likelihood ratio decomposing into a sum of single-distribution ratios. This is not the case when we have two mixtures.

Before I create a final pull request, it would be great if we could discuss the following questions:

• Can we live with the above optimization-related issues? (The implementation was still sufficiently fast for use with DP-SGD in my last project).
• The DoubleMixtureXYZ classes are currently only being tested via tests for MixtureGaussianPrivacyLoss. Do we need separate tests for each class?

[arung54]

Hi Jan, sorry for the delayed response. Wanted to follow up on this:

Can we live with the above optimization-related issues? (The implementation was still sufficiently fast for use with DP-SGD in my last project).

I think it is okay for now, and if someone requests we can work on optimizing later (either internally, or with your help).

The DoubleMixtureXYZ classes are currently only being tested via tests for MixtureGaussianPrivacyLoss. Do we need separate tests for each class?

It would be good to make sure any behavior specific to these classes are being tested. If you feel this is a lot of added work, we can potentially have you submit the current version and I can make a follow-up change that adds more robust tests.

In addition, there are some upcoming changes to the structure of privacy_loss_mechanism that might affect the PR (the amount of work needed to correct the PR for these changes should be pretty small). So it may be best to wait until those changes are out to make a final pull request :)

[jan-schuchardt]

So it may be best to wait until those changes are out to make a final pull request :)

All right, let's do that.

The existing tests do not look to complicated, so I can also take care of those.

[pritkamath]

Hi Jan.

The changes alluded to by Arun have been pushed now (see changes to privacy_loss_mechanism.py in this commit).

Basically, your PR made us realize that it would be more natural to introduce a common class of MonotonePrivacyLoss as a common abstraction of AdditiveNoisePrivacyLoss and MixtureGaussianPrivacyLoss.
Please sync your PR with those changes. Thank you and sorry about the extra trouble.

[jan-schuchardt]

Ok, great. I'm on a vacation / conference travel for the next couple of weeks, but I'll look into doing the rebase after that.