What is the information leakage of an iterative learning algorithm about its
training data, when the internal state of the algorithm is emph{not}
observable? How much is the contribution of each specific training epoch to the
final leakage? We study this problem for noisy gradient descent algorithms, and
model the emph{dynamics} of R’enyi differential privacy loss throughout the
training process. Our analysis traces a provably tight bound on the R’enyi
divergence between the pair of probability distributions over parameters of
models with neighboring datasets. We prove that the privacy loss converges
exponentially fast, for smooth and strongly convex loss functions, which is a
significant improvement over composition theorems. For Lipschitz, smooth, and
strongly convex loss functions, we prove optimal utility for differential
privacy algorithms with a small gradient complexity.

By admin