VaultGemma: Google’s New AI Model Is the First with Differential Privacy

VaultGemma is Google’s new AI model described as the most powerful large language model trained entirely from scratch, fortified with differential privacy protections. The model, designed with one billion parameters and built upon Gemma 2, is released with weights on Hugging Face and Kaggle for developers and academics to use, verify, and refine.

Google reports that differential privacy mitigates memorisation by injecting noise during training, though it can introduce challenges such as reduced training stability and higher computational costs. According to Google, this collaboration with DeepMind marks the first establishment of “scaling laws for differential privacy models,” offering precise predictions of optimal training configurations across varying budgets.

VaultGemma uses a scalable DP-SGD algorithm with an enhanced Poisson sampling method to maintain batch sizes, achieving sequence-level differential privacy of (ε ≤ 2.0, δ ≤ 1.1e-10) and claiming memory tests show it virtually never reproduces training data. Google notes that DP models still trail non-private ones but has narrowed the gap and aims to drive further improvement.