DP-SGD for Geospatial Models

Differentially private stochastic gradient descent (DP-SGD) is how you train a mobility-prediction or land-use classifier on raw location traces and still emit a model with a provable per-subject privacy guarantee. It sits one layer below the gradient aggregation techniques that this page belongs to, and it is the training-time engine of the wider Federated Learning Workflows for Geospatial Data section: aggregation decides how clients combine updates, DP-SGD decides how a single update is made private in the first place. The mechanism has exactly two levers — per-example gradient clipping to a fixed $L_2$ norm $C$, which bounds sensitivity, and Gaussian noise of scale $\sigma = z \cdot C$ added to the summed clipped gradients, whose multiplier $z$ maps to an $(\varepsilon, \delta)$ budget through a Rényi differential privacy (RDP) accountant over $T$ steps at sampling rate $q$. This page is the narrow, geospatial-specific treatment of those two levers: what an “example” is when the record is a trajectory, why non-IID regional data makes the clip bound bite unevenly, and how the mechanism composes with federated learning and secure aggregation protocols.

Parameter Configuration and Calibration

DP-SGD has four privacy-load-bearing knobs plus one modelling decision that is more consequential than all of them combined for spatial data: the unit of privacy. Fix that first, then calibrate the rest against a concrete budget.

• Unit of privacy — event-level vs user-level. If the “example” whose gradient you clip is a single GPS ping, DP-SGD protects events: an adversary cannot tell whether one fix at one timestamp was in the training set, but a user contributing thousands of correlated fixes is not protected as a person. For genuine location privacy the example must be a whole user’s trajectory (one clipped gradient per subject per step), which yields user-level DP — the guarantee compliance actually wants. User-level clipping caps each person’s total influence, so it is the only setting where the $\varepsilon$ you report is the $\varepsilon$ a data subject experiences.
• clip_norm $C$ — the sensitivity bound. After clipping, no single example can move the summed gradient by more than $C$ in $L_2$, so the mechanism’s sensitivity is exactly $C$ and the Gaussian noise is calibrated to it. Set $C$ near the median per-example gradient norm early in training (measure it on a held-out spatial partition); it is a bias knob, not just a privacy knob (see the incident section).
• noise_multiplier $z$ — the privacy dial. Noise std is $\sigma = z \cdot C$, and because sensitivity is $C$, the guarantee depends only on $z$ (and $q$, $T$), never on $C$ itself. Larger $z$ buys smaller $\varepsilon$ at the cost of utility.
• Sampling rate $q$ and steps $T$. $q = L / N$ is the lot size $L$ over the number of subjects $N$; privacy amplification by subsampling means small $q$ sharply lowers the per-step cost, while total spend grows with $T$. The RDP accountant consumes $(z, q, T, \delta)$ and returns the cumulative $\varepsilon$.

Tie it to a concrete budget: a healthcare mobility model over $N = 100{,}000$ patients, target $\varepsilon \le 8.0$ at $\delta = 10^{-6}$ (below $1/N$). With $q = 0.01$ and $T = 4{,}000$ steps, a noise multiplier of $z \approx 1.1$ lands inside that ceiling under an RDP accountant. The same $\varepsilon$ ceiling and the ledger that enforces it are owned by privacy budget management; the noise-to-utility calibration for the additive Gaussian step is worked out in calibrating Gaussian noise for spatial aggregates.

Reference Implementation

The function below is the DP-SGD step in pure numpy: it clips each example gradient independently to norm $C$, sums the clipped rows, adds a single Gaussian draw of scale $z \cdot C$ to the sum, averages over the lot, and returns the parameter update. Clipping must be per-example — not on the batch mean — because sensitivity is defined as the maximum change from adding or removing one subject; a batch-level clip bounds nothing about an individual and the $(\varepsilon,\delta)$ guarantee silently evaporates. The companion RDPAccountant is a leading-order stub for the subsampled Gaussian mechanism; production must defer the authoritative ledger to opacus or tensorflow-privacy.

from __future__ import annotations

import numpy as np
from typing import List, Tuple


def clip_per_example(per_example_grads: np.ndarray, clip_norm: float) -> np.ndarray:
    """Scale each ROW (one example's flat gradient) to L2 norm <= clip_norm.

    Per-EXAMPLE clipping is what bounds sensitivity to `clip_norm`: removing
    any single subject changes the summed gradient by at most clip_norm in L2.
    Clipping the batch mean instead would bound nothing per subject.
    """
    norms = np.linalg.norm(per_example_grads, axis=1, keepdims=True)
    scale = np.minimum(1.0, clip_norm / (norms + 1e-12))  # never scale UP
    return per_example_grads * scale


def dp_sgd_step(
    per_example_grads: np.ndarray,   # shape (L, d): ONE gradient per example/subject
    clip_norm: float,                # C — the L2 sensitivity bound
    noise_multiplier: float,         # z — sigma = z * C
    lr: float,                       # eta — learning rate
) -> np.ndarray:
    """One DP-SGD update: per-example clip -> sum -> Gaussian noise -> average.

    Returns the parameter delta (add it to the weights). The noise is drawn
    ONCE over the summed clipped gradient, with std sigma = z * C, because the
    summed gradient has L2 sensitivity exactly C after per-example clipping.
    """
    lot_size, dim = per_example_grads.shape
    clipped = clip_per_example(per_example_grads, clip_norm)     # sensitivity <= C
    summed = clipped.sum(axis=0)                                 # bounded-sensitivity query
    sigma = noise_multiplier * clip_norm                         # DP noise scale
    noise = np.random.normal(0.0, sigma, size=dim)              # single Gaussian draw
    private_grad = (summed + noise) / lot_size                  # noisy average gradient
    return -lr * private_grad                                    # weight update


class RDPAccountant:
    """Leading-order RDP accountant for the subsampled Gaussian mechanism.

    Tracks cumulative Renyi DP across steps and converts to (epsilon, delta).
    The per-step, per-order bound ~ 2 q^2 alpha / z^2 is the small-q leading
    term (Wang et al.); a production accountant uses the full moments bound.
    """

    def __init__(self, orders: Tuple[float, ...] = tuple(np.arange(1.1, 64.0, 0.1))) -> None:
        self.orders: np.ndarray = np.asarray(orders)
        self.rdp: np.ndarray = np.zeros_like(self.orders)   # accumulated RDP per order

    def step(self, q: float, noise_multiplier: float) -> None:
        """Charge one DP-SGD step at sampling rate q and multiplier z."""
        self.rdp += 2.0 * q ** 2 * self.orders / (noise_multiplier ** 2)

    def get_epsilon(self, delta: float) -> float:
        """RDP -> (epsilon, delta) via the standard tail conversion."""
        eps = self.rdp - np.log(delta) / (self.orders - 1.0)
        return float(np.min(eps))          # tightest order wins

Validation Checkpoint

DP-SGD fails silently: a batch-level clip, an off-by-lot_size noise scale, or a stalled accountant all still produce plausible-looking updates. Assert the invariants in CI.

def _validate() -> None:
    rng = np.random.default_rng(0)
    L, d, C, z, lr = 256, 32, 1.0, 1.1, 0.5

    # 1. Per-example clipping: EVERY row has L2 norm <= C (with float tolerance).
    grads = rng.normal(0.0, 5.0, size=(L, d))          # deliberately over-scale
    clipped = clip_per_example(grads, C)
    assert np.all(np.linalg.norm(clipped, axis=1) <= C + 1e-9)

    # 2. Under-norm gradients are left untouched (clip never amplifies).
    small = rng.normal(0.0, 0.01, size=(L, d))
    assert np.allclose(clip_per_example(small, C), small)

    # 3. Expected noise scale: with zero gradients the update is pure noise of
    #    std = lr * (z*C) / L per coordinate. Estimate over many draws.
    zeros = np.zeros((L, d))
    samples = np.stack([dp_sgd_step(zeros, C, z, lr) for _ in range(4000)])
    expected = lr * (z * C) / L
    assert abs(samples.std() - expected) / expected < 0.1   # within 10%

    # 4. Privacy spend is monotone and accumulates over steps.
    acct = RDPAccountant()
    eps_hist: list[float] = []
    for _ in range(4000):
        acct.step(q=0.01, noise_multiplier=z)
        eps_hist.append(acct.get_epsilon(delta=1e-6))
    assert eps_hist[0] < eps_hist[-1]                        # spend grows
    assert all(b >= a for a, b in zip(eps_hist, eps_hist[1:]))  # monotone
    assert eps_hist[-1] <= 8.0                               # inside the budget

    print("all DP-SGD invariants hold | final epsilon =", round(eps_hist[-1], 3))


if __name__ == "__main__":
    _validate()

Incident Response and Edge Cases

DP-SGD on spatial data degrades in ways that a loss curve alone will not surface. Each pattern below has a measurable trigger and a recovery path.

• Clip bound too low → systematic bias. If $C$ sits far below the true gradient norm, nearly every example is scaled down and the direction of the update survives but its magnitude is throttled — worse, it throttles the largest-norm gradients hardest, which for non-IID spatial data are typically the under-represented rural or maritime regions. The model then converges toward dense-urban patterns. Detection: log the fraction of examples hitting the clip; if it exceeds ~70% for a region, the bound is biting. Remediation: raise $C$ toward the median per-example norm measured on a held-out spatial partition, and recompute the budget only if you also change $z$ (recall $\varepsilon$ is independent of $C$).
• Clip bound too high → noise swamps signal. Because $\sigma = z \cdot C$, a large $C$ inflates the absolute noise while the informative gradient stays fixed, collapsing the signal-to-noise ratio; loss plateaus at a high value and per-round variance stays large. Remediation: lower $C$ to tighten $\sigma$, or reduce $z$ if the budget allows — never both blindly, since one changes utility and the other changes the guarantee.
• Budget exhaustion over epochs. The accountant reports $\varepsilon$ crossing the ceiling before convergence, because spend grows with every step $T$ and does not reset between epochs. Remediation: freeze the model at the last in-budget checkpoint and stop; do not “reset” the accountant between epochs — composition is cumulative and irreversible. If more training is required, raise $z$, lower $q$, or seek a documented budget increase. The detection-and-recovery playbook lives in privacy budget management.
• Event-level clipping mistaken for user-level. Training clips per-ping while the report claims user-level DP; a subject with thousands of correlated fixes is effectively unprotected. Detection: audit that exactly one gradient enters per subject per step. Remediation: aggregate each subject’s per-ping gradients into a single per-user gradient before clipping, or switch to a group-privacy accounting that multiplies $\varepsilon$ by the max fixes per subject.

Frequently Asked Questions

Related

• Gradient aggregation techniques — the parent guide where per-client clipped-and-noised updates are combined across non-IID nodes.
• Handling non-IID geospatial data in federated learning — why regional skew interacts with the clip bound and how to compensate.
• Calibrating Gaussian noise for spatial aggregates — the noise-to-utility calibration behind the additive Gaussian step.
• Privacy budget management — the cumulative epsilon ledger and exhaustion recovery that gate DP-SGD training.
• Secure aggregation protocols — the confidentiality layer DP-SGD composes with in federated settings.

Up: Gradient Aggregation Techniques · Federated Learning Workflows for Geospatial Data