Private Federated Metrics for Spatial Models

The metric set a federated spatial model needs is not the one a centralised model uses. Spatial error is anisotropic — a model that is accurate along a road and wrong across it has a fine RMSE and a useless output — and every metric must survive being clipped, securely summed and noised before anyone sees it. This page defines the metric set, the clipping bounds that make each one accountable, and the aggregation that keeps them usable. It sits under federated evaluation and monitoring in Federated Learning Workflows for Geospatial Data.

Parameter Configuration and Calibration

  • clip_high per metric — the sensitivity bound. Derive it from a percentile of the metric’s distribution on a held-out fleet, typically the 99th, rather than from the metric’s theoretical range. A bound at the 99th percentile clips 1% of clients and keeps the noise scale an order of magnitude below a “safe” bound of infinity-in-practice.
  • bucket_edges — fixed, published, and version-controlled. Histogram edges derived from the current round’s data are a release of that data. Fix them once, publish them with the model card, and change them only with a version bump that restarts the comparison baseline.
  • epsilon_eval — the evaluation allocation. A useful default is 20–30% of the per-client epoch budget. Below 10% the error bars swallow the signal; above 40% the model trains meaningfully slower for monitoring nobody reads.
  • min_cohort_per_region — the publication floor. Set it from the error-bar requirement, not from a privacy intuition: the bar scales as Δ/(εn)\Delta/(\varepsilon n), so the floor is whatever nn makes the bar smaller than the regression you must detect.
Metric Why it is spatial Clip bound from Reported as
Clipped mean loss baseline health 99th pct of held-out loss scalar + bar
Along/across-track error ratio anisotropy fixed at 10.0 scalar + bar
Error histogram tail regressions fixed edges 7 buckets
Calibration gap per bucket over-confidence fixed at 1.0 5 buckets
Coverage: cells with any sample blind spots n/a count, bucketed
Where to put the clip bound A grouped bar chart across five candidate clip bounds taken from percentiles of the loss distribution. Clipping at the maximum requires a bound of 48 and produces noise more than an order of magnitude larger than clipping at the 99th percentile, which clips only one percent of clients. Clipping at the median halves the client population and distorts the metric. Where to put the clip bound loss distribution over a 2,000-client cohort, ε = 1 0 20 40 noise ×1000 / % 0.2 50.0 p50 0.6 10.0 p90 0.8 5.0 p95 1.7 1.0 p99 24.0 0.0 p100 noise on the published mean (×1000) % of clients clipped
The 99th percentile is the usual answer: it clips a handful of clients and keeps the noise an order of magnitude below a “safe” bound at the maximum.

Reference Implementation

python
from __future__ import annotations

import math
import random
from dataclasses import dataclass
from typing import Mapping, Sequence


@dataclass(frozen=True)
class HistogramSpec:
    """Fixed bucket edges. Data-dependent edges would themselves be a release."""

    name: str
    edges: tuple[float, ...]

    def bucketise(self, values: Sequence[float], cap_per_client: int) -> list[float]:
        """One client's contribution: a bounded histogram, L1-normalised to the cap.

        Capping the total mass a client can add is what bounds sensitivity: without
        it, a device holding 10,000 samples moves the aggregate 10,000 times as much
        as one holding a single sample.
        """
        counts = [0.0] * (len(self.edges) + 1)
        for v in values:
            idx = sum(1 for e in self.edges if v > e)
            counts[idx] += 1.0
        total = sum(counts)
        if total > cap_per_client:
            counts = [c * cap_per_client / total for c in counts]
        return counts


def along_across_ratio(
    errors: Sequence[tuple[float, float]], *, clip: float = 10.0
) -> float:
    """Ratio of across-track to along-track RMSE for one client, clipped.

    A road-following model is often accurate along the direction of travel and poor
    perpendicular to it; a scalar RMSE hides that completely. Values near 1.0 are
    isotropic; large values mean the model is drifting sideways.
    """
    if not errors:
        return 1.0
    along = math.sqrt(sum(a * a for a, _ in errors) / len(errors))
    across = math.sqrt(sum(b * b for _, b in errors) / len(errors))
    if along <= 1e-9:
        return clip
    return min(clip, across / along)


def aggregate_histogram(
    client_histograms: Sequence[Sequence[float]],
    spec: HistogramSpec,
    *,
    epsilon: float,
    cap_per_client: int,
    seed: int = 0,
) -> list[float]:
    """Securely-summed, noised histogram.

    Sensitivity is `cap_per_client` in L1, so a single Laplace draw per bucket at
    scale cap/epsilon is the correct calibration. Splitting epsilon across buckets
    is a common over-charge: the buckets are one release, not several.
    """
    rng = random.Random(seed)
    n_buckets = len(spec.edges) + 1
    totals = [0.0] * n_buckets
    for hist in client_histograms:
        if len(hist) != n_buckets:
            raise ValueError("client histogram does not match the fixed edges")
        for i, v in enumerate(hist):
            totals[i] += v
    scale = cap_per_client / epsilon
    out = []
    for total in totals:
        u = rng.random() - 0.5
        out.append(max(0.0, total - scale * math.copysign(1.0, u) * math.log1p(-2 * abs(u))))
    return out


def calibration_gap(
    predictions: Sequence[tuple[float, float]], *, buckets: int = 5
) -> list[float]:
    """Mean(predicted) minus mean(observed) per confidence bucket, per client.

    Each entry is (predicted_probability, observed_outcome). A positive gap means
    the model is over-confident in that bucket — the failure that matters most for
    a routing model, because over-confidence is what gets acted upon.
    """
    sums = [0.0] * buckets
    counts = [0] * buckets
    for p, y in predictions:
        idx = min(buckets - 1, int(p * buckets))
        sums[idx] += p - y
        counts[idx] += 1
    return [sums[i] / counts[i] if counts[i] else 0.0 for i in range(buckets)]

Validation Checkpoint

python
def _validate() -> None:
    spec = HistogramSpec("abs_error_m", edges=(0.5, 1.0, 2.0, 5.0, 10.0, 25.0))

    # 1. A client with many samples must not dominate: the cap is enforced in L1.
    heavy = spec.bucketise([0.2] * 10_000, cap_per_client=100)
    assert abs(sum(heavy) - 100.0) < 1e-6
    light = spec.bucketise([0.2] * 5, cap_per_client=100)
    assert abs(sum(light) - 5.0) < 1e-6

    # 2. Anisotropy must be visible where a scalar RMSE would hide it.
    isotropic = [(1.0, 1.0)] * 100
    sideways = [(1.0, 6.0)] * 100
    assert along_across_ratio(isotropic) < 1.2
    assert along_across_ratio(sideways) > 5.0

    # 3. The ratio must be clipped, so one degenerate client cannot dominate.
    degenerate = [(0.0, 500.0)] * 10
    assert along_across_ratio(degenerate) == 10.0

    # 4. Aggregation must be unbiased at a realistic cohort size.
    rng = random.Random(9)
    clients = [spec.bucketise([abs(rng.gauss(1.2, 0.6)) for _ in range(40)],
                              cap_per_client=40) for _ in range(600)]
    truth = [sum(c[i] for c in clients) for i in range(len(spec.edges) + 1)]
    noisy = aggregate_histogram(clients, spec, epsilon=1.0, cap_per_client=40, seed=2)
    for t, v in zip(truth, noisy):
        assert abs(v - t) < 400.0, (t, v)

    # 5. Histogram buckets are ONE release: epsilon is not divided across them.
    fine = HistogramSpec("fine", edges=tuple(0.5 * i for i in range(1, 20)))
    wide_noise = aggregate_histogram(
        [fine.bucketise([1.0], cap_per_client=1) for _ in range(600)],
        fine, epsilon=1.0, cap_per_client=1, seed=3,
    )
    assert len(wide_noise) == 20
    # The per-bucket noise scale must not depend on the bucket count.

    # 6. Calibration must detect over-confidence.
    overconfident = [(0.9, 0.0)] * 50 + [(0.9, 1.0)] * 50
    gaps = calibration_gap(overconfident)
    assert gaps[-1] > 0.3, gaps

    print("federated metrics: all assertions passed")


_validate()
A scalar RMSE hides lateral drift A grouped bar chart of four models showing along-track error, across-track error and the combined scalar RMSE. Regional model A has an across-track error more than four times its along-track error — it is drifting sideways — yet its scalar RMSE is only moderately worse than the others, which is how the failure passes review. A scalar RMSE hides lateral drift the same four models, three ways of measuring them 0 2 4 error (m) 0.90 1.00 0.95 baseline 0.80 0.90 0.85 after retrain 0.85 4.20 3.03 regional model A 0.90 1.10 1.00 regional model B along-track RMSE across-track RMSE scalar RMSE
Model A is visibly broken on a map and only mildly worse on RMSE. For anything that positions objects, the ratio is the metric that matters.

Assertion 5 is the accounting point most teams get wrong. A histogram’s buckets are a single release under parallel composition — one sample lands in exactly one bucket — so a 20-bucket histogram costs the same ε\varepsilon as a 5-bucket one. Dividing the budget across buckets multiplies the noise for nothing.

Incident Response and Edge Cases

  • The ratio metric pins at its clip for a whole region. Either the model really is drifting sideways there, or the local coordinate frame is wrong — a CRS mismatch on a subset of devices produces exactly this signature. Check the device population’s projection handling before re-training.
  • A histogram bucket is persistently zero. Confirm the edges still match the model’s error scale. A model that improved by an order of magnitude has all its mass in bucket one, and the histogram has stopped being informative; bump the edge version.
  • Calibration gaps look excellent and users disagree. Calibration is measured over the local evaluation set, which is drawn from the same distribution the model was trained on. A model can be perfectly calibrated on the data it sees and badly wrong about the data it never sees — coverage, not calibration, is the metric for that.
  • One region’s metric is far better than every other’s. Check the cohort size. A small cohort’s noisy mean has a wide bar, and an outlier is more likely than an insight. Confirm significance against the bar before acting.
  • Metrics stop arriving from a region entirely. Distinguish “no clients selected” from “clients selected and dropped” from “clients returned and were rejected” — the three have different causes, and only the round-health counters separate them.
A histogram's buckets are one release, not many Two lines of the epsilon available per histogram bucket against the bucket count. The correct treatment — parallel composition, because each sample falls in exactly one bucket — is flat at the full allocation. Splitting the budget across buckets falls to five percent at twenty buckets, needlessly multiplying the noise on every bucket. A histogram's buckets are one release, not many one sample lands in exactly one bucket 5 10 15 20 0 0.5 1 buckets in the histogram ε per bucket one release, full ε (correct) ε split per bucket (over-charge)
Adding buckets to a histogram is free in ε. Splitting the budget across them is the same over-charge as splitting it across grid cells.

Frequently Asked Questions

Why clip metrics at a percentile rather than at a safe maximum?

Because the clip bound is the sensitivity, and sensitivity sets the noise. A generous bound "to be safe" makes every published metric noisier by exactly that factor, and it does not improve privacy — it just wastes accuracy. A 99th-percentile bound clips a handful of clients and keeps the metric usable.

Is the along/across-track ratio worth the extra metric slot?

For any model whose output is used to position something on a map, yes. It catches the specific failure — systematic lateral drift — that a scalar error metric averages away, and it is the failure most likely to make a routing or matching product visibly wrong.

Can I compute a per-cell metric instead of per-region?

Only if each cell clears the cohort floor, which at cell granularity it almost never will. Regions exist precisely because they are the finest partition with enough clients to publish. If cell-level detail is needed, aggregate over a longer window rather than a smaller area.

How do I know the clipping is actually applied on the device?

You cannot verify it from the server, which is why it belongs in the client's tested, separately versioned metric module. Server-side you can only detect gross violations statistically — an aggregate that exceeds cap × cohort size proves clipping failed somewhere.

Up one level: Federated Evaluation and Monitoring · Section: Federated Learning Workflows for Geospatial Data.