Federated Evaluation and Monitoring

Training a model without seeing the data is the part everyone plans for. Operating it without seeing the data is the part that gets discovered in production: you cannot pull a bad batch, you cannot inspect the sample that produced a wild gradient, and the metric that would tell you a region has regressed is itself computed from data you are not allowed to hold. A federated deployment therefore needs an evaluation and monitoring design of its own, built from private aggregates rather than from logs. This guide provides it, under Federated Learning Workflows for Geospatial Data, and it composes with the aggregation described in secure aggregation protocols.

The scenario: a road-condition model trained across a national fleet of survey vehicles and phones, where a regression in one mountainous region would be invisible in a national average.

Prerequisites

• A federated round that already runs. Evaluation is a second aggregation over the same cohort; it inherits the round’s client selection, secure aggregation and dropout handling.
• A device-side metric budget. Every metric is a release. The evaluation budget must be allocated explicitly, separately from training, and it competes with the training budget for the same per-client allocation.
• Bounded metrics. Every quantity the device reports must be clipped to a known range before it leaves. An unclipped loss has unbounded sensitivity, so one client with a pathological example can both dominate the average and void the guarantee.
• A regional partition fixed in advance. Drift detection needs strata, and strata chosen after seeing the data are a form of adaptive analysis that leaks. Fix the regions from geography or from a public partition, and keep them stable across releases.

Step 1: Move the metric computation to the device

The rule that makes federated observability tractable: the server aggregates numbers, it never computes them. Anything the server would want to compute from raw data must instead be computed on the device and reported as a bounded scalar or a bucketed histogram.

• Loss. Report the mean loss over the local evaluation set, clipped to $[0, L_{\max}]$. The clip is the sensitivity bound; pick $L_{\max}$ from a held-out distribution rather than from the loss function’s theoretical range, which is usually infinite.
• Error distribution. A mean hides the failure mode. Report a fixed-bin histogram of per-example errors — five to nine buckets — so a regression that doubles the tail without moving the mean is visible. Fixed bins matter: data-dependent bin edges are a release of their own.
• Calibration. For a probabilistic model, report the mean predicted probability and the mean outcome per confidence bucket. Miscalibration in one region is a common and otherwise invisible federated failure.
• Counts. Report the local sample count bucketed, not exactly. An exact count is a fingerprint for a small client, and the aggregate only needs the bucket.

Step 2: Aggregate securely, then add noise once

The metrics go through the same pipeline as the gradients: secure aggregation so the server never sees a single client’s metric, then calibrated noise on the aggregate so the published metric is differentially private. Two ordering rules are easy to get wrong.

First, noise is added once, to the aggregate, not per client — adding it per client is local differential privacy and costs $\sqrt{n}$ more error for the same guarantee. Second, the clipping bound and the cohort size together determine the noise scale, so the metric’s error bar is known before the round runs and can be published alongside it.

from __future__ import annotations

import math
import random
from dataclasses import dataclass
from typing import Mapping, Sequence


@dataclass(frozen=True)
class MetricSpec:
    """One reportable metric and the bound that makes it accountable."""

    name: str
    clip_low: float
    clip_high: float

    @property
    def sensitivity(self) -> float:
        """One client's maximum influence on the SUM of this metric."""
        return self.clip_high - self.clip_low

    def clip(self, value: float) -> float:
        return min(self.clip_high, max(self.clip_low, value))


@dataclass(frozen=True)
class AggregatedMetric:
    name: str
    value: float
    error_bar: float
    contributors: int

    def is_significant(self, other: "AggregatedMetric") -> bool:
        """True only when the two differ by more than their combined uncertainty."""
        return abs(self.value - other.value) > (self.error_bar + other.error_bar)


def aggregate_metric(
    client_values: Sequence[float],
    spec: MetricSpec,
    *,
    epsilon: float,
    min_cohort: int = 50,
    seed: int = 0,
) -> AggregatedMetric:
    """Securely-summed, DP-noised mean of one metric across a cohort.

    The noise is added once to the SUM, which is why the resulting error on the mean
    shrinks as 1/n. Adding noise per client instead would be local DP and would cost
    a factor of sqrt(n) in accuracy for the same epsilon.
    """
    n = len(client_values)
    if n < min_cohort:
        raise ValueError(
            f"{spec.name}: cohort of {n} is below the floor of {min_cohort}; "
            "publishing it would expose individual clients"
        )
    rng = random.Random(seed)
    total = sum(spec.clip(v) for v in client_values)
    scale = spec.sensitivity / epsilon
    u = rng.random() - 0.5
    noisy_total = total - scale * math.copysign(1.0, u) * math.log1p(-2 * abs(u))
    return AggregatedMetric(
        name=spec.name,
        value=noisy_total / n,
        error_bar=1.96 * math.sqrt(2.0) * scale / n,
        contributors=n,
    )


def regional_report(
    by_region: Mapping[str, Sequence[float]],
    spec: MetricSpec,
    *,
    epsilon_total: float,
    min_cohort: int = 50,
) -> dict[str, AggregatedMetric]:
    """Per-region metrics. Regions are disjoint, so they compose in PARALLEL.

    Each client belongs to exactly one region, so every region may spend the full
    epsilon — dividing the budget by the region count is a common and expensive
    mistake that makes regional monitoring look unaffordable.
    """
    out: dict[str, AggregatedMetric] = {}
    for i, (region, values) in enumerate(sorted(by_region.items())):
        try:
            out[region] = aggregate_metric(values, spec, epsilon=epsilon_total,
                                           min_cohort=min_cohort, seed=i)
        except ValueError:
            continue                    # region too small to publish this round
    return out

Step 3: Alert on differences that survive their error bars

A federated metric arrives with an error bar that is often larger than the movement people want to alert on, and dashboards that ignore this generate a permanent stream of false alarms. The rule is mechanical: an alert fires only when the change exceeds the combined uncertainty of the two measurements being compared. Everything else is noise, and treating it as signal trains the on-call rotation to ignore the dashboard.

For a regression that is genuinely smaller than the error bar, the remedy is not a tighter alert threshold but a larger cohort, a longer aggregation window, or a bigger evaluation budget — all of which shrink the bar. Which of the three is cheapest is a per-deployment question, and it is worth answering before the first incident rather than during it.

Step 4: Budget the observability, then defend the allocation

Every metric this design publishes is a release against the same per-client budget that training draws on, which makes observability and model quality direct competitors for a fixed resource. Teams that do not allocate explicitly discover the trade the hard way: training consumes the epoch and monitoring quietly degrades to a national average with an error bar wider than any regression it might detect.

Allocate as a fixed fraction, decided once and reviewed like any other parameter. A useful starting split is 70% training, 30% evaluation, adjusted from two observations. If regional metrics regularly have error bars larger than the movements you act on, evaluation is under-funded. If the model’s convergence is materially slower than an unconstrained baseline and monitoring is comfortably precise, it is over-funded. Both conditions are measurable, so the split becomes an empirical question rather than an argument.

Within the evaluation allocation there is a second split that matters more than its size: how much goes to the global metric versus the regional ones. A global loss is one release over the whole cohort and is therefore extremely cheap in relative terms — the noise is divided by a very large $n$. Regional metrics are the expensive ones, but they compose in parallel across disjoint regions, so publishing 30 regions costs the same as publishing one. That parallel composition is the single most useful fact in this design, and it is routinely missed: teams divide the evaluation budget by the region count and conclude that regional monitoring is unaffordable, when in fact each region may spend the full allocation because a client belongs to exactly one of them.

What genuinely costs more is time. Publishing the same regional metric every round is sequential composition across rounds, and it accumulates. The lever here is cadence rather than granularity: publishing regional metrics every fourth round at full precision detects a regression roughly as fast as publishing them every round at a quarter of the precision, and it costs a quarter of the budget. Model the detection latency you need — usually “within a day” rather than “within a round” — and set the cadence from it.

Finally, defend the allocation in writing before the first incident. Under pressure the instinct is to spend the remaining budget on additional diagnostics, which is exactly when the guarantee is most likely to be quietly exceeded. A pre-agreed rule — evaluation may borrow from the next epoch, once, with a named approver — turns that pressure into a decision with an owner instead of an unlogged overspend.

Threat model considerations

• Metrics are releases. A per-region loss computed from client data is a function of that data, and publishing it repeatedly composes exactly like any other release. A monitoring dashboard refreshing every round is one of the highest-cadence releases the system produces.
• Small regions are the disclosure risk. A region with eight contributing clients publishes an aggregate that is nearly one client’s value. Enforce a cohort floor per region and suppress below it, accepting that some regions are unmonitorable at the current fleet size.
• Unclipped metrics void the guarantee. A single client reporting a loss of $10^6$ both dominates the mean and breaks the sensitivity bound the noise was calibrated to. Clipping must happen on the device, before secure aggregation, or a malicious client can manipulate the published metric at will.
• Round-health counters leak too. “How many clients dropped out in region X” is a statement about connectivity in that region, which correlates with weather, outages and events. Coarsen them, and hold them to the same accounting as the model metrics.
• Debugging pressure is the real risk. During an incident someone will propose shipping a build that uploads a failing example “just this once”. That build is a data-collection system with no guarantee at all, and the time to refuse it is before the incident, in writing.

Validation and compliance checklist

1. Every metric has a clip bound. Pass criterion: the metric registry rejects a spec without finite bounds, and CI asserts the device applies them.
2. Noise is applied to the aggregate, not per client. Pass criterion: a test compares the published error bar against the $1/n$ prediction and fails on a $1/\sqrt{n}$ shape.
3. A regional cohort floor is enforced. Pass criterion: no published region has fewer contributors than the floor; suppressed regions are counted and the count is itself coarsened.
4. Metric budget is tracked separately from training budget. Pass criterion: the ledger shows two line items per epoch and the sum respects the cap.
5. Alerts respect error bars. Pass criterion: alert rules reference the published uncertainty, and a synthetic within-bar movement does not fire.
6. Regions are fixed and versioned. Pass criterion: the partition file is under review, and a change to it starts a new comparison baseline rather than silently redrawing history.

Failure modes and remediation

• A metric moves every round and nobody knows why. The error bar is larger than the movement. Publish the bar on the same chart; the “movement” usually disappears visually.
• One region is permanently suppressed. Its client population is below the floor. Merge it with an adjacent region for monitoring purposes — and note in the release documentation that the merged region is the finest granularity actually monitored.
• The mean is flat while user complaints rise. The failure lives in the tail. This is exactly what the bucketed error histogram is for; if it was not shipped, ship it, and accept that the diagnosis waits a round.
• Evaluation budget is exhausted mid-epoch. Monitoring stops, which is worse than a coarse metric. Reserve the evaluation allocation up front and let training back off instead, since a model that trained less is recoverable and a blind fortnight is not.
• Clipping bounds were set once and never revisited. A model that improved now has all its losses far below $L_{\max}$, so the noise scale is much larger than it needs to be. Re-derive bounds from the published distribution periodically — using published aggregates, not raw data.

A last practical note on dashboards. Because every published number carries an error bar that is often larger than the movement people want to discuss, the chart itself has to show the bar — not in a tooltip, not in a footnote, but as the primary visual element. A federated metrics dashboard drawn like a centralised one, with crisp lines and no uncertainty, will be over-read within a week, and the resulting habit of reacting to noise is very hard to unlearn once established. Draw the band first and the line second, and label the cohort size on every panel so a reader can see immediately whether a region’s apparent movement is worth a conversation.

Frequently Asked Questions

Related

• Private Federated Metrics for Spatial Models — the metric set and its calibration, in full.
• Detecting Regional Distribution Drift — turning these aggregates into a drift signal.
• Debugging Federated Rounds Without Seeing Data — the incident playbook.
• Secure Aggregation Protocols — the aggregation these metrics ride on.

Up one level: Federated Learning Workflows for Geospatial Data.