feat(relay): per-community usage metrics

[wpfleger96]

What

Emit per-community usage gauges and counter tags to Datadog so teams can answer "how many users / messages / active users does community X have right now" without manual SQL.

Metrics added

DB-polled gauges (new background poller, BUZZ_USAGE_METRICS_INTERVAL_SECS, default 300 s, min 5 s)

Metric	Tags	Query
buzz_communities_total	—	COUNT(*) on communities
buzz_community_users	community, type:human|agent	users where deactivated_at IS NULL; agent = agent_owner_pubkey IS NOT NULL
buzz_community_channels	community, type:stream|forum|dm|workflow	channels where deleted_at IS NULL
buzz_community_messages	community	events kind=9, deleted_at IS NULL
buzz_community_relay_members	community, role	relay_members by role
buzz_community_workflows	community, status	workflows
buzz_community_git_repos	community	git_repo_names
buzz_community_active_users	community, window:1d|7d|30d, type:human|agent|unknown	DAU/WAU/MAU via COUNT(DISTINCT pubkey) on events; unknown = pubkeys with no users row (profileless posters)
buzz_community_active_channels	community, window:1d|7d	distinct channels with ≥1 kind-9 in window

Fleet-wide totals (always emitted regardless of BUZZ_USAGE_METRICS_PER_COMMUNITY)

Metric	Tags
buzz_total_users	type:human|agent
buzz_total_channels	type
buzz_total_messages	—
buzz_total_relay_members	role
buzz_total_workflows	status
buzz_total_git_repos	—
buzz_total_active_users	window, type
buzz_total_active_channels	window
buzz_total_ws_connections	—
buzz_total_users_online_pod	—
buzz_total_subscriptions	—

In-memory snapshot gauges (same poller tick, from live state)

Metric	Tags	Notes
buzz_community_ws_connections	community
buzz_community_users_online_pod	community	pod-local distinct count; sum across pods for fleet total
buzz_community_subscriptions	community

Snapshot pattern used instead of inc/dec to eliminate gauge drift from asymmetric call paths.

Write-path counter tags

community tag added to:

• buzz_ws_connections_total (connection.rs)
• buzz_media_uploads_total (api/media.rs)
• buzz_workflow_runs_total (handlers/event.rs)

New per-community event volume counter (kind-only counters remain fleet-wide to avoid kind × community cardinality explosion):

• buzz_community_events_received_total{community} — per-community event throughput

New counters

• buzz_users_created_total{community} — incremented only on first insert (ensure_user returns true)
• buzz_channels_created_total{community, type} — incremented only on actual creation (ingest.rs was_created path, side_effects.rs new-channel paths)

Bug fix

Kind 44200 (agent turn metrics) now maps to "44200" in bounded_kind_label instead of collapsing to "other".

Cardinality knob

BUZZ_USAGE_METRICS_PER_COMMUNITY controls how many per-community gauge series emit:

Value	Behavior
all (default)	per-community series for every community
off	fleet totals only, no per-community series
top:<k>	per-community series for the k communities with the most messages; fleet totals always on

Fleet-wide buzz_total_* metrics always emit regardless of mode — Datadog cost lever without losing aggregate visibility.

Implementation notes

• ensure_user() return type changed from Result<()> to Result<bool> (true = new insert, false = ON CONFLICT DO NOTHING). All callers updated.
• All rollup queries use GROUP BY community_id — one query per metric family, not one query per community.
• Collect-before-publish: all DB queries run before any metrics::gauge!(...).set(...) call. If any query fails the tick aborts cleanly — no mixed fresh/stale snapshot.
• First-tick jitter: PID-seeded hash gives each pod a start delay in [0, interval_secs) to prevent thundering herd on rolling deploy. MissedTickBehavior::Skip prevents catch-up burst.
• buzz_events_received_total and buzz_events_stored_total keep {kind} only (fleet-wide). bounded_kind_label passes through 10k values in 20000..=29999 (client-controlled); crossing with community would produce millions of series. Use buzz_community_events_received_total{community} for per-community throughput.
• buzz_community_users_online_pod is distinct within a pod (not fleet-distinct). Sum across pods for fleet total, with caveat that multi-pod pubkeys count N times.
• active_user_counts SQL uses a three-way LEFT JOIN classification: human (row exists, owner NULL), agent (row exists, owner NOT NULL), unknown (no row — profileless posters). Old query incorrectly counted no-row pubkeys as human.
• Unmatched enum values in zero-fill filter_maps emit warn! log entries instead of being silently dropped.
• Multi-pod aggregation: DB-derived gauges use max: in Datadog (all pods report same DB value); in-memory gauges use sum: (per-pod partials). Documented in code comments.
• Tag is community (host string), not host — avoids collision with Datadog's reserved infra host tag.

[tlongwell-block]

Reviewed at head b423c8a (CI green, MERGEABLE) against the #1321 tenant model, at Tyler's request. Verdict up front: tenant-safe and correct; the one real risk is metric-series cardinality/cost at the scale this is meant for (thousands of communities).

Tenant safety (vs #1321) — clean

• Every community label value comes from tenant.host() / conn.tenant.host() — the server-resolved TenantContext minted only by fail-closed host resolution. Client input can never mint a label value, so label cardinality is bounded by the communities table, not by attackers.
• The new usage.rs queries are deliberately unscoped (fleet-wide GROUP BY community_id), which would be a fence violation on a request path — but they're confined to the relay-internal poller and never reachable from any client path. Results are keyed back to hosts via the communities table itself. This is the right shape for operator rollups.
• AssertSqlSafe + format! on the interval looks scary but the parameter is &'static str, so the type system pins it to compile-time literals. Fine.
• Read-only throughout; ensure_user() -> bool via rows_affected() on ON CONFLICT DO NOTHING is the correct exact-once signal.

Useful / non-vacuous — yes

• Snapshot gauges instead of inc/dec kills gauge drift (the classic failure mode of this metric family).
• Zero-fill from host_map means absence-of-row reads as a true 0, not a stale last value — and there are regression tests pinning exactly that semantics.
• RELAY_ROLES/CHANNEL_TYPES/WORKFLOW_STATUSES match the actual schema enums (relay_members CHECK is owner/admin/member; channel_type and workflow_status enums match). Verified against migrations/0001.
• Multi-pod max: vs sum: semantics are documented in code. Good.

The real concern: cardinality × cost at thousands of communities

Per community the gauges emit ~25 series (users 2 + channels 4 + messages 1 + members 3 + workflows 3 + repos 1 + active_users 6 + active_channels 2 + ws/online/subs 3). Datadog counts unique tag combos including the pod host tag, so:

5,000 communities × 3 pods × 25 ≈ 375K custom metric series, plus the write-path counters at kind × community (sparse, but real). Datadog list pricing is ~$5/100 series/mo → five figures/month. Even 1,000 communities × 3 pods ≈ 75K series.

Zero-fill makes this worse in one specific way: it guarantees the full product of series always exists — no sparsity savings for dormant communities. That's the correct choice for gauge semantics, but it means cost scales with community count, not activity.

Recommendation (can be follow-up, but decide before running on a big relay): an env knob to control the per-community family — e.g. BUZZ_USAGE_METRICS_PER_COMMUNITY=all|top:<k>|off — with fleet-wide totals always on. Or explicitly accept the cost in writing.

Secondary waste: every pod runs identical DB rollups

The DB-derived gauges are the same value on every pod (hence the max: note), so at P pods you pay P× the query cost for zero information. The event-table aggregates are the expensive ones (all-time kind-9 count scans every partition's index entries; 30d COUNT(DISTINCT) + join every tick). The 300s default and the documented rollup-table escape hatch are reasonable mitigations — but consider leader-gating the DB poll (in-memory snapshots must stay per-pod).

Nits

1. PR body says default 60s; code is unwrap_or(300). Update the body.
2. tokio::time::interval ticks immediately → every pod fires ~10 aggregate queries at boot, simultaneously during a rolling deploy. A small random jitter on first tick would smooth it.
3. Unknown enum values are silently dropped by the filter_maps — if someone adds a channel type or workflow status, its counts vanish without a trace. A warn! on unmatched values would catch the drift.
4. No idle_timeout on the Prometheus recorder, so a community removed from the table keeps its last-value series until pod restart. Minor (communities aren't deletable today).
5. handle_create_group fallback paths increment buzz_channels_created_total on create_channel success — worth one eyes-on that a retried side-effect can't double-count vs the ingest.rs was_created increment (the fallback should be unreachable when ingest pre-created, but the comment itself says "shouldn't happen").

Bottom line

Safe to merge with respect to #1321 — no fence crossings, no client-mintable labels, read-only. Metrics are well-designed and non-vacuous. Before this runs on a relay with thousands of communities, add (or consciously decline) a cardinality control; that's the only thing standing between "useful" and "expensive."