Observability and audit

This capability note describes runtime-wired observability and audit behavior.

All diagnostic logging entry points apply the configured module/default threshold before the console and file sink thresholds. --debug enables debug-level console output but does not override log_modules.*=info or an explicit module level. See log filtering.

Each client rate-limit rejection emits one rate_limit.exceeded warning, including the cap and effective IP. Both rate_limit.exceeded and request.rejected policy audit records are retained; the second record no longer emits a duplicate HTTP 429 diagnostic. Filtering warnings does not suppress either audit record or change the rate-limit response.

Unauthenticated federation version discovery is served locally without an X-Matrix authorization warning. Remote-media connection and HTTP failure warnings still report failed remote operations; they are not suppressed by an info threshold.

Included now

Startup logs every library each process runs (0.12.45, ADR-0222), at INFO, as plain text with no structured fields: Library <name> <version> (static) or Library <name> <version> (dynamic, <shared object path>; built against <version>) from the server, Thumbnail worker library … for libpng and libjpeg-turbo (or Thumbnail worker libraries not reported: <reason>, at warning when thumbnailing is enabled) and Federation worker library … from the federation worker. The lines carry library names, versions and loader paths only. The thumbnail worker's report reaches the server with every byte outside printable ASCII replaced by ?, so it cannot add a log line.

Push overload emits push.delivery.dropped at warning severity. Its scope is global or recipient for capacity drops, and allocation or admission for resource/launch failures. in_flight, cap and deliveries are numeric counts; capacity fields refer to the selected budget. Resource-failure fields use the global budget. No recipient identity, gateway URL or pusher credential is logged. Logging itself is best-effort on this already-persisted-event path.

Push retries (R14, ADR-0206) log under the rooms logger, never with a gateway URL or pushkey: push.delivery.retry_scheduled (debug) when a failed notification is queued for another attempt, push.delivery.retry_abandoned (debug) when a retry's pusher was removed, and at warning severity push.delivery.retry_dropped when the retry queue (or the recipient's share of it) is full and push.delivery.gave_up after the last retry. Each carries recipient (the user ID); retry_abandoned carries reason, and the other three attempts (attempts made so far). push.delivery.circuit_open (warning) carries gateway_host and recipient when a delivery or retry is skipped because the gateway's circuit is open; the skipped notification is retried.

Email pushers (ADR-0219, ADR-0220) log under the rooms logger with recipient (the user ID), never the email address, the relay URL or credentials: push.email.sent (debug, with unread); push.email.failed (warning, with result and, from libcurl, detail) when an email could not be built or sent; push.email.abandoned (warning, with attempts) after the last retry; push.email.pusher_removed (info, with reason) when a pusher is deleted because its address is no longer the user's 3PID; and push.email.dropped (warning, with reason) when the email queue is full.

• Admin health summaries through /_merovingian/admin/health.
• Admin metrics summaries through /_merovingian/admin/metrics.
• Admin audit summaries through /_merovingian/admin/audit (with ?category= and ?event_type= query-string filters).
• Structured log redaction helpers. contains_sensitive_marker (used by sanitize_query/sanitized_http_target to redact query-string values) recognizes the bare key token as sensitive, in addition to access_token/refresh_token/session_token — needed because GET /_matrix/client/v1/register/m.login.registration_token/validity passes the plaintext registration token as ?token=.
• Stable request-correlation fields (request_id, trace_id, span_id) for local HTTP router diagnostics.
• Health, metric, and hardening snapshot helpers.
• Prometheus text exposition for GET /_merovingian/admin/metrics.
• Durable audit rows for runtime startup, authentication, session, device, key API, room, media, and trust-and-safety actions. Federation audit events are not part of this durable set — see "Federation audit events are not durable" below.
• Durable admin action rows for moderation and trust-and-safety review actions.
• Account-moderation audit rows: account.locked and account.suspended (admin category) are appended when an admin locks/unlocks or suspends/unsuspends a user via /_matrix/client/v1/admin/lock/{userId} or /_matrix/client/v1/admin/suspend/{userId}, keyed by admin actor, target user, and a locked/unlocked/suspended/unsuspended reason code.
• Request-path account-state audit rows: request.user_locked and request.user_suspended (auth category) are appended when an authenticated request is rejected because the caller is locked (M_USER_LOCKED) or suspended (M_USER_SUSPENDED), keyed by actor, target path, and reason.
• auth.password_changed (auth category) records a password change and notes when logout_devices: true revoked the user's other device tokens/sessions.
• auth.sso.login_token.issued and auth.sso.login_token.redeemed (auth category) record the two ends of the SSO m.login.token lifecycle (Matrix v1.19 CS API §"Client login via SSO"): the former when homeserver::complete_sso_login mints a short-lived login token, the latter when POST /login successfully consumes it via homeserver::redeem_login_token. Neither the raw login token nor the resulting access token is logged. sso.redirect.rejected (warning severity, via log_diagnostic) fires when GET /login/sso/redirect[/{idpId}] rejects a redirectUrl that is not covered by server.sso.redirect_url_allowlist — see docs/threat-model.md ("Open redirect and login-token exfiltration via SSO redirectUrl").
• federation.edu.direct_to_device.store_incomplete (warning severity, via log_diagnostic) fires whenever an inbound m.direct_to_device EDU targets one or more devices but fewer messages persist to to_device_messages than were targeted (malformed per-device field, store rejection, or backend write failure). Fields: origin, targeted, stored — never the message content or ciphertext. Previously this failure mode was silent: the sink always returned EduDispositionStatus::accepted regardless of what actually persisted, so a lost megolm room-key share left no trace anywhere in the logs (issue #464).
• key_query.remote_key_query.entries_dropped and key_query.remote_key_query.rejected (warning severity, via log_diagnostic, 0.12.10) fire when the client /keys/query federation proxy discards part or all of a remote /user/keys/query response: entries for users the server was not asked about, devices whose user_id or device_id does not match their position, or cross-signing keys with the wrong user_id or usage. Fields: origin and dropped (a count) — never the key material. A peer that keeps tripping this is either broken or trying to inject E2EE identities for users it does not own.
• federation_proxy.authorization_unparsed (warning severity, via log_diagnostic) fires when an inbound federation request's Authorization header cannot be parsed as X-Matrix credentials, which the proxy answers with 502 malformed federation authorization. Fields: target (sanitized), header_bytes, scheme_ok. The header value itself is never logged — it carries the peer's reusable origin/key/signature credential. Previously this rejection produced no log line at all, so a peer whose PDUs were being dropped was indistinguishable from one that never called.
• Server signing-key lifecycle diagnostics. signing_key.window_refreshed (info) fires when the published valid_until_ts of the active key is rolled forward; signing_key.window_refresh_failed (warning) fires when that write does not persist, meaning peers keep re-fetching against the stale window; signing_key.provider_rebuilt (info) fires when the runtime signing provider is rebuilt because it did not hold the preferred key; and dispatch.signing_identity_refreshed (info) fires from rotate_server_signing_key when the federation dispatch worker is handed the rotated key. signing_key.loaded is debug, not info: it runs on every request path that needs the signing identity. Fields carry server_name, key_id, public_key, and valid_until_ts — never secret material (secret_size is a length only).
• federation.acl_rejected (warning severity, via audit_federation) fires when an inbound federation request, PDU, or room-local EDU is denied by the room's m.room.server_acl (MSC4436). Diagnostic companions pdu.acl_rejected and edu.acl_rejected carry origin, room_id, event_id/edu_type, and reason so operators can trace why a server was blocked without exposing key material. Like every audit_federation event, this row lives only in the in-memory federation audit ring — see below.

Federation audit events are not durable

federation.key_resolution_overloaded records a core inbound transaction whose sender/authorising-key lookup was locally deferred. Its actor is the authenticated origin, its target is the request route, and its reason is a fixed admission message, not key/signature material. It denotes retryable 503 without transaction acceptance, not a relay trust failure; genuine failures in the same transaction retain their separate accounting. This event uses the same bounded in-memory ring, not durable audit storage. See ADR-0133.

federation.pdu_deferred records an inbound PDU that ingest_pdu_event could not yet place and kept for a retry (R03, ADR-0165); its actor is the origin, its target the request route and its reason the ingestion reason. It sits in the same bounded in-memory ring as the other federation.* events, as does federation.pdu_missing_prev_state, which now means the PDU was not kept and the sender was told so in the /send response. The transaction.accepted diagnostic carries pdu_deferred beside pdu_missing_prev.

federation.pdu_deferral_expired records a retained PDU the retry job gave up on: 10 retries did not place it, it was kept for over 24 hours, or its stored text no longer parsed as the event it was kept for. Unlike the other federation.* events it is written through append_local_audit() (policy category) by the retry job, so it reaches the persistent audit_log and GET /_merovingian/admin/audit. Its actor is the origin, its target the event ID and its reason which limit was hit. Expiry also logs a warning with the room ID and attempt count.

audit_federation() (src/federation/inbound_request.cpp) appends every federation.* audit event — federation.accepted, federation.rejected, federation.rate_limited (also emitted when a source address exhausts its bad-X-Matrix-signature budget, ADR-0081), federation.duplicate, federation.membership_rejected, federation.pdu_rejected_auth, federation.acl_rejected, and the rest of that family — to FederationRuntimeState::audit_events, a bounded in-memory deque. It never calls database::append_audit_event. This is a different code path from append_local_audit() (src/homeserver/local_services.cpp), which is what auth, room, media, and trust-and-safety call sites use, and which does write through to the persistent audit_log table.

The practical consequences:

• Federation audit events are lost on process restart.
• GET /_merovingian/admin/audit reads only runtime.database.persistent_store.audit_log (see admin_audit_summary in src/homeserver/runtime.cpp), so it never returns federation.* events regardless of ?category= or ?event_type=.
• The "audit category" column for federation.* rows still follows the same observability::AuditCategory enum (always policy — audit_federation hard-codes AuditCategory::policy), but that category applies to an in-memory row, not a database one.

Failure routing (0.5.0)

Failure call sites use diagnostic/audit helpers that append an audit row at severity warning or above independently of the diagnostic threshold. The companion request.rejected audit row for HTTP 429 is appended directly to avoid a duplicate warning. This is not the full catalogue of audit events — it is only the failures that are audit-routed instead of also emitting a duplicate diagnostic warning for the same event. See "Other durably persisted event families" below for the rest of what append_local_audit records.

Call site	Logger	Audit category	Audit event type
Rate-limit 429	rate_limit	policy	rate_limit.exceeded
Login rejected	auth	auth	login.rejected
Login refused by the failed-login throttle	auth	auth	login.throttled
Access-token rejected	auth	auth	access_token.rejected
Client-server request rejected	client_server	policy	request.rejected
Locked-user request rejected	client_server	auth	request.user_locked
Suspended-user request rejected	client_server	auth	request.user_suspended
Registration policy denied	auth	policy	registration_policy.denied
Federation ACL rejected	federation	policy	federation.acl_rejected

rate_limit.exceeded, access_token.rejected, request.rejected, login.rejected and login.throttled are rate-capped per kind (10 durable rows per 60 seconds, the rest counted): see "Audit volume bounds" below.

Other durably persisted event families

Everything above goes through log_diagnostic_audit, which only routes warning-or-above diagnostics to the audit log. Most durable audit rows are appended directly via append_local_audit (src/homeserver/local_services.cpp) from success and failure paths alike, independent of diagnostic severity. Grepping for append_local_audit call sites finds these event-name families:

• auth.* (src/auth/AGENTS.md-covered flows, via src/homeserver/auth_service.cpp): auth.user_registered, auth.login, auth.refresh, auth.refresh.issue, auth.refresh.rejected, auth.logout, auth.logout_all, auth.account_deactivated, auth.password_changed, auth.openid.request_token, auth.sso.login_token.issued, auth.sso.login_token.redeemed.
• device.* (src/homeserver/auth_service.cpp, src/homeserver/client_server.cpp): device.deleted, device.updated.
• account.locked / account.suspended (src/homeserver/client_server.cpp, admin category) — see "Account-moderation audit rows" above.
• room.* (src/homeserver/room_service.cpp, admin category): room.created, room.joined, room.joined_remote, room.left, room.left_remote, room.invited, room.third_party_invited, room.banned, room.kicked, room.unbanned, room.forgotten, room.knocked, room.event_sent.
• media.* (src/homeserver/media_service.cpp, moderation category): media.upload_rejected, media.upload_quarantined, media.upload_accepted, media.quarantined, media.released, media.removed, media.remote_fetch_accepted, media.remote_fetch_rejected, media.create_reserved (POST /_matrix/media/v1/create, ADR-0212), media.preview_image_stored and media.preview_image_quarantined (a URL preview's og:image, ADR-0213).
• trust_safety.policy_rule.* (src/homeserver/client_server.cpp, policy category): trust_safety.policy_rule.upsert, trust_safety.policy_rule.delete, plus the dynamic event types trust-and-safety policy matches emit via append_policy_audit.
• runtime.started (src/homeserver/runtime.cpp, admin category).
• appservice.outbox.full (src/homeserver/appservice_delivery.cpp, policy category, written through log_diagnostic_audit at warning level): an event was not queued for an appservice because its queue is at appservice.max_queued_events. Actor and target are the appservice id; the reason carries the cap and the number of events refused so far. Emitted for the first refusal and every thousandth after it, so a bridge stuck at its cap does not write a row per event; the exact count is the metric above.

The access_token.rejected row carries a reason that distinguishes the failure mode: token hashing failed, session not found, user not found, and — for an expired-but-not-revoked token (#275) — token expired. The expired reason is emitted only when a session matching the token hash exists, is not revoked, but is past its expires_at, so operators can tell a stale TTL from a genuinely unknown token.

The audit row is keyed by the same actor / target / reason that appears in the structured log line, so an operator can pivot from stderr to audit_log without re-parsing. The log side and the audit row are emitted from the same call site to keep them in lockstep.

Audit sink installation on worker threads (issue #420): the active LocalDatabase* the sink writes through is a thread_local (homeserver/local_services.cpp), installed by LocalDatabaseScope only on the thread that constructs HomeserverRuntime (main). Every call site above that routes through the sink-based observability::log_diagnostic_audit overload (rather than the explicit-database overload) previously silently no-op'd when it ran on an HTTP ThreadPool worker thread — in practice this meant registration_policy.denied was never recorded in production, since /register is dispatched on a pool worker. net::ThreadPool gained an on_thread_start hook; main.cpp's serve_until_shutdown passes a callback that installs the audit database on every worker thread in both the main and sync HTTP pools before they start dequeuing work.

Operator recipe

# All rate-limit hits in the last 24h, by category
curl 'http://127.0.0.1:8008/_merovingian/admin/audit?category=policy&event_type=rate_limit.exceeded'

# All login rejections for a specific user (filter by event_type only)
curl 'http://127.0.0.1:8008/_merovingian/admin/audit?event_type=login.rejected'

# All audit rows
curl 'http://127.0.0.1:8008/_merovingian/admin/audit'

A malformed category= value returns 400 with a clear unknown audit category: <name> error rather than silently dropping the request. Unknown event_type= values are treated as a no-match filter; the response is empty (still 200).

Audit volume bounds (AUTH-1, ADR-0080)

Five audit events are triggered by requests from clients that have proved nothing: access_token.rejected (unknown, invalid or expired bearer token), rate_limit.exceeded and request.rejected (413, 429 and 503 outcomes decided before routing), login.rejected (a password login that failed) and login.throttled (one refused by the failed-login throttle). The last two were at first left ungated on the reasoning that the per-IP limit bounds them, which a flood from many addresses defeats. They must not give such a client control over durable writes or memory, so the audit-append layer bounds the volume.

• Rate-capped rejection rows. append_local_audit passes these five kinds through observability::AuditRateGate. At most 10 durable rows per event kind per 60-second window are written; further events in the window are only counted. The first row written for that kind after the window carries suppressed=<n> appended to its reason (for example session not found suppressed=4990), so an investigator sees that events were dropped and how many. The count is reported once, and only when the kind occurs again; it is not persisted separately. Every other audit event (successful logins, admin actions, room and media moderation, authenticated security events) is written on every occurrence and is never gated. The diagnostic log line the call site emits is unchanged, so a suppressed event still leaves one log line and no audit work.
• Bounded in-memory windows. LocalDatabase::audit_events and PersistentStore::audit_log hold the most recent max_in_memory_audit_events (1 024) rows and drop the oldest first. The audit_log table holds every row. PersistentStore::audit_log_evicted counts rows that have left the window, and audit_events_appended_total reports size + evicted so it stays monotonic. Startup hydration keeps the newest rows in insertion order on both backends: SQLite streams ORDER BY rowid through the window; PostgreSQL fetches the newest max_in_memory_audit_events rows (ORDER BY ctid DESC LIMIT) and replays them oldest first. audit_log has no ordering column (migration 001), so on PostgreSQL the physical position ctid stands in for insertion order; that holds because the table is append-only.
• GET /_merovingian/admin/audit shows the window. It lists the retained rows, and its first line reads audit events=<retained> plus evicted=<n> once older rows have left the window. Query the audit_log table for older history.
• The safety-report listing does not use the window. /_matrix/client/v3/admin/safety/reports calls database::load_audit_events_by_type_prefix(store, "trust_safety.", 1000), which reads the audit_log table (SQLite or PostgreSQL) newest first with the prefix as a bound parameter compared literally (no LIKE), and lists the newest 1 000 rows oldest first. A store with no database backend, or a failed read (logged as audit.query_failed), answers from the window instead. Unauthenticated traffic that fills the window therefore cannot hide an earlier abuse report.
• Bounded field length. actor, target and reason are cut to database::max_audit_field_bytes (255) bytes at database::append_audit_event and append_local_audit, on a UTF-8 character boundary (observability::bounded_utf8). Invalid UTF-8 bytes and ASCII control characters (including NUL and newline) are replaced by U+FFFD, so a value can neither split a multi-byte character, forge a log line, nor put a NUL in a text column. Diagnostic field values passed to log_diagnostic_audit are cut to the same length before they are logged. A login user or device_id of up to the 64 KiB body cap is therefore recorded as at most 255 bytes; login semantics are unchanged.
• Path-only audit targets (OPS-2, ADR-0136). sanitized_audit_target discards the entire query string before logging or storing an audit target, including unknown and percent-encoded parameter names. append_local_audit and remember_local_audit use the same bounded value for debug output and their respective storage paths; construction, INSERT statement generation and summaries enforce it too. Query-free identifiers remain unchanged. Existing database rows, backups and historical logs are not scrubbed by this change. Do not put credentials in paths, actors or reasons: this is a query-removal boundary, not detection of arbitrary secret text.

Diagnostic capture tests must preserve the actual console threshold and whether a module override existed, rather than copying the effective default into a new override. The logger exposes exact threshold and optional module-policy exchanges for scoped restoration. Audit-output fixtures assert synchronous writers, configure levels before redirecting the console buffer, and restore on normal or exceptional scope exit. Capture fixtures also require exclusive access to std::cout and logger configuration: the absence of asynchronous writers does not exclude concurrent synchronous callers.

Security posture

Runtime metrics and audit summaries are bounded operational summaries. They report counts, event types, actors, targets, and reason codes, but do not expose plaintext passwords, bearer tokens, key payloads, media bytes, or event content.

Redaction has one boundary, and the legacy macros go through it. The LOG_* macros passed a caller-built std::string straight to SingleLog, so a call site could bypass redaction simply by not using StructuredLogField. Every named SingleLog method now routes its composed line through redact_log_message, which applies the same sensitivity rule log_field_is_sensitive applies to structured fields, rewriting a sensitive key=value token to key=<redacted>. The LOGF_* macros and the string_format helper they were built on — a runtime std::string forwarded as a printf format argument, CWE-134 — had no call site in src/ and were deleted outright rather than deprecated.

One log record is one physical line (AUTH-9). A logged value such as a login identifier.user or device_id is client-controlled. SingleLog escapes the module name and message of every line it writes, console and file alike, through escape_log_controls before appending the record's own \n:

Input	Written as
line feed, carriage return, tab	\n, \r, \t
any other C0 control (U+0000–U+001F) and DEL	\xHH (lowercase hex), e.g. \x1b
a UTF-8 encoded C1 control (U+0080–U+009F)	\u00HH, e.g. \u009b
every other byte, including the rest of UTF-8	unchanged

A value therefore cannot forge an extra log record or send a terminal control sequence to an operator's console. Backslash is not escaped, so a value that literally contains the two characters \n is indistinguishable in the log from an escaped line feed; the record boundary is still unforgeable. Redaction runs on the escaped line and treats every whitespace character as a token boundary, so a sensitive field at the end of a line keeps the record terminator.

Structured field values are length-capped (AUTH-9). A client can also put a very long value into a logged field, which would inflate every line that mentions it. Every structured field value is rendered by render_log_field_value (src/observability/observability.cpp), the single function diagnostic_message (the log_diagnostic* family, and so every console and file line built from fields) and structured_log_summary use. It redacts first (redact_log_value), then caps (cap_log_field_value), so a truncated value can never reveal part of a secret and a redacted value (the fixed text <redacted>) is already under the cap.

• Cap: max_log_field_value_bytes = 2048. The cap bounds the bytes that reach the log, that is the value after control-character escaping: the cap is measured in escaped width (a \n counts 2, a \x1b counts 4, a \u009b counts 6), so a value full of control characters cannot expand past it. Escaping itself still happens once, at the sink, on the whole line.
• Boundary: the cut never splits a multi-byte UTF-8 character or an escape sequence; a unit that does not fit is dropped whole. A value that fits (exactly 2048 escaped bytes) is emitted unchanged; one byte more is truncated.
• Marker: ...[truncated N bytes], appended directly after the kept prefix, where N is the number of bytes of the original value that were dropped (decimal, no separators). It is plain ASCII with nothing for the escaper to change, so a capped value is at most 2048 bytes plus about 40 bytes of marker. Example: a 1 MiB value of a is logged as 2048 a followed by ...[truncated 1046528 bytes].
• Not capped: field keys, event names, and module names (developer chosen, never client text), and the plain LOG_* message strings that do not go through StructuredLogField. Audit database rows are separate bound parameters, already cut to 255 bytes by append_local_audit.
• There is no JSON or other structured output sink: the console and the log file are the only destinations of a log record, and both receive the line composed by SingleLog::make_log_line.

The event signer no longer logs the canonical signing payload or the signed event JSON. It emits each one's byte count and SHA-256 digest instead, which is still enough to compare byte-for-byte with a federation peer when triaging a BadSignatureError.

Dropped work is counted and announced once. Two paths deliberately shed load rather than block, and both used to do it silently:

• The bounded console and file log queues discard entries once they reach max_log_queue_size. They now keep a drop counter (console_dropped_message_count(), file_dropped_message_count()) and emit one warning per drop episode — a contiguous run of drops — not one per dropped message, which would make the flood worse. That warning is written straight to stderr, bypassing the queue it is reporting as full.
• local_audit_sink no-ops while the thread-local LocalDatabase is unset or closed. It now warns once per episode so an operator can see that durable audit persistence, not just one event, was lost.

The audit sink pointer itself is a std::atomic function pointer. It is written during thread startup and read from arbitrary threads, and a plain pointer gave those no happens-before edge. It stays lock-free, so the read path on every warning-or-above diagnostic is a single atomic load.

AuditLogEvent no longer carries an append_only flag. It was initialised to true and consulted by nothing — audit_log_insert_statement() only ever emits INSERT — so it asserted a guarantee the code did not enforce. Real enforcement is a database trigger rejecting UPDATE and DELETE on audit_log, which belongs in a migration; until that exists, the honest statement is the one above.

The opt-in client_server event=sliding_sync.debug_response diagnostic is available only when the process starts with --debug; it is not an audit event and is never persisted. Its connection identifier data is limited to presence and byte length, and it contains no request bodies, tokens, event content, or encryption material.

Scrape/export contract

GET /_merovingian/admin/metrics is the stable scrape endpoint for operators. It returns 200 OK with Content-Type: text/plain; version=0.0.4; charset=utf-8 and two correlation headers:

• X-Merovingian-Request-Id: req-...
• Traceparent: 00-<trace_id>-<span_id>-01

The body uses Prometheus text exposition with stable # HELP and # TYPE metadata ahead of each metric family. Core families currently exported are:

• merovingian_server_identity{server_name="..."} 1
• merovingian_runtime_started
• merovingian_database_schema_version
• users_total
• sessions_total
• rooms_total
• events_total
• audit_events_appended_total
• admin_actions_total
• merovingian_health_status{component="...",status="ok|degraded|failed"}
• merovingian_federation_outbound_rejected_total: outbound federation transactions refused because the dispatch queue, or the destination's share of it, was full (ADR-0177)
• merovingian_federation_outbound_pending: outbound federation transactions waiting to be sent
• merovingian_appservice_outbox_queued{appservice_id="..."} (gauge: events queued for the appservice), merovingian_appservice_outbox_dropped_total{appservice_id="..."} (counter: events refused because the queue was at appservice.max_queued_events, ADR-0191) and merovingian_appservice_ephemeral_dropped_total{appservice_id="..."} (counter: ephemeral items dropped because the in-memory queue was full, ADR-0190), one series per registered appservice. A non-zero dropped counter means a bridge has been unreachable long enough to lose data and is worth alerting on.
• all media_* repository counters and gauges already tracked by the runtime

Metric names and label keys are ASCII-safe, payload-free operator fields. New families must carry a # HELP line, a # TYPE line, and must not encode secret material in either names or label values.

Trace correlation contract

Structured diagnostics for local HTTP router request handling now carry these fields in the log line itself:

• request_id=req-...
• trace_id=<32 lowercase hex chars>
• span_id=<16 lowercase hex chars>

The contract is intentionally narrow: the correlation identifiers exist to join an operator's scrape or admin query to the structured request diagnostics that served it. They are not a distributed tracing system and are not yet persisted into the durable audit rows.

Deliberately not included

• Distributed tracing beyond the local request-correlation contract above.
• Operator dashboards.
• Retention and export policy for production audit archives.