Skip to content

HTTP transport

The HTTP transport capability is a narrow Matrix API transport boundary, not a general web framework.

Current scope

Implemented now:

  • conservative HTTP request limits
  • request-line size checks
  • header byte limits
  • header count limits
  • method token validation
  • request target validation
  • bounded HTTP/1.1 request-head parsing
  • bounded HTTP/1.1 request-body reads: a continuous minimum body rate (16 KiB/s after a 10 s grace), on top of the per-read poll timeout (see "Slowloris policy" below)
  • a connection dispatcher that holds every connection not being served, so no worker thread waits on an idle or unreadable connection, with a per-client worker share (see "Connection dispatcher" below, ADR-0077)
  • structured request error codes
  • content-length validation
  • transfer-encoding rejection until streaming support exists
  • single-request HTTP/1.1 adapter from parsed request heads to client-server Matrix JSON handlers
  • general-purpose JSON parsing for unsigned payloads retains doubles but rejects signed/unsigned integer values outside the canonical range before signed conversion (integer_out_of_range, HTTP-7); UINT64_MAX cannot become a negative room-tag/account-data value
  • production-named client-server API boundary in client_server.hpp
  • slowloris progress policy scaffolding
  • per-endpoint rate-limit policy scaffolding
  • HTTP request-head fuzz target
  • TCP listener and accept loop via merovingian::net::TcpAcceptor
  • accepted client sockets (both plain-HTTP and TLS accept loops in http_server.cpp) are created with accept4(..., SOCK_CLOEXEC), matching the listening socket, so they cannot leak into a posix_spawn/fork()ed worker subprocess (federation worker, thumbnail worker) while a connection — e.g. a long-poll /sync — is still open
  • RAII signal-safe shutdown via merovingian::net::ShutdownSignal (SIGINT, SIGTERM)
  • per-connection request read, parse, and dispatch via merovingian::homeserver::serve_http
  • dispatch-mode separation so client listeners use the Matrix JSON client_server adapter while federation/internal compatibility paths can keep using the local router
  • OpenSSL-backed TLS server context and connection wrappers, with OpenSSL resolved from the operating-system package
  • TLS listener accept path with bounded handshake timeout
  • per-room striped mutex serialisation of runtime mutation (256-way stripe keyed by room ID; see docs/architecture.md "Per-room inbound PDU ingestion") so independent rooms can prepare, commit, and apply concurrently instead of serialising on one global mutex
  • a dedicated sync_pool (128 threads by default) separate from the main request pool, so long-polling /sync clients cannot starve federation and other short-lived requests
  • self-sufficient CORS emission: every response carries Access-Control-Allow-Origin and Vary: Origin; OPTIONS preflight responses additionally carry Access-Control-Allow-Methods, -Headers, and -Max-Age derived from the runtime's server.cors.* config (0.4.60 preflight; 0.5.30 extended to all non-OPTIONS responses via a single handle_client_server_request boundary). Since 0.12.9 this includes the errors the transport layer answers before routing — parser errors, request timeouts, head-too-large and body-too-large. Those were formatted with no CORS headers, so a browser saw a 400, 408 or 413 as an opaque CORS failure and could not tell it from a network outage. When the head never parsed there is no http::RequestHead to consult, so the Origin is read straight out of the raw head bytes. Access-Control-Allow-Credentials is never emitted on these responses, and apply_cors_headers refuses to pair it with a wildcard origin at all — config::validate already rejects that combination at startup, but a runtime CORS snapshot assembled in process never passes through validation. Reverse proxies must not add their own CORS headers; see docs/user-manual.md Reverse proxy section.
  • response-header validation at both the client-server header assembler and final wire formatter, dropping invalid header names/values instead of emitting them on the wire
  • X-Content-Type-Options: nosniff on every response
  • HTTP/1.1 persistent connections (keep-alive, RFC 9112 §9.3): sequential request rounds over one connection, per-request framing with exact Content-Length body draining, an operator-tunable idle window, a process-wide parked-connection cap, and per-connection request and lifetime caps — see "HTTP keep-alive" below

Not implemented yet:

  • llhttp dependency wrapper
  • request body streaming implementation
  • runtime application of the slowloris progress policy to the request-head read deadline (the head deadline and inter-byte caps in http_server.cpp are the inline enforcement of that policy)
  • HTTP/2
  • HTTP pipelining (more than one outstanding request per connection): pipelined bytes are buffered and answered strictly in order, one response at a time, so request boundaries are never lost

Connection dispatcher (ADR-0077)

A worker thread of the main request pool (server.http.request_threads, default 32) is only ever given a connection that has something to read. Until the 2026-09-29 audit (HTTP-1) a worker stayed with its connection through the wait for the first byte, the wait between keep-alive requests, and every partial read; the pool was a hard-coded 8 threads, so eight sockets from one client — kept alive with one request every idle − 1 seconds, or trickling bodies, or simply connecting and saying nothing — stalled every listener.

homeserver::HttpConnectionDispatcher (in http_server.cpp, over the generic net::ConnectionParker) now owns every connection while it is not being served: a new connection until its first byte, a TLS connection until its ClientHello and again after the handshake until its first request byte, and a kept-alive connection between requests. It holds them all on one thread in one poll(2) set and submits a connection to the pool only once it is readable (or already has input buffered above the socket: pipelined bytes, or TLS records OpenSSL has decrypted — TlsConnection::has_pending_input). The dispatcher enforces the waits itself and closes a connection that is not readable in time:

Wait Bound
New connection (and after a TLS handshake) until its first byte 5 s
Kept-alive connection until its next request server.http.keep_alive_idle_seconds (15 s)

Dispatch is bounded twice:

  • at most request_threads connections are out of the dispatcher at once, so the pool's queue never holds more than one item per worker;
  • at most max(1, request_threads / 4) of them may come from one client address (the TCP peer, keyed by http::client_address_key exactly as the per-IP connection cap keys it). A readable connection whose client is at its share stays in the dispatcher, out of the poll set so it cannot spin, and is dispatched (oldest first) as soon as one of that client's workers returns. Connections from a server.trusted_proxies address are exempt from this share — a reverse proxy carries many clients over one address, and must enforce per-client fairness itself — but not from the global bound.

Ownership: a connection is an HttpConnection behind a std::unique_ptr, owned by exactly one of the dispatcher, one pool task, one sync-pool task or one media-fetch-pool task. Destroying it closes the socket and releases its per-IP slot and any parking reservation, on every path. Threads and locks: the dispatcher thread, the pool workers, the sync pool and the accept threads share only the parker's internal mutex, a leaf lock never held while polling, closing a connection, or calling back into the pool; the parking budgets are atomics. A released worker share wakes the dispatcher through a non-blocking, close-on-exec self-pipe.

Shutdown: src/main.cpp joins the accept threads, then HttpConnectionDispatcher::request_stop() closes every parked connection and joins the dispatcher thread (bounded: the thread is woken at once and does no I/O but close), then the pools drain. A worker that finishes a kept-alive round after that closes its connection instead of re-parking it. The dispatcher's thread starts in start(), after process hardening (ADR-0082).

The TLS handshake still runs on a worker, once the ClientHello bytes are readable; it is bounded by the 15 s handshake timeout and counts against the client's worker share.

Connections accepted but not yet sent their first request are capped at listeners.max_queued_connections (default 1024; 0 disables the cap), which used to bound the pool's queue: beyond it a new connection is closed at once.

HTTP keep-alive

Matrix v1.19 is served over HTTP/1.1, where persistent connections are the default. Merovingian serves each connection as a sequence of request rounds (serve_request_round in src/homeserver/http_server.cpp), each on whichever worker the dispatcher hands the connection to: read one request head, drain exactly its Content-Length bytes, route, write one response, then either close or hand the connection back to the dispatcher for the next request.

Framing decisions (RFC 9112 §9.3, implemented in merovingian::http::connection_preference_for_response):

  • HTTP/1.1 requests default to Connection: keep-alive; a request carrying the close token is answered with Connection: close and the connection is closed after that response.
  • HTTP/1.0 requests default to close; only a request carrying the keep-alive token keeps the connection open.
  • Kept-alive responses carry Connection: keep-alive and the advisory Keep-Alive: timeout=N hint matching the configured idle window. The hint is not a promise: the server may still close early (shutdown) and the client must retry on a new connection.
  • HTTP-8: the response to a connection's 1 000th request, or to any request once the connection is an hour old, carries Connection: close, and the connection is closed after it. Neither is configurable.
  • A response is kept alive only if a slot in the process-wide parked budget (server.http.keep_alive_max_connections) can be reserved for it; the reservation is taken when the response is framed and held until the connection is next dispatched, so the keep-alive header is never contradicted by a full budget afterwards. A parked connection holds no worker, so the cap bounds open descriptors and memory; it need not relate to the pool size.

The next request on a kept-alive connection is subject to the full per-request machinery (head deadline and inter-byte caps, body rate and size caps, rate limits) exactly as on a fresh connection. Bytes read past a request's body (pipelined follow-up requests) are carried into the next round, so request boundaries are never lost.

Sync-pool interaction: a /sync long-poll round is handed to the dedicated sync pool as before, and the main-pool worker (and the client's worker share) is released at once. When the long-poll response has been written and the connection is kept alive, the sync task hands the connection back to the dispatcher, preserving the pool separation (long-poll threads never serve ordinary request rounds).

Configuration (server.http.*, restart required — read when listeners start):

Key Default Meaning
server.http.keep_alive true Enable persistent connections. false restores one-request-per-connection.
server.http.keep_alive_idle_seconds 15 Idle window per parked connection, 1..300.
server.http.keep_alive_max_connections 256 Process-wide cap on connections parked awaiting a next request, 1..4096. A parked connection is held by the dispatcher, not a worker.
server.http.max_connections_per_ip 256 Open connections one client key may hold, 1..65535 (see below).
server.http.ipv6_client_prefix_length 64 Prefix length IPv6 clients are grouped by, 1..128.
server.http.request_threads 32 Threads in the main request pool, 4..256. One client address may hold at most a quarter of them; client-triggered outbound proxying at most half (ADR-0079).

Per-client connection cap (ADR-0072)

Connection admission used to be bounded only by the global queue depth and the global parked keep-alive cap, so one host could open connections just under the slow-request thresholds, fill the global budget and lock everyone else out; the per-IP rate limiter only runs once a request has been parsed. This cap bounds how many connections one client key may hold; it never bounded how many workers (it was eight times the old pool). The per-client worker share of the dispatcher (above) does that.

Both accept loops (serve_http and serve_tls_http, which serve the client and federation listeners) now call admit_connection straight after accept4. The peer address is reduced to a key by http::client_address_key (IPv4 as-is, IPv4-mapped IPv6 as the IPv4 address, other IPv6 masked to server.http.ipv6_client_prefix_length), and http::ConnectionLimiter admits the connection only while that key holds fewer than server.http.max_connections_per_ip connections. A refused socket is closed before a byte is read or a TLS handshake starts, and a connection.per_ip_cap_reached diagnostic is logged (without the address).

The admitted connection's ConnectionLimiter::Slot is RAII and belongs to the HttpConnection, so it travels with the connection wherever it goes (the dispatcher, a pool task, the sync-pool long-poll) and is released when the connection is closed, on every path.

Addresses listed in server.trusted_proxies are exempt: a reverse proxy carries many clients over its own address, and per-client limiting there is the proxy's job.

Direct serve_one_http_connection callers (tests, one-off embeds) keep the historical one-request-per-call contract: with no dispatcher the policy disables keep-alive and the round is answered with Connection: close.

Response-header safety

Runtime-generated response headers are validated with the shared HTTP header grammar before they are stored or written to the wire. This prevents CR/LF and other invalid octets from being reflected through CORS or future dynamic header surfaces. The wire formatter also injects X-Content-Type-Options: nosniff when the response did not already set it.

Outbound HTTP client

merovingian::http::OutboundClient is the federation outbound HTTP boundary.

The public surface comprises OutboundRequest, OutboundResponse, OutboundResult, OutboundError, the pure validate_outbound_request helper, and the OutboundClient class itself. The client is stateless and holds no per-instance resources; operations report failures through OutboundResult rather than exceptions.

A single OutboundClient instance is safe to share across threads. The runtime hands one instance to both the federation dispatch-worker thread and the HTTP request-handler thread pool. A libcurl easy handle must never be driven by more than one thread at a time, so perform() uses a per-thread handle: each thread lazily creates its own handle on first use and frees it at thread exit. Because every call resets the handle before configuring it, the handle is reused across calls (preserving per-thread connection and TLS-session reuse) without leaking state between requests. Sharing a single handle across threads previously caused intermittent network_error failures on federation key queries that broke E2EE.

The validator enforces the security invariants that hold regardless of backend choice:

  • the request method must be a known token (GET, POST, PUT, DELETE)
  • the URL must be an absolute https:// URL with a host segment
  • at least one address must be supplied in pinned_addresses; the client does not resolve hostnames so the SSRF policy in merovingian::federation::security remains the single source of truth

Federation discovery applies security.federation.deny_ip_ranges in addition to the mandatory built-in SSRF blocks. The discovery network owns this immutable policy, so IP-literal, explicit-port, well-known, SRV, fallback, redirect and filtered URL lookups retain it on both cached and uncached paths. Main and federation-worker startup wire the same policy. Configured ranges cannot enable loopback or private destinations. The server-name discovery cache retains at most 1024 positive and negative entries combined, with a 60-second freshness TTL; when full, a new name evicts the oldest retained result. Expired entries are swept before cache hits and insertion. Invalid server names are refused before network work and never cached; concurrent admissions obey the same cap. A zero constructor capacity disables caching.

OutboundRequest::allow_cleartext_http (added 0.12.1) is a narrow, opt-in escape hatch from the https-only rule above, defaulting to false for every existing caller (federation, the push-gateway client, the identity-server client all stay https-only with no way to override it). Setting it to true additionally permits http:// for that one request. There are exactly two sanctioned callers, matching the comment on the field in include/merovingian/http/outbound_client.hpp:

  1. appservice::AppserviceClient (src/appservice/appservice_client.cpp): an appservice registration file's url is operator-configured — a local filesystem artifact the operator wrote, not something a network peer can influence — and the Application Service API's own canonical registration example (and most real-world bridges) gives a plain http://127.0.0.1:... URL. This caller also intentionally bypasses CachedServerDiscovery's private/loopback-address rejection (via .upstream().lookup_addresses() directly) for the same reason: that rejection exists specifically for attacker/client-influenced destinations, which an appservice URL is not. The bypass is AppserviceClient's own code, not an effect of the flag.
  2. A URL preview hop (ADR-0213), made only through gated_fetch in src/homeserver/url_preview_service.cpp: the page, each redirect it follows itself, and the og:image. The URL is client-named, so the opposite of the appservice case applies: every hop's host is resolved and refused if any address is private, loopback, link-local, CGNAT, reserved or in security.federation.deny_ip_ranges, and the connection is pinned to exactly the checked addresses (CURLOPT_RESOLVE plus the per-socket peer check), so a second DNS answer cannot redirect it. The request carries no credential, and its response is treated as hostile whether it came over TLS or not. http is allowed only so that cleartext sites can be previewed; it adds no destination the gate would otherwise refuse.

Every other caller (federation, the push-gateway client, the identity-server client) must leave it false. TLS verification (CURLOPT_SSL_VERIFYPEER/VERIFYHOST) is unaffected — still on unconditionally whenever the connection does end up being TLS.

perform() is libcurl-backed. Each request runs with the following non-negotiable defaults so federation traffic cannot regress its security posture:

  • CURLOPT_SSL_VERIFYPEER = 1 — reject untrusted certificate chains
  • CURLOPT_SSL_VERIFYHOST = 2 — require the certificate to match the URL host
  • CURLOPT_FOLLOWLOCATION = 0 — redirects are refused
  • CURLOPT_PROTOCOLS_STR = "https" — no cleartext fallback, unless the request set allow_cleartext_http = true (see above), in which case "https,http"
  • CURLOPT_NOSIGNAL = 1 — signal-driven resolution disabled so timeouts remain safe across threads
  • CURLOPT_CONNECTTIMEOUT and CURLOPT_TIMEOUT driven by the request fields
  • CURLOPT_RESOLVE populated from pinned_addresses so the connection is locked to addresses validated by the federation security policy

Transport and media redirects use libcurl's URL API with path normalization disabled. Credentials, fragments, authority escapes, controls, backslashes and IPv6 zones are refused. The same URL handle supplies the transfer authority and pin key; one resolve entry retains every approved address. An open-socket callback checks the actual numeric peer and port before opening a socket. Environment proxies and socket reuse are disabled so each request checks its own pin set; deployments need direct outbound access. TLS session caching remains. See ADR-0097.

The response body is captured up to max_response_body_bytes. The write callback guards against unsigned underflow: it checks body.size() >= cap before evaluating bytes > cap - body.size(), preventing wrap-around when the accumulated body already meets the cap. Oversized responses abort the transfer and surface as response_too_large. A 3xx response surfaces as redirect_rejected with the status and headers preserved on the result for audit logging.

Response headers are likewise capped (issue #413): at most 256 headers and 64 KiB of cumulative header bytes are stored; libcurl imposes no default limit on either, so a hostile peer streaming an unbounded header count could otherwise trigger a bad_alloc that escapes the noexcept header callback and calls std::terminate, aborting the whole process. Exceeding either cap aborts the transfer (response_too_large); allocation failures inside the callback are also caught and mapped the same way instead of propagating.

libcurl error codes map onto OutboundError: TLS verification failures collapse to tls_verification_failed, connect/resolve failures to connection_failed, timeouts to timeout, and the catch-all is network_error.

The TLS backend is whatever the system libcurl was built against. A per-platform integration suite (Linux, FreeBSD, OpenBSD) is wired up in slice 3 alongside the federation outbound transaction integration so backend drift surfaces in CI rather than at runtime. The subprojects/curl.wrap fallback is deferred until a known-good WrapDB release is pinned.

Identity Service API client

The identity module's IdentityServerClient is a client of the Matrix Identity Service API, not a federation peer, so it authenticates with a bearer id_access_token (not X-Matrix federation auth). It issues store-invite (third-party invite issuance), lookup, bind, unbind, and requestToken calls to a remote identity server over the same SSRF-safe OutboundClient + CachedServerDiscovery resolver used for federation: hostnames resolve through CachedServerDiscovery with pinned addresses, and private/loopback addresses are rejected so a misconfigured or hostile IS hostname cannot redirect the homeserver onto an internal network. Trusted IS hosts are configured via server.identity_server.trusted_servers; the homeserver fails closed (403) when an id_server named in a 3PID invite is not in that list. The store-invite call is performed outside runtime.mutex so an unreachable identity server cannot block unrelated room mutations. The same resolver, trust gate, bearer-auth, fail-closed, and release-mutex-for-network conventions apply to the bind, unbind, and requestToken handlers (v0.11.10); see docs/auth-identity.md.

Identity discovery test-seam (test_forced_identity_resolution)

IdentityServerClient accepts an optional std::map<std::string, identity::TestForcedIdentityResolution> const* forced_resolution (defined in include/merovingian/identity/identity_client.hpp, stored on HomeserverRuntime::test_forced_identity_resolution and keyed by IS host). When an entry exists for the target host, perform() uses the entry's pinned_addresses and in-memory trusted_ca_pem and skips CachedServerDiscovery entirely, so a self-signed local mock IS listening on 127.0.0.1 can be reached over real TLS without weakening the production SSRF path — the map is empty in production and has no production construction path. Unlike the federation seam, the identity seam carries no resolved_port: the IS base URL already names the port, and OutboundClient builds the CURLOPT_RESOLVE entry host:port:address from the URL's host:port and the seam's address, so the mock IS port must match the URL/id_server the homeserver is configured to call. This keeps the homeserver → identity dependency direction correct (the seam struct lives in the identity header, not runtime.hpp) and lets the conformance/integration suites exercise store-invite, bind, unbind, and requestToken hermetically.

TLS listener boundary

TLS is a runtime listener boundary, not a replacement for the HTTP parser. The listener accepts TCP, upgrades the accepted socket through merovingian::homeserver::TlsServerContext, then passes a stream abstraction to the same bounded HTTP/1.1 request path used by cleartext loopback listeners.

Configuration enforces TLS on any public (non-loopback) listener. A loopback listener may only run in cleartext when the operator explicitly declares it is behind a local reverse proxy (reverse_proxy=true). Public listeners must set reverse_proxy=false.

TLS startup fails closed when OpenSSL cannot initialise, load the certificate chain, load the private key, or verify that the private key matches the certificate. Handshakes use a bounded timeout aligned with the current per-connection read deadline. The server currently enforces TLS 1.2 or newer and keeps connection lifetime to a single HTTP request.

The client socket stays non-blocking for the life of the connection, not just for the handshake (fixed in 0.12.7 — previously accept_tls_connection restored blocking mode on the success path, and TlsConnection::read called SSL_read_ex directly on the now-blocking socket). TlsConnection::read and TlsConnection::write both route through one private pump() that retries the SSL call and, on SSL_ERROR_WANT_READ/WANT_READ, polls for readiness against the connection's own deadline (inherited from the handshake timeout), returning failure once that deadline expires rather than letting the underlying recv/send block in the kernel. pump() retries with the same buffer and length across iterations, as OpenSSL requires after WANT_READ/WANT_WRITE — advancing the buffer would be a protocol violation. See ADR-0054 for the full rationale and the rule this sets for future code: nothing may put a TLS client socket back into blocking mode, and no code below the HTTP layer may perform a blocking I/O call on a connection descriptor.

The cipher list configured with SSL_CTX_set_cipher_list is restricted to authenticated ephemeral (ECDHE/DHE) AEAD suites: ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:!aNULL:!MD5:!RC4:!3DES:!RSA:!SHA1. The previous HIGH:!aNULL:!MD5:!RC4:!3DES still permitted plain-RSA key exchange, so an attacker who recorded traffic and later obtained the server's private key could decrypt it retrospectively. This list governs TLS 1.2 and below only; TLS 1.3 ciphersuites are negotiated separately from OpenSSL's compiled-in defaults, which this code never overrides.

Plain-HTTP sockets obey the same rule

The non-blocking rule above is not TLS-specific, and since 0.12.9 the plaintext path obeys it too. Accepted client sockets carry SOCK_NONBLOCK from accept4 itself, so the descriptor is non-blocking from the instant it exists rather than from the instant a worker picks it up. PlainConnectionStream::read and ::write retry on EAGAIN against a deadline, and the sync-pool's direct send_all(fd, ...) path waits for POLLOUT the same way.

Reads on a plaintext socket were already bounded by the HTTP layer's poll(POLLIN) — for a plaintext socket, unlike a TLS one, readable bytes really do mean a readable request. Writes were not: a peer that accepted a connection and then stopped reading parked a worker inside ::send() for as long as it liked, and one such client could exhaust the pool.

OpenSSL is the selected TLS provider for this boundary. The project-owned wrapper keeps OpenSSL-specific types out of higher-level transport code, which contains provider maintenance without making provider replacement part of the current plan. OpenSSL is dynamically linked from the host package manager so TLS security updates can arrive through normal distro and BSD package channels.

Request limits

Default request limits are intentionally conservative:

Limit Default
Start line 8192 bytes
Headers 32768 bytes
Header count 100
Body 1048576 bytes

The parser rejects oversized or malformed request heads before any endpoint handling.

Structured errors

Request parser failures use stable error names and HTTP statuses. Oversized start lines, headers, header counts, and bodies map to 413. Malformed request lines, invalid methods, invalid targets, and invalid content lengths map to 400. Unsupported transfer encoding maps to 501 until streaming support exists.

The public client-server and internal local-router entry points also preflight the path portion, before the first ?, for literal NUL and %00 (DB-6). They return 400 M_INVALID_PARAM before a path-component decoder or handler runs; the public refusal retains configured CORS headers and does not reflect the untrusted target. The shared allocation-free core::path_component_contains_nul check preserves one-pass decoding: %2500 remains a literal %00, not NUL. This is not a query/body/binary payload filter. Recognized via and server_name join-server query values are checked separately before their path-style decoding: NUL returns 400 M_INVALID_PARAM, rather than an exception escaping either the public join handler or the direct-local room join route. Unknown query values and once-encoded percent signs are not blanket-rejected. Send, redact, state GET/PUT, report and typing parsers strip the query before extracting or decoding path components. Unrelated ordinary, %00, %2500 and literal-NUL query values cannot change transaction scope, state keys, reported event IDs or typing subjects. The authenticated SQLite regression first failed 48 / 174 checks, then passed 230 / 1 after path/query separation. The join-hint regression first reproduced direct-local exceptions (16 / 108), then passed 188 / 1 with memory and reopened durable business snapshots. Unrelated std::invalid_argument exceptions are not caught and relabelled as malformed paths. The no-proxy handle_federation_http_request entry point also checks the path before routing or verified-worker replay and emits a structured Matrix error without reflecting the target. Real authenticated loopback socket-HTTP coverage checks nine malformed room/member/event ID cases, complete 400 responses, keep-alive with 200 controls on the same socket, %2500 compatibility and unchanged memory/reopened SQLite rows and watermarks (246 assertions / 1 case after a genuine 16-failure RED). Proxy routing runs before that dispatcher. FederationProxy::handle therefore applies the same narrow guard before bypass/authentication/shard-key decoding. A second real socket scenario uses a healthy separate worker, durable startup fixtures and no failed-room-sync fallback: genuine RED failed 7 / 293, then GREEN passed 311 / 1 (both transport cases together: 557 / 2). It also checks structured M_INVALID_PARAM, same-socket controls, one-pass compatibility and unchanged business rows/watermarks. Independent source review approves this federation slice. The five client query/path and direct-local join-hint repairs passed bounded independent review with reviewed-source hash binding. The subsequent review-gap full CLI completed exit 0: 56 OK, no failures or timeouts, fresh text/JSON reports, all return codes zero and 127 unchanged snapshotted files. This checkpoint does not accept subsequent edits. Further tracing found raw query text reaching to-device transaction IDs, knock room IDs and shared user-metadata suffixes. Those boundaries now take the path before the query separator. The additional real authenticated SQLite regression failed 34 / 169 checks before the repair, then passed 255 / 1 after a declaration-order build failure was corrected. It checks original to-device transaction replay and recipient delivery, intended account-data/tag keys (including read/delete), and durable knock membership. The current normal DB-6 integration group passes 1595 / 12 with unchanged socket and earlier admission/query groups. Isolated real PostgreSQL URI-only and URI-plus-file reruns after this repair each pass 178 / 2 with verified fixture cleanup, cluster shutdown/removal and unchanged sources. The extended review found no blocking issue in the additional three-boundary repair, but its decoder-completeness check failed: profile GET/PUT, shared room action/receipt suffixes and presence still use raw target slices. Query-contained endpoint markers can reach their decoders; ordinary receipt query text can change the recorded event ID. Parent source inspection corroborates these pre-existing gaps. A separate registered real SQLite scenario then failed 206 / 691 assertions after a complete no-query control passed; every nonempty query/marker case failed. Profile GET/PUT, shared room suffixes (including receipts) and presence now take the existing path-only request view. The new scenario checks correct profile/presence rows after reopen, canonical whole state, exact in-memory receipt event IDs, durable fully-read markers and leave-last membership. GREEN passes 733 / 1; current normal DB-6 integration passes 2328 / 13, with prior groups unchanged. Current isolated PostgreSQL admission reruns each pass 178 / 2 in URI-only and URI-plus-file modes, with verified cleanup/teardown and all 22 then-tracked sources matching executed integration/unit/database snapshots. The subsequent review rejected an introduced /members query-loss regression. Its real-dispatch conformance RED failed 50 / 164 checks; extracting supported queries separately from the original request, while keeping the path suffix clean, passes 164 / 1. A separate RED 16 / 73 exposed a pre-existing intersection of combined member filters; the pinned specification requires union, now passing 73 / 1. Current history conformance passes 1816 / 17, normal DB-6 integration remains 2328 / 13, and both current isolated PostgreSQL modes pass 178 / 2 with verified teardown. All 23 current admission sources match conformance/integration/unit/both-database snapshots; scoped static findings are unchanged. Fresh member-repair source review passed with no security or logic findings; reviewed production/tests match executed GREEN. The postmembers full checkpoint subsequently passed 56 OK targets, all return codes zero and no failures/timeouts, with 129 unchanged snapshotted files. That is acceptance of its earlier snapshot only.

The remaining nondecoder device GET/PUT/password-UIA DELETE and backup-version DELETE handlers now select IDs from the path, never the unrelated query suffix. Real public device login establishes the client cache as well as durable rows. After healthy no-query controls, device RED failed 56 / 216 assertions and backup RED failed 24 / 181; repaired scenarios pass 224 / 1 and 181 / 1. They check intended identities, SQLite reopen, target-session revocation without losing the owner's device/session, preservation of another backup version and repeated deletion. Current normal DB-6 integration passes 2733 / 15, history conformance remains 1816 / 17 and unit groups remain green. Both isolated PostgreSQL admission/compatibility modes pass 178 / 2 with verified cleanup/teardown; all five completed execution manifests match the same 23 current sources. Bounded independent source review of this addition passed with no security or logic findings. The fresh combined checkpoint passed 56 OK targets with all return codes zero and no failures/timeouts, verified against 130 unchanged snapshotted files including the focused sources. The source freeze ended after acceptance. Exact UIA challenge structure and stronger reopened owner-token and survivor-metadata comparisons remain nonblocking coverage suggestions; this is not whole DB-6 or release acceptance. Other room-action branches, presence GET, arbitrary encoded device IDs, backup encryption/session cascade and new live-PostgreSQL/wire coverage of these operations are not established by these SQLite scenarios. DB-6 is not closed. This is not TLS, hardened-worker, raw-wire-NUL, query/body, or all-route socket coverage; see proposed ADR-0137.

Slowloris policy

The slowloris guard tracks bytes received versus elapsed time using:

  • minimum bytes per second
  • grace period
  • header deadline

The request-head read applies the equivalent deadlines inline (request_head_deadline, inter-byte cap, per-recv poll timeout in http_server.cpp); a request head that dribbles bytes is dropped with a 408 once any bound is exceeded.

Request-body reads are bounded by a continuous minimum body rate (2026-09-29 audit HTTP-1, ADR-0077): once a grace of 10 s has passed since the body read started, the bytes received so far (including any that arrived with the head) must be at least 16 KiB/s × (elapsed − 10 s). The moment the count falls behind that line the read ends, the server answers 408 and closes. An honest upload at or above 16 KiB/s never trips it, whatever its size; a client that stalls is cut off about 10 s into the body. The size caps (the 1 MiB transport cap, or security.media.max_upload_size for an authenticated upload) still bound the body absolutely.

History: before 0.12.7 a body read had only a fresh 15-second poll per 4096-byte chunk, so a dribbled 1 MiB body could park a thread for 65 minutes. 0.12.7 added a 5 s inter-byte cap and a total deadline of 30 s plus the declared length at 16 KiB/s. That still let one byte every 4.9 s hold a worker for about 94 s per POST and for most of an hour on an upload declaring the 50 MiB maximum, which is why the rule is now continuous.

A media upload declaring more than the 1 MiB transport cap is read under the larger max_upload_size cap only when its request head carries an access token that authenticates (homeserver::media_upload_authentication_refusal: a live session, or an application service's as_token). Otherwise the server answers 401 M_MISSING_TOKEN or M_UNKNOWN_TOKEN (with soft_logout for an expired token, and CORS headers) from the head alone and closes the connection without reading the body (HTTP-1, HTTP-6).

Every cap must bound the poll that waits on it. The caps above are evaluated between reads, so a recv allowed to outlast one makes that cap unenforceable. recv_with_timeout therefore takes a poll budget, and both the head loop passes the smallest of the per-read timeout, the overall deadline and the remaining inter-byte allowance, and the body loop the smaller of the per-read timeout and the moment the body rate rule would trip; budget expiry is reported distinctly from a peer close so the loop re-checks and the cap that actually expired ends the request and is the one logged. This was not a hypothetical: until 0.12.7 the poll was a fixed 15 seconds while the inter-byte cap was 5, so the inter-byte cap could never fire on either the head or the body, and a client stalling mid-request was still released only at the 15-second poll. When adding a new cap to either loop, add it to the budget as well or it will not take effect.

A body deadline is inert on a TLS listener unless the socket beneath it cannot block. poll(POLLIN) proves that TCP bytes are available, never that a complete TLS record is: a peer that sends part of a record and stops makes the socket readable, so poll returns immediately, the TLS read is entered, and — on a socket restored to blocking mode after the handshake — the call can block in the kernel indefinitely, past every deadline the request-head and request-body logic believe they are enforcing. This is why the request-body deadline (since replaced by the body rate rule) shipped together with the change described in "TLS listener boundary" below: fixing one without the other leaves the fixed one meaningless on TLS listeners. See ADR-0054.

A connection waiting for its first byte or its next keep-alive request is held by the dispatcher (ADR-0077), bounded only by the first-byte timeout or the keep-alive idle window — a quiet connection is not a slow client, and it costs no worker. The phase-aware http::connection_should_close expresses the same composition as a policy function. Once bytes arrive, the head and body bounds above apply in full.

Rate-limit policy

Runtime rate limiting is enforced on client-server requests and authenticated inbound federation traffic before dispatch. The public federation version endpoint, GET /_matrix/federation/v1/version, is served locally without X-Matrix authentication or rate limiting, as specified by Matrix v1.19. Its implementation version comes from the Meson project version. Two independent wall-clock token-bucket tiers are maintained:

  • Per-IP, keyed by (effective_client_ip, normalized_route).
  • Per-user, keyed by (authenticated_user_id, normalized_route) for requests that present a valid access token.

Client-server endpoints the spec documents as "Rate-limited: No" bypass both tiers in allow() (client_server.cpp): the GET /thirdparty/* lookups, an appservice's own ping and network-directory calls, and every push-rule endpoint except PUT /pushrules/global/{kind}/{ruleId}, which is "Rate-limited: Yes" and keeps its bucket.

Both bucket tables hash their keys with http::BucketKeyHash — keyed BLAKE2b through crypto::generic_hash_bytes, under a per-process random key — not std::hash<std::string>. Bucket keys are attacker-influenceable, and the default hash is neither collision-resistant nor randomised per process, so a single precomputed collision set would degrade the table into a linear scan on every deployment: the structure the server uses to defend against floods would become the target of one. kMaxBucketsPerTable still bounds table size; the hash bounds the cost of each lookup within it.

A quiet server does not freeze a bucket because the window rolls over on elapsed real time, not on request count. When a cap is exceeded the server returns 429 M_LIMIT_EXCEEDED with a Retry-After header (seconds). The deprecated retry_after_ms body field is also included for older clients. A 429 does not tear down a keep-alive connection: connection framing is decided per request round and is status-independent (see "HTTP keep-alive" above), so a throttled client can wait out its window on the same connection and retry.

Route tiers

Every client-server route is classified into one of six explicit tiers by the prefix table in http::rate_limit_tier_for() (src/http/rate_limit.cpp) — one greppable place, no magic. Path parameters such as roomId, deviceId, and mediaId are coalesced into placeholders by normalized_target() so the same cap applies regardless of which room, device, or media ID appears in the URL.

Tier Routes Default per-IP policy
auth_sensitive /login, /register, /refresh, and every */requestToken route (matched by suffix) — unauthenticated, so the per-IP bucket is the only defense 20/60s
media /_matrix/media/* and /_matrix/client/v1/media/* 120/60s; thumbnail refinement 240/60s
sync /sync plus the MSC4186 and simplified MSC3575 sliding-sync long-polls 3000/60s
federation /_matrix/federation/* routes reaching the client-server dispatcher 3000/60s
admin /_merovingian/admin/* 30/60s
generic every other client-server route client_rate_limits.default_per_ip (600/60s)

Built-in per-endpoint refinements inside a tier: device and key APIs at 120/60s, search at 20/60s, and thumbnails at 240/60s. Search requests do real work (a bounded in-memory scan, see ClientApiLimits::max_search_events_scanned) rather than a cheap lookup. The built-in per-user cap is 5/60s on /login.

The per-user tier keys on the authenticated user, which before a login succeeds is nobody — so a /login guessing spree against one account spread across many source IPs previously accumulated against nothing at all, regardless of tier. There is now an additional, independent per-account failed-login throttle layered on top of these IP/user buckets, tracking failures against the claimed user ID rather than an IP or an authenticated identity; see docs/auth-identity.md "Failed-login throttle" for the mechanism, thresholds, and trade-offs.

Classification is method-agnostic: a GET against /login is the same enumeration surface as a POST, and the */requestToken family spans several path parents, so it is matched by suffix.

Operator overrides

Per-IP policy resolution is most-specific-first:

  1. client_rate_limits.per_ip.<target-prefix> (longest prefix match wins),
  2. client_rate_limits.tier.<name> for the route's tier,
  3. the built-in per-endpoint refinement (keys/devices 120/60s, search 20/60s),
  4. the tier default; the generic tier resolves to client_rate_limits.default_per_ip.

Per-user resolution: client_rate_limits.per_user.<target-prefix> first, then the built-in 5/60s login cap; routes with neither have no per-user cap (the per-IP cap still applies). Tier names are auth_sensitive, media, sync, federation, admin, generic; an unknown name is a parse-time finding, not a silently ignored key. All client_rate_limits.* changes require a server restart. window_seconds must be 1..3600 — both rate_limit_policy_is_valid() (engine) and config validation reject anything outside that range.

Defaults remain the operator-agreed secure values from the 0.5.0 design doc; tiering only makes them explicit and complete. One deliberate tightening: the auth-sensitive tier now covers /refresh, the */requestToken family, and non-POST hits on /login//register, which previously fell into the 90/60s generic fallback.

Different thumbnail media IDs normalize to the same per-IP action bucket. The 240/minute refinement accommodates ordinary room-rendering bursts while uploads and full media downloads retain the 20/minute tier default. Prefer a thumbnail-prefix override over raising the whole media tier if local usage still needs a higher cap; the user manual provides an example. Each denied request emits one rate_limit.exceeded warning, while preserving both its rate-limit and request-rejection audit records. Suppressing diagnostics does not change enforcement or HTTP 429 retry information.

Fail-closed on an unresolvable policy (issue #412): the per-IP policy resolves to a value on every route unless a configured entry, tier override, or the default fails rate_limit_policy_is_valid() (e.g. window_seconds > 3600). An unresolvable per-IP policy makes RateLimitEngine::check() deny the request — even when a valid per-user policy exists, because the per-IP bucket is the only defense on unauthenticated routes. A misconfigured policy must never silently disable rate limiting.

Bounded bucket tables (issue #427): m_ip_buckets/m_user_buckets are hash maps capped at 100,000 entries each, so a client rotating a spoofable X-Forwarded-For value (see below) cannot grow the table. Each bucket also has an engine-owned recency-list entry; an existing bucket moves to the tail, and a new key at capacity evicts the head and corresponding map entry. This keeps admission eviction expected O(1) instead of scanning all 100,000 entries.

normalized_target() normalizes only recognized route shapes and preserves their action names. It coalesces variable room, event, transaction, media, directory-alias, device, user-data, room-tag, presence, key-backup and SSO identity-provider components into route templates. All unmatched paths share one fallback bucket, while known static endpoints keep separate keys. Policy lookup, for operator policies and the built-in refinements alike, checks both the normalized template and the original query-free target. Operators can therefore configure route-wide template prefixes without losing existing prefixes written against the original path, and a built-in cap such as the 120/min /_matrix/client/v3/keys/ refinement still applies to a path that falls back. Every implemented route with a variable path component must have a template; the fallback is only for paths the server does not implement (ADR-0092).

Inbound federation

/send transactions are limited per verified origin server name (the X-Matrix-authenticated peer, not the IP) by a weighted trio: security.federation.per_origin_transaction_rate (600/60s), per_origin_pdu_rate (6000/60s), per_origin_edu_rate (12000/60s). Every other inbound federation endpoint (query, backfill, membership, key and state routes) is limited by security.federation.per_origin_request_rate (3000/60s), checked after signature verification and the server-ACL check, before dispatch. Non-/send traffic is counted only against per_origin_request_rate and /send only against the weighted trio, so a transaction and its contents are never double-counted.

Admin routes

/_merovingian/admin/* (health, metrics, audit, media moderation) is served on the public client listener. The client-server dispatcher routes the /_merovingian/admin/ prefix to the local router before the general user-token gate, so require_admin() owns the auth outcome: 401 M_MISSING_TOKEN/M_UNKNOWN_TOKEN for a missing or invalid token, 403 M_FORBIDDEN for a valid token belonging to a non-admin user. The routes inherit the same allow() rate-limit gate as every other client-server request — throttled exactly once per request, no double-count — under the admin-tier default (30/60s, operator-tunable via client_rate_limits.tier.admin or a per-prefix entry). Operator-only and low-volume, but still throttled against brute-force token guessing.

In-memory counter trade-off

Rate-limit counters live entirely in process memory (m_ip_buckets / m_user_buckets on the RateLimitEngine). They are not persisted: there is no per-request database write to update a counter, by design. The trade-off is that a restart (or a worker crash under a federated deployment) resets the counters, so a client that was being throttled can immediately retry. This is an accepted operator sign-off: the cost of a per-request durable write — and the latency and contention a shared counter table would add to the hottest path in the server — is not worth the marginal benefit, because rate limiting is a best-effort abuse throttle rather than a hard correctness invariant. If a durable cap is required for a specific route, an operator should front the homeserver with a proxy that enforces it.

Trusted-proxy client IP resolution

When the direct TCP peer's address is listed in server.trusted_proxies, effective_client_ip reads every X-Forwarded-For line, in order, as one list and walks it from the right past entries that are themselves trusted proxies; the first other entry is the client (ADR-0073). Each proxy appends the address it received the request from, so only the entries written by trusted proxies can be believed: the leftmost entries are whatever the client sent, and never choose the bucket. That entry is validated as a real IPv4 or IPv6 literal (federation::ip_address_is_valid()) before it is trusted — a trusted proxy is only trusted to forward its own view of the client address correctly, not to hand the server an arbitrary string. If the header is missing, or that entry is not a valid IP literal, the limiter falls back to the direct peer address rather than trusting it verbatim.

rate_limit_client_key then reduces the address with http::client_address_key, so IPv6 clients are grouped by server.http.ipv6_client_prefix_length (default /64; IPv4 is keyed as is). The client-server rate limiter and the federation key-resolution budget both key on it.

Without the literal check, an attacker able to reach a trusted proxy (or a proxy that fails to overwrite an inbound X-Forwarded-For header) could rotate through malformed pseudo-IP values to mint a fresh rate-limit bucket per request and defeat per-IP limiting on /login, /register, and every other endpoint entirely.

Federation signing-key resolution admission

Network-capable relayed-PDU sender lookups share origin bootstrap's source-keyed rate budget, per-runtime in-flight cap and bounded (server_name, key_id) negative cache (FED-10). Authenticating a relay does not authorize unlimited discovery for other domains named in its events. Parallel pre-resolution and authorization fallbacks use the same admission wrapper; overload rejects work rather than queueing it. A rejected admission is not cached as a failed lookup.

Core inbound resolution preserves local denial separately from actual lookup failure. A locally deferred sender or restricted-join authorising key returns 503 without charging or resetting the relay's trust. The original transaction is not recorded as accepted, and its EDUs are deferred, so it remains retryable. Genuine failures, including negative-cache hits, retain their penalties in mixed transactions. Some PDUs may already have reached ingestion before the 503; event-ID idempotency remains necessary on retry. Direct membership/invite paths likewise return 503 before their acceptance hook. See ADR-0133.

Main's PDU, membership and invite IPC re-verification uses the same key-only resolution/admission helper. Matching cached keys need no discovery. Misses share main's in-flight cap, negative cache and one <unknown> source bucket: these frames contain no main-validated client address, and a worker-provided address must not select the bucket. This deliberately limits cold-key misses across all main relay channels rather than claiming per-client fairness. Local denial returns the existing PDU main_overloaded status or an unaccepted 503 membership/invite response, before mutation. Signature, event-time validity and signed-event identity checks still run. See ADR-0134. These remain per-runtime controls, not aggregate limits for every process. Backfilled events, snapshot state and fetched auth-chain events now use that same admitted key-only helper and unknown-source bucket. Their existing backfill HTTP limits remain separate. A local key denial stops subsequent recovery calls and returns the existing PDU overload result, not ordinary missing state; transaction handling therefore preserves 503, deferred EDUs and unchanged-transaction retry. Already verified outliers from earlier recovery steps are not rolled back. See ADR-0135. Fresh post-backfill full-suite verification and independent source-only review passed. Additional sticky-denial/partial-progress tests and end-to-end worker socket-HTTP overload propagation coverage remain before whole-FED-10 closure; source review does not establish transport behavior.

A separate cached_pdu_key_lookup reads verification keys without DNS or HTTP. Production returns owned local signing keys, including retained keys, or fresh persistent remote keys; the federation core checks the requested server/key identity and event authorization retains signature and event-time validity checks. These reads spend no outbound budget and release no outbound reservation.

Known-peer rotation can use the same direct key read while preserving existing discovery/trust state. An unknown origin still needs admitted discovery even when its persistent key is fresh; freshness is not proof that a complete peer resolver will avoid the network. Production therefore does not set the legacy boolean full-resolution cache hint. Raw and already-verified request paths populate the signed request's source using rate_limit_client_key; missing sources retain the shared unknown bucket. In-flight guards release only their own reservations, and parallel task permits/callable lifetimes are RAII-managed during exceptions. See ADR-0132.

Sync long-poll thread pool

/sync long-polls are dispatched to a dedicated sync_pool (128 threads by default, server.http.sync_threads), separate from the main request pool (server.http.request_threads, default 32) that serves every other client-server and federation request. This split exists because a burst of long-polling clients on the main pool could previously exhaust it entirely, starving federation and other short-lived requests. See docs/architecture.md "Runtime model" for the full pool layout and src/sync/AGENTS.md for sync-specific conventions.

Both v3 and sliding-sync waits share admission budgets keyed by authenticated account and device, rather than bearer token or IP address: four waits per account and two per device by default (sync_max_per_user, sync_max_per_device). The configured global cap defaults to 128 and is clamped to the sync pool's worker count; per-account and per-device caps cannot exceed that global cap. All three caps are server.http settings. Queued tasks count against these limits. Excess waits get 429 M_LIMIT_EXCEEDED with retry_after_ms, and a refused sync-pool submission also returns 429 immediately instead of waiting on a main worker. Slots are released on response, disconnect, exception and failed handoff. Timeouts are capped at 120 seconds. See ADR-0091.

A waiting sync is run again after each wake-up and once more when its timeout expires (on the sync pool, in the no-pool fallback and in dispatch_local_http_request). Those re-runs pass ClientServerDispatchOptions::rate_limit_admitted (rerun_waiting_sync), so the rate limiter counts each /sync once, when it first arrives; a client allowed one more request is no longer refused with 429 at the moment its waiting sync is answered.

Request lock and blocking network calls

Both request entry points — handle_client_server_request and handle_local_http_request — take HomeserverRuntime::mutex at entry and release it around blocking work. Inbound federation holds it for setup and narrower callbacks, not across the federation core or prepared database commits. Holding it still blocks other clients and federation operations that need those scopes.

A synchronous outbound call made while holding it therefore stalls the whole process for the length of the remote's timeout. Two paths used to do exactly that: POST /_matrix/client/v3/keys/query (one federation /user/keys/query per remote server, each budgeted remote_timeout_seconds) and remote media download/thumbnail fetches. One unreachable peer was enough to freeze local reads for 20–44 seconds at a time.

The entry points now publish their guard through homeserver::RequestLockScope, and each blocking network call runs inside a homeserver::RuntimeLockRelease scope that releases the mutex for the round trip and re-acquires it on exit — including when the call throws. Both types live in include/merovingian/homeserver/request_lock.hpp.

Rules for anything added to these paths:

  • Only the network call goes inside the unlock scope. Reads and mutations of runtime state stay outside it, before or after.
  • The signing secret is copied into an owned core::SecretBuffer before the scope opens and moved into OutboundCall::secret_key; the call owns the key and can sign after the mutex is released (ADR-0086).
  • The scope is a no-op when no guard is published and none is passed in (the federation worker, a test calling a service function directly).
  • It releases every recursion level the calling thread holds, not only the guard it was handed, and restores exactly that many on exit. See "One release primitive" below for why that matters.

Outbound header validation (ADR-0180)

The outbound client refuses a request before any DNS, connection or serialisation when a header name is not an RFC 9110 token or a header value holds a control character other than HTAB (OutboundError::invalid_header). Header values can carry client input, such as an identity server access token, and libcurl writes header lines verbatim.

Client-triggered outbound proxying (ADR-0079)

Releasing the runtime mutex during a network call frees the mutex, not the thread: the request still occupies one of the main-pool threads until the peer answers or the call times out. Several client requests make this server call another one before they can answer, and the spec lets clients make them without authentication, so a peer that accepts a connection and never answers could pin the whole pool (audit findings HTTP-2 and OUT-7). Every one of them therefore runs under a small in-flight budget and a short total deadline.

The paths under the budget:

Request Outbound call
GET /_matrix/client/v3/publicRooms?server=<remote> federation GET /publicRooms
POST /_matrix/client/v3/publicRooms?server=<remote> federation GET or POST /publicRooms
GET /_matrix/client/v3/directory/room/{alias} for a remote alias federation GET /query/directory
GET /_matrix/client/v3/thirdparty/*, when an appservice will be called appservice GET /_matrix/app/v1/thirdparty/* (ADR-0181)
GET /_matrix/media/v3/download and /thumbnail for remote media, without a media fetch pool discovery, federation media, legacy fallback, redirect follow
GET /_matrix/client/v1/media/download and /thumbnail for remote media, without a media fetch pool as above

The server hands remote media to its media fetch pool instead (next section); the media rows apply only to callers with no such pool: tests, embedded callers and dispatch_local_http_request.

Rules:

  • Budget. http::InFlightBudget (RAII Slot, thread-safe, never waits), held by HomeserverRuntime::client_outbound_budget. Global cap half of the main request pool (server.http.request_threads / 2, so 8 by default; the other half of the pool always serves requests that never leave this server); per-client cap 1, keyed by rate_limit_client_key (the same key the rate limiter uses, so trusted_proxies and the IPv6 prefix grouping apply). Authenticated callers count too. Local publicRooms, local aliases and local media never take a slot.
  • Refusal. Over either cap the answer is 429 M_LIMIT_EXCEEDED with retry_after_ms 1000 and a Retry-After: 1 header, immediately, before any discovery or outbound call. Nothing queues.
  • Deadline. 10 s for directory lookups, 30 s for remote media, in total: server discovery and the request(s) that follow draw from one OutboundDeadline. Never longer than the operator's security.federation.remote_timeout when that is set. The federation-worker round trip for these calls may exceed the deadline by at most 2 s (ClientOutboundProxyPolicy::worker_margin_seconds), where other worker calls allow 10 s. A deadline that runs out during discovery or between the federation request and the legacy fallback ends the call as 502.
  • Lock. The slot is taken while the runtime lock is held and before it is released for the call (RuntimeLockRelease); it is dropped on every exit path by its destructor.
  • Constants, not configuration. The caps and deadlines live in ClientOutboundProxyPolicy (include/merovingian/homeserver/client_outbound_proxy.hpp), derived from the main request pool size (client_outbound_proxy_policy_for_pool, applied by start_runtime from server.http.request_threads, ADR-0077). A separate configuration key would let an operator raise the caps above the pool size, which is the defect. Tests lower the deadline through runtime.client_outbound_proxy_policy.
  • Not covered. Outbound calls to operator-configured destinations that an unauthenticated client can still trigger (appservice query_user and query_room_alias, the identity-server requestToken routes, the trust-safety policy server) are not under this budget: the destination is chosen by the operator, not the caller. They still hold a thread for the round trip.

The alternative of a separate thread pool or an asynchronous proxy path was considered and rejected for now; see ADR-0079. For remote media it was later adopted (next section).

Remote media fetch pool (ADR-0121)

A browser shows an image and its thumbnail, and every remote image on screen, at once. Under the one-per-client budget above every request after the first was refused with 429, and Element does not retry a media 429, so a federated user's attachment stayed broken. Remote media therefore leaves the main pool.

  • Defer. A main-pool round runs the client-server handler in RemoteMediaFetchMode::defer (RemoteMediaFetchScope, thread-local like RequestLockScope). fetch_remote_media_live does everything quick — remote_fetch_enabled/allow_remote, the trust-and-safety policy, the remote media cache — and on a cache miss records a deferral and returns. handle_client_server_request answers DispatchResult::Status::needs_media_fetch with no response.
  • Admit, then hand off. The round takes a slot in ClientServerRuntime::media_fetch_budget keyed by rate_limit_client_key: at most server.http.media_fetch_max_in_flight fetches running or queued (64) and server.http.media_fetch_max_per_client for one client (8). It then submits a MediaFetchHandoff, which owns the connection and the slot, to the media fetch pool (server.http.media_fetch_threads, 16 workers, queue as deep as the global cap). Over a cap, or if the pool refuses the task, the answer is 429 M_LIMIT_EXCEEDED with retry_after_ms 1000, at once, and the connection is closed.
  • Run again. The pool runs the request again in RemoteMediaFetchMode::admitted with rate_limit_admitted, so the rate limiter does not count it twice and the fetch takes no ADR-0079 slot. It writes the response and parks a kept-alive connection with the dispatcher; the next request runs on a main-pool worker.
  • One fetch per file. RemoteMediaFetchCoalescer lets one request fetch a given remote file (canonical origin and media ID). Another request for it waits, with the runtime mutex released and within its own 30 s deadline, then serves the copy the first stored. Without this the second admission displaces the first copy from the cache while it is being read.
  • Shutdown and abandoned requests. A handoff that starts after its pool stopped answers 503 without fetching. One whose client has gone is dropped.
  • Directory lookups (publicRooms?server=, remote aliases) are unchanged: they still run on the main pool under the ADR-0079 budget.

Log events: media_service remote_fetch.deferred (a main-pool request handed to the pool), remote_fetch.coalesced (waiting on another request's fetch); http_server media_fetch.over_budget, media_fetch.client_gone.

resolve_policy_server_hook (0.12.1)

resolve_policy_server_hook (src/homeserver/runtime.cpp) performs a synchronous outbound call to trust_safety.policy_server_url when trust_safety.enabled is set. handle_federation_http_request was already fixed (#415) to call it only after releasing the lock, but three other call sites called it directly while still holding HomeserverRuntime::mutex: register_local_user (client registration), create_room (room creation and room upgrade), and media_policy_decision (remote media download and thumbnail fetch — right before the already-unlocked remote fetch call that follows it). A slow or unreachable policy server therefore still froze every other client and federation request for up to policy_server_timeout, despite #415.

The fix wraps the remainder of resolve_policy_server_hook — the injectable trust_safety_policy_server test hook and the real OutboundClient::perform call alike — in one RuntimeLockRelease scope, so every call site is fixed at the source rather than needing four separate call-site changes. Everything the function still reads (trust_safety_config, runtime.config, the request built from them) is read before that scope, while the lock is still held; nothing after it touches runtime state. See tests/integration/test_request_lock_contention_flow.cpp for the regression coverage (registration, room creation, and media download, each gated on a blocking policy-server hook while an unrelated request is asserted to complete promptly).

One release primitive, and why it drains every level (0.12.1 - 0.12.6)

HomeserverRuntime::mutex is a recursive mutex specifically so that a service function such as create_room can take its own lock and remain independently callable outside a request handler (local_smoke_flow.cpp does exactly this). But create_room is also called from client_server.cpp, which already holds its own outer guard for the whole request. A recursive mutex permits that nested acquisition silently — the same thread increments a recursion count — so nothing failed loudly.

The consequence: unlock() drops one level, not the lock. A handler that released the guard it had in hand, while an outer frame still held one, kept the mutex locked for the entire duration of the network call it thought it had released it for. Every other client request and every inbound federation transaction blocked behind it for the length of the remote timeout.

That defect reached production three times, in three different shapes, found three different ways:

Where Version How it was found
create_room via resolve_policy_server_hook 0.12.1 A regression test deadlocked rather than failing an assertion
leave_room via the client leave route 0.12.3 Review on #485; it had been documented as closed when it was not
invite_user_by_threepid via the client invite route 0.12.6 A regression test written for issue #487 before any code changed

The first two were fixed at the call site, by releasing the caller's guard before delegating. That works, but it depends on every future caller of every self-locking function remembering to do it, and on choosing between two primitives that differed in a way nothing checked: NetworkIoUnlock acted on the thread's published guard (RequestLockScope), ScopedGuardRelease on a guard in hand. At a nested call site those are different objects, and picking the wrong one produced code that compiled, passed its tests, and held a global mutex across a network round trip.

0.12.6 removes the choice. There is one primitive, homeserver::RuntimeLockRelease, whose two constructors say only which unique_lock should observe the release:

auto const released = RuntimeLockRelease{};        // the thread's published guard
auto const released = RuntimeLockRelease{guard};   // a guard in hand

Both release every recursion level the calling thread holds and restore exactly that many on scope exit, including when the guarded call throws. They can do so because runtime.mutex is now a homeserver::RuntimeMutex (runtime_mutex.hpp): a std::recursive_mutex that also records its owning thread, so a release scope can ask held_by_current_thread() and keep unlocking until the answer is no. std::recursive_mutex cannot answer that question, which is why the mechanism could not be made safe without the wrapper.

A release scope drops those levels through the mutex, never through a guard: the levels belong to unique_lock objects in frames it cannot reach. Every unique_lock on the mutex therefore keeps reporting owns_lock() == true for the lifetime of the scope, the one passed in included. That is sound only because nothing outside request_lock.cpp reads the flag and the depth is restored exactly before any guard can act on it. An earlier revision of this change released the named guard first and drained the rest through the mutex; review on #490 caught that an inner scope reaching for the thread's published guard would then act on a stale flag, unlock a mutex the thread no longer held, underflow the recursion depth, and strand runtime.mutex locked for the life of the process. Nested scopes now read held_by_current_thread() and correctly find nothing left to release; the regression is tests/unit/test_request_lock.cpp, "release scopes nest without corrupting the recursion depth", which fails against the old shape with Operation not permitted.

Consequences worth knowing:

  • Releasing the caller's guard before calling a self-locking function is no longer necessary. The existing call sites that do (create_room and join_room in local_http_router.cpp, create_room in client_server.cpp) are kept because they are correct and explicit, not because they are load bearing.
  • A blocking call should open its release scope where the call is, not where the caller is. The callee knows it is about to do network I/O; the caller may not.
  • RuntimeLockRelease::levels_released() reports how many levels the scope dropped and will restore. More than one means some outer frame was holding the mutex across this call, which used to be the bug; zero is the normal answer for a nested scope.
  • Do not read owns_lock() on a guard to decide anything. It is not updated while a release scope is open, and no code outside the primitive reads it.

join_room, and why the released region became a function

join_room's released region spans roughly 350 lines, and nine of the forty-four values declared inside it are consumed after the re-lock (verified_critical_state, verified_auth_chain, signed_event, event_id_result among them). Wrapping it in a release scope mechanically would have meant hoisting those nine declarations above the scope: stripping const from each, requiring every type to be default-constructible, and splitting nine initialisations into declare-then-assign inside a 1000-line function.

0.12.6 extracts the region into perform_federated_join, which returns a FederatedJoinOutcome carrying exactly those nine values plus an optional failure the caller returns verbatim. The lock boundary is now the function boundary: nothing needs hoisting, every declaration keeps its const, and the release itself is a two-line RuntimeLockRelease scope around the call. The same idea at smaller scale — an immediately-invoked lambda returning a small result struct — is what invite_user_by_threepid and the three local_http_router.cpp sites use.

No hand-written unlock()/lock() pair around runtime.mutex remains in include/ or src/. scripts/reject-unsafe.sh rejects new ones, matching ->unlock() as well as .unlock() since 0.12.6, and exempting only the two files that implement the primitives themselves.

Load/soak evidence

tests/integration/test_runtime_lock_soak_flow.cpp (gated behind the build_load_tests Meson option, parallel to build_live_tests) drives real concurrent traffic — over real TCP sockets, HTTP/1.1 keep-alive throughout — against a running server: several users each long-polling their own room's /sync, each doing ordinary authenticated reads, each sending messages into their own room, and several simulated remote servers sending signed, X-Matrix-authenticated inbound federation transactions, all at once. It reports throughput and p50/p95/p99 latency per category to stderr. Its own default duration is short enough to run safely as a CI correctness check (nothing deadlocks or starves); a real measurement run sets MEROVINGIAN_LOCK_SOAK_SECONDS to something longer:

meson configure build-wsl -Dbuild_load_tests=true
MEROVINGIAN_LOCK_SOAK_SECONDS=60 ./build-wsl/tests/merovingian-load-tests "[load-soak]"

See docs/todos/production-milestone.md, "Global runtime lock", for the measured before/after numbers this harness produced and the resulting narrowing decision.

This harness is a manual tool, not a CI job — deliberately, matching the existing build_live_tests precedent (also gated behind an opt-in Meson option with zero .github/workflows/ wiring; see docs/testing-standards.md). A real measurement run needs tens of seconds to minutes of wall-clock time to be meaningful, which does not fit a per-PR CI budget, and headline throughput/latency numbers on shared CI runners are noisy and non-reproducible in a way that would make a regression gate here more misleading than useful. Its default (unset MEROVINGIAN_LOCK_SOAK_SECONDS) 2-second run does stay CI-safe as a correctness check — nothing deadlocks or starves — and CAN be added as a fast job later if that is wanted, but that is a distinct decision from running it as a soak/perf gate. Use it manually, locally or on a dedicated benchmarking host, whenever a future lock-narrowing change needs before/after evidence.

Fuzzing

fuzz-http-request exercises the request-head parser against arbitrary input. It is registered with the existing fuzz target group.

Configurable transport budgets (0.12.18)

server.http now configures sync workers/admission and HTTP start-line/header/body ceilings. Defaults are 128 sync workers and global long-polls, 8 per user and 4 per device; admission is clamped to the actual worker count. Head defaults remain 8KiB start-line, 32KiB headers and 100 headers. Generic HTTP bodies remain 1MiB. Federation PUT /_matrix/federation/v1/send/{txnId} uses security.federation.max_transaction_size (20MiB), rather than being shadowed by the generic body cap. Media uploads retain their early token check and media quota.

Worker IPC derives a common bidirectional frame ceiling from the larger of join/backfill response caps and the configured request limits. Response bytes expand by padded base64 (4/3); arbitrary request/head bytes nested in JSON can expand sixfold. Each budget reserves 2MiB envelope headroom and a 24MiB minimum; arithmetic saturates below the uint32 ciphertext length boundary, leaving space for the 17-byte tag. This is a per-frame allocation ceiling, not preallocated memory. Main and worker must restart together after a policy change.

See operator budgets for validated ranges and combined constraints. Search/login/admin controls and fixed slow-client/sandbox boundaries retain their tighter budgets. These defaults have behavioral regression coverage, not a 100-user throughput benchmark.