{"version":"2.1.295","anchor":"gateway-notes-upstream-ttfb-ms-now-applies-to-non-anthropic","canonical_anchor":"gateway-upstream-ttfb-timeout-now-only-applies-when-operator","heading":"Gateway first-byte timeout now applies to every upstream type when set; upstreams accept a models list","tier":"use","area":"LLM Gateway","scope":"org","heads_up":false,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.295\/e\/gateway-notes-upstream-ttfb-ms-now-applies-to-non-anthropic","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.295","markdown":"### Gateway first-byte timeout now applies to every upstream type when set; upstreams accept a models list\n\nThe gateway's `timeouts.upstream_ttfb_ms` now covers non-Anthropic upstreams when you set it, upstreams take a models list, and Bedrock can count tokens\n\n**What**\n\nThe apps gateway sits between Claude Code and the model providers it forwards requests to, which are called upstreams. Its first-byte timeout setting, `timeouts.upstream_ttfb_ms`, controls how long the gateway waits for the first piece of a reply. It has been reworked, and upstreams gained new options.\n\n- `timeouts.upstream_ttfb_ms` is now optional in the gateway config. The schema no longer fills in a default. A default of 120000 milliseconds is still applied, but the gateway now records whether the operator wrote the value themselves.\n\n- The timeout, capped, is now passed on the upstream call for streaming requests and `count_tokens` requests. A request that times out returns an error message that names `timeouts.upstream_ttfb_ms`.\n\n- When an operator has set the value, the gateway logs an info message at startup. The message says the setting now also applies to streaming and `count_tokens` requests on non-anthropic upstreams. Before, the timeout applied only to `provider: anthropic` upstreams.\n\n- The `anthropic`, `bedrock`, `anthropicAws`, `anthropicGoogleCloud`, `vertex` and `foundry` upstreams now accept a `models` list.\n\n- `count_tokens` (a request that asks how many tokens a prompt uses) on Bedrock upstreams is now handled. Before, it always returned a 501 \"not supported\" error.\n\n**Why**\n\nIf you run the gateway with non-Anthropic upstreams and have set `timeouts.upstream_ttfb_ms`, the timeout now also applies to those upstreams. Slow first replies there can therefore fail where they did not before. Bedrock users can now get token counts through the gateway.\n\n- Area: LLM Gateway\n- Names: `timeouts.upstream_ttfb_ms`\n- Tier: Use it now\n- Useful: 4\/5\n- Signal: 2\/5\n- Scope: org\n- Heads-up: no"}