OpenAI's GPT-Live turns the chat API into a persistent socket

What happened

OpenAI shipped GPT-Live, a new API surface that swaps the familiar `POST /v1/chat/completions` request/response cycle for a persistent WebSocket-style session. Instead of paying to reload context on every turn, you open a connection, stream input events (text, audio, tool results, image frames), and the model streams events back for as long as the session is alive. The model is kept warm on the server side — its KV cache, tool state, and conversation history persist across turns until you close the socket or hit an idle timeout.

The launch spec covers three transports: raw WebSockets, WebRTC for audio, and a gRPC bidi stream for server-to-server. Pricing is a hybrid: a flat per-connected-minute rate on top of standard input/output token pricing, with a discount if the session is idle. Tool calls now stream partially — you get the tool name and arguments token by token, so you can start executing before the model finishes emitting the call. Interruptions are a first-class event: when the user starts talking, the client emits a `cancel` frame and the model's KV cache is rewound to the interruption point rather than discarded.

This is the first mainstream OpenAI API where the connection itself is the billed unit, not the request. That's a small line in the docs and a big change in how you architect anything built on it.

Why it matters

Every serious agent framework has been reinventing the same wheel: how do you keep a model "hot" so that turn latency doesn't dominate the loop? The standard tricks — prompt caching, prefix reuse, keeping a running scratchpad — all exist because the underlying API is stateless and every turn re-pays the context cost. Anthropic's prompt caching, Google's context caching, and DeepMind's session tokens are all partial fixes for the same architectural mismatch.

GPT-Live is the honest version of that fix: the session is the state. The KV cache lives on OpenAI's box for the life of the connection. First-token latency in the OpenAI docs shows p50 at 190ms and p95 at 340ms for text turns after the first — roughly 4-6x faster than a comparable Chat Completions call with a 32k context. For voice, WebRTC ingestion means the audio path never leaves the streaming pipeline, and reported end-of-turn to first-audio-out latency is 280ms at p50. Those numbers, if they hold in the wild, put GPT-Live on the same latency budget as a phone call.

The catch is that stateful long-lived connections break most of the assumptions your infra was built on. HTTP load balancers don't like them. Kubernetes rolling deploys terminate them. Retry logic that works for idempotent POSTs becomes a footgun when the retry means opening a new session and re-uploading the entire conversation. If your assistant runs behind a serverless function, congratulations — you now need a stateful proxy layer, because Lambda and Cloud Run cost more per minute of idle socket than the API itself.

The pricing model deserves attention. At the launch rate, an idle session runs cheap, but any real product will keep a socket open per active user. A B2C app with 10k concurrent users pays for 10k concurrent minutes whether or not those users are actively prompting. That's a step-function change from the pay-per-token model that made GPT-4 economics tractable for consumer apps. Expect a wave of "session pooling" middleware — sharing one warm session across multiple users when the conversation history is short enough to hot-swap.

The community reaction on Hacker News is split along predictable lines. Voice-app builders (Vapi, Retell, Deepgram's LLM layer) are the obvious winners — they've been duct-taping the Realtime API's beta to make this work for a year. Agent framework maintainers (LangGraph, CrewAI, Mastra) are quieter, because a persistent session partly obviates their orchestration layer: if the model itself remembers the loop, you don't need a Python graph to remember it for it. Infra folks are asking the obvious question: what happens when the process on OpenAI's side crashes mid-session, and does the client get a resumable session token or does it start over?

What this means for your stack

If you're building anything voice-first, migrate. The old Realtime beta is being sunset in 90 days per the deprecation notice, and GPT-Live is a strict superset. Budget a week for the connection-lifecycle work — reconnect logic, exponential backoff on the socket, and a heartbeat that doesn't rack up connected-minute charges.

If you're running a text-based agent, the calculus is subtler. You should adopt GPT-Live for any loop where turn latency dominates and drop it for anything that runs in short bursts. A code review bot that fires once per PR is worse on GPT-Live than on Chat Completions — you pay for connected minutes you don't use. A pair-programming copilot that stays open all day is dramatically better, because you stop paying to reload the file context on every keystroke.

The operational shift is real. Your existing observability (per-request tracing, retry budgets, circuit breakers) was designed for stateless RPC. You need session-level dashboards: how many sockets are open, what's the p95 idle time, how often do we lose a session mid-turn and have to reconnect. If you're on a proxy like LiteLLM or Portkey, wait a beat — they'll ship GPT-Live adapters in the next couple of weeks and it's not worth writing your own reconnect logic if you don't have to.

One watch-out: the `cancel` frame semantics are subtly different from every other streaming API in the space. Cancelling mid-tool-call leaves the model in an ambiguous state (did the tool run? did it partially run?) and the docs push that back to the client. If your tools are non-idempotent — anything that hits a payments API, sends an email, or writes to a database — you need idempotency keys at the tool level, not the session level.

Looking ahead

The more interesting question isn't whether GPT-Live wins — it's whether Anthropic and Google ship the equivalent by year-end. Both have the infrastructure; neither has committed to a persistent-session pricing model publicly. If they hold the line on stateless APIs, OpenAI has a real latency moat for voice and agentic workloads. If they match, we're looking at the end of the stateless LLM API as the default shape, and the beginning of a much messier era where every LLM client is really a distributed system.