Measured: OpenAIChatGenerator honours OPENAI_BASE_URL but ignores OPENAI_API_BASE — and how I replay a pipeline offline

[xizhuomengcontin]

Two things here: a small measured fact about Haystack that is useful on its own, and what I do with it.

OpenAIChatGenerator honours OPENAI_BASE_URL, not OPENAI_API_BASE

I wanted to point a pipeline at a local endpoint without editing the pipeline, so I checked which variable actually redirects it. Local HTTP server on 127.0.0.1:9978 returning a canned chat-completion and logging the path it is asked for, haystack-ai 3.1.1:

OPENAI_BASE_URL=http://127.0.0.1:9978/v1
  -> request arrived at /v1/chat/completions, replies[0].text == "ok"

OPENAI_API_BASE=http://127.0.0.1:9978/v1
  -> nothing arrived; the call went to the real API and returned 401

So OPENAI_BASE_URL works and OPENAI_API_BASE is silently ignored. Both spellings are common across the ecosystem — I have probed frameworks that accept both, and one that accepts neither because it pins the origin in its own config — and pointing the wrong one at a proxy or gateway gives you a run that quietly went to production. Worth knowing whatever you are pointing Haystack at.

What I use it for: running the same pipeline again for free

Once the calls can be redirected, they can be recorded and replayed. I run the pipeline once for real under a recorder, and after that the same run happens offline:

npm i -g orcareplay
orca record generic-openai -- python my_pipeline.py   # real run, recorded
orca replay last                                      # same run, no model called, nothing spent

Nothing in the pipeline changes — the recorder just moves OPENAI_BASE_URL for the child process. orca replay last --from N --model <other> replays the run up to step N from the recording and then continues on a different model, so when you are comparing models the model is the only variable.

Disclosure: OrcaReplay is mine — Apache-2.0, Node 20+, no account, replay needs no key. The measurement above is about Haystack and holds regardless of what you use it with.

Three limits

• A matching replay is not a determinism result. It proves the recorded exchange reproduces; it does not prove the generator or the model is stable, and it does not prove a fresh run would answer the same way.
• Replay blocks model-provider egress only. It is not a sandbox. Everything else your pipeline does still happens for real — a LinkContentFetcher still fetches, a custom component still calls whatever it calls.
• Embedding calls are not captured by the default adapter. For Haystack that is the important one: a RAG pipeline replays at the generator layer but the retriever is not served from the recording, so a retrieval-dependent behaviour will not be reproduced.

If the base-URL behaviour above is already documented somewhere I missed, I would genuinely like the pointer.