I’m adding OpenAI and other model providers to an application, and I’m not sure how much of the integration I can test without making real API calls. Live calls cost money, change from run to run, and can fail for reasons unrelated to my code. But mocking everything seems like it could miss provider-specific bugs. How do people strike a sensible balance?
A useful approach is to split testing into layers. Most application behavior can be tested without a model at all; provider-specific request and response handling needs its own checks; and a small number of live calls can catch issues that fixtures cannot.
Test your application against a stable interface
Your application should not need to know every detail of an OpenAI, Claude, Grok, Perplexity, or DeepSeek response. Put provider-specific work behind an adapter that returns a small, consistent result your app can use, such as text, finish reason, and usage information when available.
Then use a fake adapter in ordinary unit tests. It can return a known response, raise a timeout, or simulate an empty result. These tests are quick and repeatable, and they let you check what your application does with each outcome without spending tokens.
class FakeLLM:
def __init__(self, result):
# Keep a predictable response and record what the app sends.
self.result = result
self.prompts = []
def generate(self, prompt):
# Capture the prompt so the test can check the application boundary.
self.prompts.append(prompt)
return self.result
def make_summary(llm, text):
# This application function depends on the adapter interface, not a vendor SDK.
result = llm.generate(f"Summarize this text:\n{text}")
return result["text"]
def test_make_summary_uses_adapter_and_returns_text():
# Supply a fixed result so this test never contacts a model provider.
llm = FakeLLM({"text": "A short summary.", "finish_reason": "stop"})
summary = make_summary(llm, "A long passage that needs summarizing.")
# Check both the user-visible result and the prompt passed to the adapter.
assert summary == "A short summary."
assert llm.prompts == [
"Summarize this text:\nA long passage that needs summarizing."
]
This kind of test is not meant to prove that a provider’s API works. It proves that your own application handles the adapter contract as intended. Add cases for errors and edge conditions that matter to your product—for example, a timeout, a refusal result, or a response with no text—rather than trying to predict every possible model answer.
Check each provider adapter at its boundary
The adapter is where provider-specific details live: request fields, authentication headers, response parsing, and error conversion. Test those details separately. A mocked HTTP transport or SDK client lets you inspect the outgoing request and feed the adapter a recorded response fixture.
Keep a few representative fixtures from each provider’s documented response shape, including an ordinary completion and any special cases your code handles. A fixture can reveal a parser regression when a library upgrade changes a field or when your mapping assumes a response always contains text. Store fixtures with their provider and API version noted; otherwise, a test failure months later can be hard to interpret.
These checks should verify the adapter’s contract, not freeze every byte of a provider response. Assert that the right model and required parameters were sent, and that expected response fields are mapped correctly. Avoid tests that fail because an irrelevant field was reordered or a usage counter changed.
Use recorded outputs carefully for prompt changes
Recorded response fixtures are helpful for testing parsing and downstream behavior, but they are not a dependable measure of answer quality. If you change an LLM prompt, a live model may phrase a correct answer differently from the saved one. Exact-string comparisons can turn harmless variation into a false alarm.
For prompt regression checks, keep a small set of representative inputs and evaluate qualities that matter to the task: required facts present, forbidden claims absent, output parseable, or a human reviewer’s rating. A handful of cases can catch obvious regressions, but it should not be treated as proof that a prompt is better across every user request.
Keep a small live-test lane
Mocks cannot tell you whether credentials still work, the selected model is available to your account, or the real endpoint behaves as your adapter expects. Keep a few live smoke tests for those checks, but run them deliberately—perhaps on a schedule, before a release, or manually after changing provider configuration.
- Use a dedicated test account and a low-cost model where practical.
- Send short, non-sensitive inputs; never put customer data in a smoke test.
- Set a strict request timeout and a spending limit.
- Check basic properties, such as receiving a non-empty result, rather than an exact sentence.
- Keep live tests separate from the fast test suite so a provider outage does not make every code change look broken.
If the team is unsure where to start, test most application behavior with a fake adapter, test each provider adapter with mocked transports and fixtures, and reserve live calls for a small smoke-test set. That gives you repeatable feedback without pretending that a mock can validate a real provider connection.
