I’m comparing OpenAI, Claude, Grok, DeepSeek, and Perplexity for a feature that summarizes long documents. The headline prices look straightforward, but I’m not sure they tell me what the feature will actually cost once prompts, long answers, and provider-specific token counting are involved.
What’s a fair way to compare the cost of doing the same job across these APIs? I’d like to avoid picking a provider based on a single short test and then finding the production bill is much higher.
Compare a completed task, not just a price per token
A price per million tokens is useful, but it doesn’t tell the whole story. One model might produce a shorter answer, another might need more context to follow your instructions, and a third might use additional internal reasoning tokens. If the output quality differs, you may also end up retrying or sending follow-up requests. Those differences all affect the cost of delivering one successful result.
For a document-summary feature, define what counts as a completed task: for example, a summary that covers the key points, stays under a chosen length, and passes whatever checks your product needs. Then run a representative set of documents through each provider and record the cost per successful task. That makes the comparison more useful than asking which API has the cheapest listed input rate.
What can make estimates misleading?
- Provider-specific tokenizers: The same text can be counted differently by different models. A prompt that uses a certain number of tokens with one provider may not use the same number elsewhere.
- Input and output tokens: Long instructions and retrieved documents increase input usage, while verbose answers increase output usage. Measure both on real requests rather than assuming output will be a small part of the bill.
- Cached input pricing: Some APIs offer different pricing for eligible repeated input. Whether your requests qualify depends on the provider’s rules and how your prompts are structured, so don’t treat the discounted rate as automatic.
- Reasoning tokens: Certain models may use tokens for internal reasoning that are billed even when those tokens do not appear in the final answer. Check the usage details returned by the API.
- Tool-call payloads: If a request includes search results or other tool output, that material may add to the input cost. A model’s response can also lead to another API call, which belongs in the total.
Pricing and usage reporting can change, so check each provider’s current documentation before turning a test result into a budget. Keep the date and model identifier alongside your measurements; otherwise, an old comparison can quietly become a misleading one.
A practical comparison that stays fair
Use the same real-world sample set for every model. Include short and long documents, messy input, and examples where the summary must follow a specific format. Keep the task requirements consistent, but don’t assume every provider interprets identical wording equally well. The aim is to compare what your application would actually send and receive, not to prove that one prompt is universally optimal.
For each run, log the provider and model, input and output usage, any reported cached or reasoning usage, the number of API calls, and whether the result met your quality criteria. Apply the provider’s current rates to those usage figures. If the API doesn’t expose a particular usage detail, mark that gap rather than pretending the estimate is exact.
It helps to separate the per-request cost estimate from the successful task cost. The first describes one API call. The second includes the requests your application needed to get an acceptable result, such as a retry or a follow-up call. For product planning, successful task cost is usually the number worth comparing.
Finally, test at the volume and response length you expect in production. A provider that looks inexpensive on short inputs may behave differently when users submit lengthy documents or request detailed summaries. A small, regularly refreshed benchmark can reveal those shifts without requiring a complicated billing system.
How are other teams measuring model costs in practice—especially when some usage details are reported differently across APIs?
