<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[LONG AI Engineering Notes]]></title><description><![CDATA[LONG AI Engineering Notes]]></description><link>https://longai-eng.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sun, 22 Nov 2026 17:10:47 GMT</lastBuildDate><atom:link href="https://longai-eng.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[What 51 upstream API sellers taught me about routing, billing and protocol compatibility]]></title><description><![CDATA[Every "one API for every model" product I have tried makes the same bet: pick one upstream vendor per model, retry when it fails, and hope the retry does…]]></description><link>https://longai-eng.hashnode.dev/what-51-upstream-api-sellers-taught-me-about-routing-billing-and-protocol-compatibility</link><guid isPermaLink="true">https://longai-eng.hashnode.dev/what-51-upstream-api-sellers-taught-me-about-routing-billing-and-protocol-compatibility</guid><category><![CDATA[AI]]></category><category><![CDATA[api]]></category><category><![CDATA[webdev]]></category><dc:creator><![CDATA[Long AI]]></dc:creator><pubDate>Fri, 20 Nov 2026 16:24:23 GMT</pubDate><content:encoded><![CDATA[<p>Every "one API for every model" product I have tried makes the same bet: pick one upstream vendor per model, retry when it fails, and hope the retry does not cost a customer their p99.</p>
<p>I went the other way. Instead of a vendor list, I built a marketplace of spare capacity and let the router choose a seller per request. Three years of conventional gateway design go out of the window at that point. These are the parts that actually hurt.</p>
<h2>Routing is not the same thing as retrying</h2>
<p>Backoff on a 429 looks responsible and is usually wrong. When thirty clients back off against the same saturated seller, the queue simply refills. The fix that worked:</p>
<ul>
<li>Track health per seller <strong>per model</strong>. A seller that is reliable on a cheap small model can be the worst option on a reasoning model.</li>
<li>Treat 429, 5xx and mid-stream disconnects as routing signals. Re-route before the first token; after the first token, finish the stream and degrade the seller instead of cutting the customer off.</li>
<li>Cap concurrency per seller so you saturate without stampeding.</li>
<li>Bill on delivery, never on attempt. A request that produced no tokens gets no line item - which also removes any incentive to silently retry and call it success.</li>
</ul>
<p>Across about fifty-one upstream providers this moved rate-limit failures from a visible percentage of calls to something you have to graph to notice.</p>
<h2>Protocol compatibility is a test suite, not a promise</h2>
<p>"OpenAI compatible" hides a lot. <code>stream_options</code>, tool-call deltas, <code>logprobs</code>, <code>response_format</code>, providers that silently drop <code>max_completion_tokens</code>, providers that return usage numbers your tokenizer disagrees with. Every adapter needs conformance tests against the same schema, written before the adapter, or you will debug differences in production on behalf of users who cannot see your side of the request.</p>
<h2>Price transparency is the actual product</h2>
<p>A discount is a marketing number. What developers ask for is a price they can see before sending, and a line item afterwards that matches it. That forces the gateway to price a request - model, tokens, region, cache - before routing, then reconcile against delivered usage. The awkward case is a seller reporting different usage than predicted; the answer is to reconcile, not to round in your own favour.</p>
<h2>Agents want a different surface</h2>
<p>A large share of traffic now arrives from agents and IDEs, not chat UIs. They need model discovery and schema-shaped tool calling. Shipping an MCP server alongside the HTTP API was the cheapest integration path I found: list models, call completions, no bespoke SDK.</p>
<h2>Three things I would do differently</h2>
<ol>
<li>Write the conformance suite before the second adapter. Retrofitting nine is miserable.</li>
<li>Retain raw upstream response bodies for 24 hours. Otherwise "the model answered differently through your gateway" is undebuggable.</li>
<li>Choose latency or price early. Optimising for both produces a router nobody can reason about.</li>
</ol>
<p>The result is at <a href="https://ai.dijinlong.com">LONG AI</a> - 100+ live LLM and video models with OpenAI SDK compatibility, and the MCP server is open source. Happy to go deeper on routing or settlement in the comments.</p>
]]></content:encoded></item></channel></rss>