Provider Conformance Suite
Run the public TrustedRouter provider conformance suite against an OpenAI-compatible endpoint before applying to the marketplace.
Test the gateway contract before you apply.
trustedrouter-provider-check exercises a provider's model catalog and OpenAI-compatible chat endpoint the way TrustedRouter's attested gateway translator will use them. Run it against the public base URL and a model you intend to advertise.
- Catalog and advertised-model availability
- Non-streaming and SSE chat completion shapes
- Usage accounting and tool-call delta structure
- Stream termination, errors, and first-token behavior
View the GitHub repository Provider marketplace
Open source under the Apache-2.0 license.
uvx trustedrouter-provider-check --base-url https://your-endpoint.example/v1 --model your-modelWhat the suite catches
| Failure mode | Real-world symptom |
|---|---|
stream=true answered with plain JSON | The gateway expects SSE, so it rejects the content type or waits for events that never arrive. |
Missing [DONE] | The stream appears unfinished and can hang until the idle timeout. |
Usage suppressed despite include_usage | The gateway cannot rely on the provider's final token accounting for settlement and reporting. |
Tool-call deltas missing index | Interleaved argument fragments cannot be assigned safely to the correct tool call. |
Mid-stream error after HTTP 200 | The client receives a truncated answer after the request already looked successful, and safe fallback is no longer possible. |
| Buffered streaming | The final body may be correct, but buffering defeats first-token latency and can trigger gateway deadlines. |
An advertised model returns 404 on chat | The catalog publishes a route that the gateway cannot actually serve. |
Your serving engine is not the contract.
vLLM, SGLang, TGI, TensorRT-LLM, llama.cpp, ollama, and custom stacks can all pass when their public API is conformant. The suite asserts the contract of TrustedRouter's attested gateway translator, not the defaults or private interfaces of any one model server.
Compatibility includes failure behavior.
A response can look OpenAI-compatible in a one-line curl test while still breaking streaming, tool calls, usage settlement, fallback safety, or route latency. The suite probes those seams directly.
How to read the results
Fix before onboarding.
fail means the gateway rejects the behavior, silently drops required data, or breaks when it encounters it.
Tolerated, but risky.
warn identifies behavior the gateway currently tolerates but that remains operationally risky or likely to become a failure under load.
Numbers are advisory.
First-token and streaming measurements help find buffering and deadline risk. They are advisory, not a certification or universal performance threshold.
The public suite tracks the deployed contract.
The suite vendors production contract functions from the attested gateway translator. A parity gate in this repository's CI verifies that those vendored functions have not drifted from production.
That gate keeps the check honest: passing the public suite predicts passing the production gateway contract. It does not certify a provider, replace marketplace review, or guarantee future uptime.