ai-auto-router-client

ai-auto-router-client

A lightweight, zero-external-dependency Python client for the AI Auto Router BYOK gateway.

Connects your application directly to your AI Auto Router instance using standard library primitives (urllib). Completely decouples client-side request orchestration from your private routing algorithms, Jev decision trees, and provider quality scoring.

Installation

pip install ai-auto-router-client

Quick Start (3-Line Drop-in)

from ai_auto_router_client import AutoRouterClient

# Defaults to the hosted API (https://api.sentiogate.dev). Self-hosting? Pass base_url="http://localhost:8000".
router = AutoRouterClient(api_key="sg_live_...")
resp = router.chat(
    messages=[{"role": "user", "content": "Explain raft consensus in 2 sentences"}],
    provider_keys={"openai": "sk-...", "anthropic": "sk-ant-..."}
)
print(resp["choices"][0]["message"]["content"])

Streaming Support

for chunk in router.stream_chat(
    messages=[{"role": "user", "content": "Write a quicksort in Python"}],
    provider_keys={"openai": "sk-..."}
):
    delta = chunk.get("choices", [{}])[0].get("delta", {}).get("content", "")
    print(delta, end="", flush=True)

Timeouts

The default request timeout is 130 s (the community pool can take up to the server's 120 s deadline); pass timeout=<seconds> to change it.

Community Pool (no provider keys)

With a SentioGate API key and no provider keys, requests are served by the free community pool:

• One fixed model. Use model: "auto" (default) or "sentiogate-community"; responses report the model as sentiogate-community. Routing options (difficulty, mode, constraints, hedging*, plugins, cascade) don't apply on the community pool and are ignored.
• BYOK and community are mutually exclusive. If your Vault holds provider keys, asking for sentiogate-community (chat or batch) returns 403 community_pool_disabled_with_byok. Use "auto" or a provider/model id to route with your own keys instead.
• Rate limits. When the upstream limits the pool, every caller gets a fast 429 upstream_rate_limited with Retry-After set to the seconds left on the shared cooldown (60 s to 15 min). While the pool tests recovery, waiting callers get Retry-After: 5. The 5-hour window being full returns 429 community_pool_exhausted (not worth retrying soon).

Errors, Retry-After and Retries

Every failure raises RouterClientError with status_code, code (e.g. upstream_rate_limited), error_type, retry_after (seconds, from the Retry-After header) and response_body. Streaming errors arrive before the first chunk and raise the same way.

from ai_auto_router_client import AutoRouterClient, RouterClientError

# Retries are off by default. max_retries > 0 retries only transient 429s, only after the server's
# Retry-After (at least 1 s), and never when that is longer than max_retry_wait seconds.
router = AutoRouterClient(api_key="sg_live_...", max_retries=2, max_retry_wait=30)
try:
    router.chat(messages=[{"role": "user", "content": "hi"}])
except RouterClientError as e:
    if e.code == "upstream_rate_limited":
        print(f"Community pool cooling down; retry in {e.retry_after:.0f}s")

Batch Queue & Overflow

Enqueue non-urgent tasks or absorb capacity limits asynchronously:

# 1. Enqueue batch job
enq = router.enqueue_batch(
    messages=[{"role": "user", "content": "Generate 10 synthetic test cases"}],
    model="sentiogate-community"
)
print(f"Enqueued job: {enq['job_id']}")

# 2. Poll result
status = router.poll_batch(enq["job_id"])
if status.get("status") == "completed":
    print(status["response"]["choices"][0]["message"]["content"])

Key Benefits

• Zero Third-Party Dependencies: Written entirely in pure Python standard library (urllib, json). No httpx, requests, or heavy runtime bloat.
• Strict Privacy / Core Isolation: Client never contains internal scoring logic or proprietary provider decision weights. All routing decisions are executed server-side.
• Provider Key Forwarding: Passes keys transiently via HTTPS to your router; zero persistence of keys on disk.
• Telemetry Outcome Reporting: Report user accept/reject outcomes to power your adaptive router learning.

License

Apache-2.0