agentsave-inferroute

Part of AgentSave — the Python SDK that cuts AI agent token costs ~23%. This repo (agentsave-inferroute) is an Enterprise-tier component and requires a valid AgentSave Enterprise license key. Only deploy this if you operate a vLLM or sGLang inference cluster and hold an Enterprise license.

agentsave-inferroute — PPD Inference Router

agentsave-inferroute is a FastAPI HTTP proxy sidecar that implements PPD (append-Prefill-Decode) routing for multi-turn LLM agent workloads. It classifies each request as a Turn 1 (prefill-heavy) or Turn 2+ (decode-heavy) conversation turn, then dispatches it to the backend optimized for that access pattern — targeting a ~68% TTFT reduction for Turn 2+ requests over a uniform-routing baseline.

---

What is PPD Routing?

Modern LLM inference backends are tuned differently depending on the memory access pattern of the workload:

• Turn 1 (prefill-heavy): The model processes a long prompt (system prompt, tool schemas, initial user message) for the first time. The computation is dominated by the attention prefill pass over a large KV cache miss. A prefill-optimized backend (higher parallelism, larger chunked prefill batch) handles this efficiently.

• Turn 2+ (append-heavy / decode-heavy): The conversation history is already cached. The model appends only the new user message and generates the next assistant turn. The computation is dominated by autoregressive decoding over a warm KV cache. A decode-optimized backend (larger batch, speculative decoding, lower prefill overhead) handles this efficiently.

PPD routing separates these two request classes at the proxy layer and sends each to its optimal backend. The routing decision is made per request with zero latency overhead — the classifier is a single pass over the messages[] array.

The ~68% Turn 2+ TTFT reduction figure is a static architectural estimate for v0.1.0, derived from the PPD routing design and the static improvement estimates embedded in the scorer (ttft_improvement = 68.0 ms, tpot_degradation = 5.0 ms). It has not been measured on a real vLLM/SGLang cluster — real instrumentation against live inference backends is future work. Treat this as a starting point to validate and tune for your own hardware, model, and traffic mix, not a benchmarked result.

---

Architecture

flowchart LR
    Agent["AI Agent\n(any framework)"]
    IR["agentsave-inferroute\n:8080\n/v1/chat/completions"]
    CL["Turn Classifier\nTURN1 / TURN2+"]
    SC["PPD Scorer\nroute_score = w_ttft·Δttft − w_tpot·Δtpot"]
    PB["Prefill Backend\n(vLLM / sGLang)\nTurn 1"]
    DB["Decode Backend\n(vLLM / sGLang)\nTurn 2+"]
    MT["/metrics\n(Prometheus-compatible)"]

    Agent -->|POST /v1/chat/completions| IR
    IR --> CL
    CL --> SC
    SC -->|score ≤ 0 or TURN1| PB
    SC -->|score > 0 and TURN2+| DB
    IR -->|fire-and-forget| MT

The proxy is transparent to the upstream agent — it accepts and returns standard OpenAI-compatible /v1/chat/completions JSON, including streaming ("stream": true).

Classification rule: if messages[] contains any entry with "role": "assistant", the request is TURN2_PLUS; otherwise it is TURN1.

Scoring rule: route_score = PPD_W_TTFT × ttft_improvement − PPD_W_TPOT × tpot_degradation. With default weights (0.7 / 0.3) and static estimates (68.0 ms / 5.0 ms), score = 46.1, so all TURN2_PLUS requests are decode-routed.

---

Quick Start

Prerequisites

• Docker
• A running vLLM or sGLang cluster with two endpoints: one prefill-optimized, one decode-optimized
• A valid AgentSave Enterprise license key (set as AGENTSAVE_TOKEN)

Run as a Docker sidecar

docker run -d \
  --name inferroute \
  -p 8080:8080 \
  -e BACKEND_URL=http://your-primary-backend:8000 \
  -e BACKEND_TYPE=vllm \
  -e AGENTSAVE_TOKEN=your-enterprise-license-key \
  ghcr.io/aks-builds/agentsave-inferroute:latest

Point your agent at http://localhost:8080 instead of your inference backend directly. All /v1/chat/completions calls are automatically classified and routed.

Verify it is running

curl http://localhost:8080/health
# {"status":"ok","backend_type":"vllm","version":"0.1.0"}

---

Configuration

All configuration is via environment variables. No config file is required.

Backend adapter signal details

---

API Endpoints

POST /v1/chat/completions

Accepts any OpenAI-compatible chat completions request body. Classifies the turn, scores the routing decision, and proxies the request to the appropriate backend.

• Requires a valid Enterprise license. If AGENTSAVE_TOKEN is missing, malformed, expired, or not tier "enterprise", this endpoint returns 403 with a JSON {"detail": "..."} body explaining the problem, and never contacts the backend. /health still works unauthenticated for orchestrator liveness checks.
• Streaming ("stream": true) is fully supported via StreamingResponse.
• The request body and all headers are forwarded to the upstream backend unchanged, except for the routing signal injected by the adapter.
• Upstream timeout: 120 seconds.

Example:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Summarize the quarterly report."}
    ]
  }'

GET /health

Liveness probe. Always returns HTTP 200 — regardless of license state, so orchestrators can still tell the container is alive — with a JSON body including the configured backend type, router version, and current license status.

{
  "status": "ok",
  "backend_type": "vllm",
  "version": "0.1.0",
  "license": {"valid": true, "tier": "enterprise", "org": "Acme Corp", "error": null}
}

GET /metrics

Prometheus-compatible metrics endpoint. Reports routing decisions and backend activity for observability integration.

---

Testing

The test suite covers the classifier, scorer, dispatcher, adapters, metrics emitter, and full end-to-end routing paths.

# Install development dependencies
pip install -e ".[dev]"

# Run the full suite
pytest tests/ -v

77 tests across 10 files, verified on Python 3.11, 3.12, and 3.13. CI runs the full suite on every push and pull request.

---

Contributing

This is a proprietary Enterprise component. External contributions are not accepted at this time. If you are an Enterprise customer and have found a bug or have a feature request, open an issue or contact your AgentSave account team.

For the open-source SDK, see aks-builds/agentsave.

---

License

Proprietary. Use of this software requires a valid AgentSave Enterprise license. See LICENSE for terms.

---

agentsave-inferroute v0.1.0 — part of the AgentSave ecosystem