agentura

🧠 HarnessAgent

Run SQL agents, code assistants, or research bots on any LLM. Bring your own framework. Memory, safety, and failure recovery come included.

---

What is this?

Think about what actually happens when you run an AI agent in production. The LLM call needs to work. It needs to not cost $500 a day. It needs to not loop forever when the API is slow. It needs to remember context from three messages ago. It needs to not crash your app when one provider goes down.

HarnessAgent handles all of that. You write the task. It handles the rest.

---

Architecture

graph TB
    subgraph CLIENT["🌐 Client Layer"]
        UI[Web / CLI / SDK]
        API[REST API POST /runs]
    end

    subgraph HARNESS["⚙️ HarnessAgent Core"]
        RUNNER[AgentRunner lifecycle manager]

        subgraph AGENTS["🤖 Agent Layer"]
            BASE[BaseAgent run loop]
            SQL[SQLAgent]
            CODE[CodeAgent]
            LG[LangGraph Adapter]
            AG[AutoGen Adapter]
            CR[CrewAI Adapter]
        end

        subgraph MEMORY["🧠 Memory System"]
            STM[Short-Term Redis]
            LTM[Long-Term Qdrant / Chroma / Weaviate]
            GRAPH[Knowledge Graph NetworkX / Neo4j]
            RAG[Graph RAG Engine]
        end

        subgraph LLM["🔮 LLM Router"]
            ROUTER[Health-aware Circuit-broken Router]
            ANT[Claude]
            OAI[GPT-4o / GPT-5]
            LOCAL[vLLM / SGLang / llama.cpp]
        end

        subgraph TOOLS["🔧 Tool System"]
            REG[Tool Registry]
            MCP[MCP Servers]
            SQL2[SQL Tools]
            CODE2[Code Sandbox]
            FILE[File Tools]
        end

        subgraph SAFETY["🛡️ Safety"]
            GUARD[Guardrail Pipeline]
            HITL[Human-in-the-Loop]
            RATE[Rate Limiter]
            CB[Circuit Breaker]
        end
    end

    subgraph OBS["📊 Observability"]
        MLFLOW[MLflow Traces]
        OTEL[OpenTelemetry]
        PROM[Prometheus]
        GRAFANA[Grafana Dashboard]
    end

    subgraph IMPROVE["🔄 Self-Improvement"]
        HERMES[Hermes Loop]
        ERR[Error Collector]
        PATCH[Patch Generator]
        EVAL[Evaluator]
    end

    UI --> API --> RUNNER --> BASE
    BASE --> LLM & MEMORY & TOOLS & SAFETY
    BASE --> OBS
    BASE -.->|failures| IMPROVE
    IMPROVE -.->|better prompts| AGENTS
    ROUTER --> ANT & OAI & LOCAL
    STM & LTM & GRAPH --> RAG
    REG --> MCP & SQL2 & CODE2 & FILE

---

Features

---

LLM Support

No GPU? llama.cpp runs on any Mac or CPU machine. Tool calling works through ReAct text injection when native function calling is not available.

---

Quick Start

# 1. Clone and install
git clone https://github.com/thepradip/HarnessAgent.git
cd HarnessAgent
poetry install

# 2. Configure (set at least one API key, or a local model URL)
cp .env.example .env

# 3. Start infrastructure (Redis, Qdrant, Neo4j, MLflow, Prometheus, Grafana)
docker compose up -d

# 4. Start the API and worker
make api      # terminal 1, FastAPI on port 8000
make worker   # terminal 2, async agent worker

# 5. Run your first agent
curl -X POST http://localhost:8000/runs \
  -H "Content-Type: application/json" \
  -d '{"agent_type": "sql", "task": "How many users signed up this week?"}'

# Watch steps in real time
curl http://localhost:8000/runs/{run_id}/steps

No API key? Use llama.cpp locally:

# Put a GGUF model in ./models/ then:
docker compose --profile local-cpu up -d llamacpp
# Add to .env: LLAMACPP_BASE_URL=http://localhost:8080

---

Use Cases

SQL Data Agent — Ask business questions in plain English. The agent reads your schema into a knowledge graph, writes safe SELECT queries, and returns formatted results with PII redacted.

Code Assistant — Give it a ticket or a spec. It reads your workspace, writes the code, lints it, runs it in a Docker sandbox, and fixes errors until it passes.

Research Agent — Feed it documents or URLs. It ingests them into the vector store and knowledge graph, then answers multi-hop questions with citations.

Multi-Agent Pipeline — Chain specialists through the planner: a researcher feeds a coder, which feeds a reviewer. All agents share the same memory pool.

Existing Framework — Already using LangGraph, AutoGen, or CrewAI? Drop your graph or crew into the adapter. You get traces, cost tracking, circuit breaking, and safety without changing a line of your agent logic.

---

Project Structure

HarnessAgent/
├── src/harness/
│   ├── agents/          # BaseAgent loop, SQLAgent, CodeAgent
│   ├── adapters/        # LangGraph, AutoGen, CrewAI wrappers
│   ├── api/             # FastAPI routes, JWT auth, SSE streaming
│   ├── core/            # Config, circuit breaker, cost tracker, rate limiter
│   ├── eval/            # Datasets, runners, scorers for Hermes evaluation
│   ├── filesystem/      # Isolated workspaces, Docker sandbox, checkpoints
│   ├── improvement/     # Hermes loop, error collector, patch generator
│   ├── ingestion/       # PDF/HTML/MD loaders, chunker, knowledge graph extraction
│   ├── llm/             # Anthropic, OpenAI, local providers, router, factory
│   ├── memory/          # Redis, vector backends, graph, Graph RAG engine
│   ├── messaging/       # Redis Streams inter-agent bus
│   ├── observability/   # MLflow tracer, OTel spans, Prometheus metrics, audit log
│   ├── orchestrator/    # AgentRunner, HITL manager, planner, scheduler
│   ├── prompts/         # Versioned prompt store, patch application
│   ├── safety/          # Guardrail pipeline factory and per-tenant policies
│   ├── tools/           # Tool registry, MCP client, SQL / code / file tools
│   └── workers/         # RQ agent worker, Hermes background scheduler
├── configs/             # Model capabilities, MCP server definitions
├── docs/                # Architecture diagrams and full reference docs
├── infra/               # Prometheus scrape config, OTel collector, Grafana
├── tests/               # 96 unit tests, 2 integration test suites
├── docker-compose.yml   # Full infrastructure: Redis, Qdrant, Neo4j, MLflow, Grafana
├── Dockerfile           # Multi-stage: api, worker, hermes targets
├── Makefile             # install, test, lint, api, worker, hermes, docker-up/down
└── pyproject.toml       # Poetry dependencies and tooling

---

Tech Stack

---

Dashboards

Once docker compose up -d is running:

---

Configuration

Everything goes in .env. Copy .env.example and set what you need.

# Cloud LLMs
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
OPENAI_MODELS=gpt-4o-mini          # comma-separated, e.g. gpt-4o-mini,gpt-4o

# Local LLMs (no API key needed)
VLLM_BASE_URL=http://localhost:8000
LLAMACPP_BASE_URL=http://localhost:8080

# Memory backends (chroma is default, zero setup)
VECTOR_BACKEND=chroma              # chroma | qdrant | weaviate
GRAPH_BACKEND=networkx             # networkx | neo4j

# Hermes self-improvement
HERMES_AUTO_APPLY=false            # keep this off until you trust it
HERMES_PATCH_SCORE_THRESHOLD=0.7

# Cost and safety
COST_BUDGET_USD_PER_TENANT=100.0
RATE_LIMIT_RPM=60

Full reference: docs/guides/CONFIGURATION.md

---

Testing

# Run unit tests
PYTHONPATH=src python3 -m pytest tests/unit/

# Run integration tests (needs SQLite, no Docker required)
PYTHONPATH=src python3 -m pytest tests/integration/

# With coverage
PYTHONPATH=src python3 -m pytest tests/ --cov=src/harness --cov-report=term-missing

Current: 96 unit tests passing, 0 failures.

---

Documentation

• Architecture Overview — C4 diagrams, every system flow
• Quick Start Guide — three setup paths: cloud, local, production
• Configuration Reference — every env var explained
• Deployment Guide — Docker Compose to Kubernetes
• Component Reference — all 17 components documented
• Code Walkthrough — follow a request through the actual code
• Troubleshooting — common issues and fixes
• HTML Docs — open in browser, click Export PDF for a printable version

---

Future Scope

Planned improvements focused on making HarnessAgent more efficient at scale.

---

Contributing

Fork, branch off main, write tests for anything new, open a PR.

git checkout -b feat/your-feature
PYTHONPATH=src python3 -m pytest tests/unit/
ruff check src/ tests/

Things that would be useful: new LLM provider adapters, additional vector backends, more tool integrations, Kubernetes Helm chart, and examples for specific use cases.

---

License

MIT. See LICENSE.

---

Architecture  |  Quick Start  |  Components  |  Issues