agento-template

Agent_head — Main Autonomous Orchestrator

Agent_head is the central orchestrator agent for the orchestra multi-agent system. It coordinates multiple specialized Worker Agents and direct MCP (Model Context Protocol) tool servers through a unified LangGraph ReAct loop. This enables complex, multi-step tasks that require coordination across different domains and tools.

Agent_head can also natively run as an MCP server itself or as a REST API Backend, enabling you to build expansive agent networks spanning multiple orchestrators collaborating on heavy computation tasks.

---

📚 Documentation

Detailed system mechanics and configuration guides have been moved to dedicated documentation files:

• capabilities and User Guide: Explore the different operational modes, details on Multi-Agent brains, native tooling details, and config.yaml breakdowns.
• Technical Implementation: Deep dive into the internal component architecture (LangGraph loop, summarizers, loading systems). Predates the engine//channels/ rewrite — see below.
• docs/README.md: index of all documentation, including .claude/docs/, which covers the engine/channels rewrite, the FastAPI route inventory, current security findings, and subsystems added since late August 2026 (heartbeat, tool approval, MCP OAuth, media store, device pairing, P2P networking) that this README and docs/architecture.md do not describe.

---

What is Agent_head?

Agent_head acts as the highly-agile "brain" of your software interface:

• Autonomous Execution: Uses LangChain/LangGraph for reasoning and tool calling
• Multi-Agent Coordination: Spawns and delegates tasks to specialized worker agents
• MCP Server Mode: Expose the orchestrator as an MCP server for other agents/clients
• Agent Networking: Connect multiple Agent_head instances together with shared sessions
• MCP Integration: Connects to any MCP-compatible tool servers
• Persistent Session History: Full conversation archive in SQLite — never lost, even after summarisation
• Memory & Context: Maintains conversation history, facts, and auto-injects relevant context
• Interactive & Batch Modes: REPL interface or single-shot task execution
• Rich Logging: Per-job structured logs for debugging and auditing
• API Server: REST API with streaming SSE for programmatic access

Architecture

                          ┌──────────────┐
                          │ Claude       │
                          │ Desktop /    │ ← External MCP clients
                          │ Cursor / etc │
                          └──────┬───────┘
                                 │ (MCP)
┌────────────────────────────────▼────────────────────────────────┐
│                        Agent_head                               │
│                    (Orchestrator + MCP Server)                  │
│                                                                 │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────────────┐  │
│  │ LangGraph    │  │   Memory     │  │   MCP Server          │  │
│  │ ReAct Agent  │  │   (RAG)      │  │   (8 tools exposed)   │  │
│  └──────────────┘  └──────────────┘  └───────────────────────┘  │
│                                                                 │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────────────┐  │
│  │ Summarizer   │  │  Sessions    │  │  Progress Streaming   │  │
│  │ (windowed)   │  │  (SQLite)    │  │  (SSE / ctx.info)     │  │
│  └──────────────┘  └──────────────┘  └───────────────────────┘  │
└────────────────────────────┬───────────────────────────────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
       ┌──────▼─────┐ ┌──────▼─────┐ ┌──────▼─────┐
       │  Worker    │ │  Worker    │ │   MCP      │
       │  Agent A   │ │  Agent B   │ │  Tools     │ ← filesystem, search, etc.
       │  (Agent_a) │ │  (Agent_a) │ │  (direct)  │
       └────────────┘ └────────────┘ └────────────┘

       ┌──────────────────────────────────────────┐
       │         Agent Network (optional)          │
       │                                          │
       │  Agent_head ◄──SSE/HTTP──► Agent_head    │
       │      ↕                        ↕          │
       │  Agent_head ◄──stdio──► Agent_head       │
       │                                          │
       │  Shared sessions, multi-agent identity   │
       └──────────────────────────────────────────┘

Key Components

Features

• Multi-Modal LLM Support: Ollama, OpenAI, Google Gemini, Anthropic, AWS Bedrock, NVIDIA NIM
• Worker Agent Delegation: Automatic task routing to specialists
• MCP Server Mode: Expose the orchestrator as an MCP server with 8 tools
• Agent Networking: Connect multiple agents — shared sessions, identity tagging, supervisor monitoring
• All MCP Transports: stdio, SSE, and Streamable HTTP
• Configurable Progress Streaming: None / Summary / Full verbosity per call
• Direct MCP Tools: Filesystem, shell execution, web scraping, etc.
• Memory System: Fact storage and semantic search (RAG / ChromaDB)
• Persistent Session History: Two-layer SQLite storage — windowed working copy for the LLM, unabridged archive for debugging
• Auto-Context Injection: Relevant memory auto-fed to LLM before each turn
• Rolling Summarisation: Keeps the LLM context window bounded; full history still preserved in archive
• Image & Audio Tools: Screenshot, OCR, TTS, transcription, recording
• Command Tools: Claude Code-style shell execution (run_command, get_command_output, kill_command, list_commands, monitor_command) — persistent cwd, timeouts, background tasks with completion messages, streamed monitors; confined to ./workspace with a scoped env, and every command except read-only allowlisted ones (command_tools.auto_approve) requires human approval
• Notification Listening: Real-time tool-change monitoring via Agent_notify
• Structured Logging: Per-job logs with full traces
• Graceful Error Handling: Tool failures don't crash the agent

Installation

Prerequisites

• Python 3.10+
• uv package manager (recommended) or pip
• For Ollama: Running Ollama server with models pulled
• For OpenAI / Gemini / Anthropic: API keys configured

Setup using UV

uv tool install --force git+https://github.com/tharindumendis/agent_orchestrator_template.git

Setup via Source

# Clone the repo
git clone <repository-url>
cd agent_orchestrator_template

# Create virtual environment
uv venv .venv

# Activate environment (Windows)
.venv\Scripts\activate

# Install dependencies
uv sync

# Or with pip
pip install -e .

Development Setup

uv sync --group dev
# Or
pip install -e ".[dev]"

Security profiles

Pick what kind of agent you're building and start from its profile:

agento --init --profile personal         # private assistant on your own computer
agento --init --profile public_chatbot   # public domain chatbot for any website/app
agento --init --profile server_bot       # 24/7 server bot with scheduled jobs
agento --init --profile sandbox          # chat agent confined to ./workspace

Who can reach the agent (admin token, paired phones, API keys, your app's JWTs, anonymous visitors, Telegram/Discord senders, teammates' agents) and what each of them may do is all configuration — see .claude/docs/security-model.md.

Configuration

Run the setup wizard to generate a config in the current directory:

agento --init

(--setup is kept as a back-compat alias for --init.) This creates .agents/config.yaml which is auto-loaded on next run. On the first plain agento run afterward, .agents/runtime_config.json is generated from it exactly once — from then on runtime_config.json is the sole live source of truth for the whole process; config.yaml is never rewritten automatically. Edit config.yaml for version-controlled defaults, or patch the running config via REST/MCP/the web UI for live changes. Two more CLI flags round out the lifecycle:

agento --reset          # regenerate runtime_config.json from config.yaml, discarding runtime patches
agento --global-path    # print the OS-wide fallback config directory and exit

Agent Configuration

agent:
  name: "OrchestratorAgent"
  version: "1.0.0"
  debug: false         # true → writes full prompt/response logs to .agents/logs/runs/
  system_prompt: |
    You are a powerful autonomous orchestrator agent...
  max_iterations: 50

Model Configuration

The legacy single model: block is still supported (auto-migrated into the registry below on first load), but new configs should use the model registry: any number of models can be registered, and each model-consuming position (agent, summarizer, heartbeat) picks one directly by id — no shared "role" name to fight over reusing:

models:
  registry:
    - id: "primary"
      provider: "ollama"   # "ollama" | "openai" | "gemini" | "anthropic" | "bedrock" | "nvidia"
      model_name: "qwen3:32b"
      temperature: 0.0
      base_url: "http://localhost:11434"   # Ollama only
      api_key: ""                          # OpenAI / Gemini / Anthropic
    - id: "fast-summarizer"
      provider: "ollama"
      model_name: "qwen3:8b"
      temperature: 0.0

agent:
  model_id: "primary"          # "" = fall back to the first registry entry

Models can also be registered and reassigned at runtime — via GET/POST /engine/models, DELETE /engine/models/{id}, PATCH /engine/models/positions/{position}, or the "Models" tab in agento-webui — without editing config.yaml or restarting. A running session can additionally be pinned to its own model independent of the global default via POST /engine/sessions/{session_id}/model (or the per-session selector in the web UI's chat header).

Worker Agents

worker_agents:
  - name: "core-agent"
    description: "General-purpose worker"
    command: "agento-mcp"
    args: []
    env:
      WORKER_AGENT_CONFIG: "./service_config/worker_config.yaml"

Direct MCP Clients

mcp_clients:
  - name: "filesystem"
    command: "npx.cmd"
    args: ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/dir"]

  - name: "shell"
    command: "npx.cmd"
    args: ["-y", "shell-exec-mcp"]

Memory Configuration

memory:
  enabled: true
  backend: "rag"           # "rag" (ChromaDB) or "jsonl"
  memory_dir: "./memory"
  max_save_length: 500
  auto_feed_top_k: 3       # chunks injected before each turn
  auto_feed_category: "all" # "all" | "history" | "facts"

  rag_server:
    command: "uvx"
    args: ["agent-rag-mcp"]
    env:
      RAG_CONFIG: "./service_config/rag_config.yaml"

Embedding models used by the RAG backend are likewise registered by id and referenced, rather than inlined, mirroring the LLM registry above:

embedding_models:
  registry:
    - id: "default"
      provider: "sentence_transformers"
      model: "all-MiniLM-L6-v2"

memory:
  rag:
    embedding_model: "default"

Add/remove entries and set the default at runtime via GET/POST /engine/embedding-models, DELETE /engine/embedding-models/{id}, PATCH /engine/embedding-models/default, or the "Models" tab in agento-webui.

Chat History (Session Persistence)

chat_history:
  backend: "sqlite"
  connection_string: "./session_db/sessions.db"

Two tables are maintained automatically:

Summarizer

summarizer:
  enabled: true
  summarize_every_n_messages: 8   # compress after this many new Human+AI messages
  keep_recent_messages: 8         # keep this many raw messages for the LLM feed
  save_to_memory: true            # persist extracted facts to long-term memory
  model_id: "fast-summarizer"     # references models.registry — "" falls back to the agent's model

(The legacy inline summarizer.model: block is still supported for back-compat and is auto-migrated into models.registry on first load, but model_id is the preferred way to pick a lighter/cheaper model for summarization now.)

Note: Summarisation only shrinks the working copy fed to the LLM. The full unabridged conversation is always preserved in the session_archive table and accessible via GET /history/sessions/{id}/full-export.

Usage

Interactive REPL

agento

This command seamlessly creates the .agents hidden folder in your working directory and instantiates config.yaml alongside specialized agent configs, keeping your agent logic tightly bound to your project environment.

Starts an interactive prompt (all enabled channels — CLI, web, MCP, etc. — per config.yaml). Each turn is persisted to the engine's session store as it happens. Note: the CLI channel's session is keyed to your local user identity for the life of the running process — it does not currently resume the same conversation across a full restart of agento. Type quit or press Ctrl-C to exit.

Running the Orchestrator

Single-shot task, no interactive channels:

agento --task "Analyse the codebase and suggest improvements"

Custom config path:

agento --config /path/to/custom_config.yaml

Disable specific channels (all are on by default per config.yaml; these flags force one off regardless of config):

agento --no-web              # skip the FastAPI web/WebSocket server
agento --no-cli               # skip the interactive CLI channel
agento --no-mcp               # skip the FastMCP server
agento --no-telegram          # skip the Telegram channel
agento --no-discord           # skip the Discord channel

Model selection is config-driven, not a CLI flag — set it in config.yaml's models.registry (see "Model Configuration" below) or change it live via the POST /engine/models API / POST /engine/sessions/{id}/model for one session.

Export Config for Editing

# Export to current directory (.agents/ subfolder)
agento --init

# Export to a specific project directory
agento --init /path/to/my-project

(--setup remains a back-compat alias for --init.)

API Server

agento-api                          # http://127.0.0.1:9000 by default (channels.web.port)
agento-api --port 9001
agento-api --config /path/to/config.yaml

Endpoints

SSE Event Types

{"type": "tool_call",   "name": "...", "args": {...}}
{"type": "tool_result", "name": "...", "content": "..."}
{"type": "token",       "content": "..."}   // intermediate AI text
{"type": "done",        "content": "..."}   // final answer
{"type": "error",       "content": "..."}   // something went wrong

MCP Server Mode

Run Agent_head as an MCP server so other agents, Claude Desktop, or Cursor can connect:

# stdio transport (default — Claude Desktop, Cursor, other agento-mcp instances)
agento-mcp
agento-mcp --config /path/to/config.yaml

# SSE transport (LAN / internet agent networks)
agento-mcp --transport sse --port 9000 --host 0.0.0.0

# Streamable HTTP (modern MCP standard, production use)
agento-mcp --transport http --port 9000

8 MCP tools exposed:

Session History — How It Works

Agent_head uses a two-layer storage model so you never lose conversation history:

Every turn
  │
  ├─► append_to_archive()    ← PERMANENT — every message, never trimmed
  │
  ├─► save_session()         ← WORKING COPY — windowed slice for LLM input
  │
  └─► [if threshold hit] summarize()
        │
        └─► save_session(trimmed_history)   ← WORKING COPY shrinks
            (archive stays untouched)

On session resume:

• LLM is fed the working copy (summary + recent N messages)
• Full archive is available via GET /history/sessions/{id}/full-export for debugging

Logs and Debugging

Debug Mode

# config.yaml
agent:
  debug: true

When enabled, every prompt fed to the LLM is written to .agents/logs/runs/<session_id>.log.

Can also be flipped live from agento-webui's Config page (Agent tab → "Debug Mode" toggle) — it takes effect immediately on already-running sessions, no restart required, since it's a pure logging switch rather than something baked into a session's compiled graph.

Raw AI Response Dumps

While agent.debug: true, every AI response received during a live session's turn (not just the legacy single-shot path above) is additionally written out as its own complete JSON file — full content, additional_kwargs (provider-specific extras), response_metadata (model name, finish reason, etc.), tool_calls, and usage_metadata (token counts), not just the reply text:

logs/sessions/raw_responses/
├── sess-abc123_turn0001_2026-08-19T02-10-03.120Z.json
├── sess-abc123_turn0002_2026-08-19T02-10-07.884Z.json
└── ...

Job Logs (MCP mode)

Each MCP task creates a structured log in the configured log_dir:

logs/mcp/jobs/
├── 2025-04-20_12-00-00_abc123.log
└── 2025-04-20_12-05-30_def456.log

Troubleshooting

Worker agents not connecting

• Check paths / commands in config.yaml
• Ensure worker virtual environments are activated
• Verify the worker command is on PATH (e.g. agento-mcp)

MCP tools failing

npx @modelcontextprotocol/server-filesystem --help   # verify install

• Check transport settings (stdio vs SSE vs HTTP)
• Look for port conflicts

Memory not working

• Check ChromaDB installation: pip show chromadb
• Verify memory/ directory has write permissions
• Ensure the RAG server command (uvx agent-rag-mcp) is installed

History / archive not saving

• Confirm chat_history.backend: "sqlite" in config
• Check that connection_string path is writable

Performance Tuning

• Lower summarize_every_n_messages to compress more aggressively
• Set keep_recent_messages: 4-6 for smaller context windows
• Use a lighter model for the summarizer (separate summarizer.model config)
• Reduce max_iterations for faster single-turn responses

Project Structure

agent_orchestrator_template/
├── main.py                         # Entry point (installed as `agento`) — starts all enabled services
├── api/
│   └── server.py                   # FastAPI REST + WebSocket server (legacy /sessions/* + new /engine/* routes)
├── engine/                         # Centralized agent engine (new architecture)
│   ├── factory.py                  # EngineFactory — routes channel messages to sessions
│   ├── session_registry.py         # SessionRegistry — per-channel_id AgentSession lifecycle
│   ├── event_bus.py                # Pub/sub for turn events (tool_call/token/done/...)
│   ├── persistence.py              # SQLite persistence (engine.db) — messages + turns
│   ├── heartbeat.py                # Proactive scheduled/heartbeat turns
│   ├── notifications.py            # Outbound agent → human notification sink
│   ├── tool_approval.py            # Approval-gated tool execution
│   ├── media_store.py              # Media attachment storage
│   ├── models.py                   # EngineEvent and related dataclasses
│   └── current.py, turn_runner.py
├── channels/                       # Interface adapters that attach to the engine
│   ├── base.py                     # BaseChannel abstraction
│   ├── cli_channel.py              # CLI/TUI channel
│   ├── web_channel.py              # Web/browser channel
│   ├── discord_channel.py
│   └── telegram_channel.py
├── core/                           # Shared runtime library used by both architectures
│   ├── agent/                      # LangGraph ReAct loop, REPL, TUI
│   │   ├── agent.py                 # Single-shot orchestrator
│   │   ├── agent_runner.py          # Interactive per-turn ReAct loop
│   │   ├── repl.py                  # Plain-text REPL (TUI fallback)
│   │   └── tui.py, tui_*.py          # Textual full-screen TUI + config/env/history screens
│   ├── config/                     # Config loading, runtime store, secrets
│   │   ├── config_loader.py, config_models.py
│   │   ├── runtime_config.py, runtime_overlay.py
│   │   └── secret_ref.py, secrets_store.py, mutation_policy.py, mcp_import.py, paths.py
│   ├── llm/                         # LLM provider factory + summarization
│   │   ├── llm.py                   # Provider dispatch (openai/anthropic/bedrock/ollama/...)
│   │   ├── conversation_summarizer.py
│   │   ├── summarization_apply.py, summarizer_settings.py, token_estimate.py
│   ├── mcp/                         # MCP client/server
│   │   ├── mcp_loader.py, mcp_server.py, mcp_tools.py
│   ├── memory/                      # Long-term memory + conversation history backends
│   │   ├── history.py, history_sqlite.py         # Conversation history (legacy path)
│   │   ├── memory.py, memory_sqlite.py, memory_rag.py  # Long-term memory backends
│   │   └── rag/                     # Chunk → embed → store → watch pipeline
│   ├── notifications/               # Inbound notification sources (Discord/Twitter/webhooks/...)
│   │   └── social/
│   ├── session/                     # AgentSession primitive (used by MCP server + engine)
│   │   ├── session.py, session_logger.py, job_logger.py
│   └── tools/                       # Tool assembly pipeline
│       ├── tool_registry.py, task_scheduler.py, skill_loader.py
│       └── image_tools.py, audio_tools.py, notify_tool.py, self_config_tool.py, sleep_tool.py, summarize_tool.py
├── skills/                         # Skills directory (SKILL.md files)
├── docs/                           # Architecture + reference docs
├── config.yaml                     # Default config
├── pyproject.toml                  # Package config
├── .agents/                        # Local project config (auto-created by --init)
│   ├── config.yaml                 # Factory defaults — never rewritten by the running app
│   ├── runtime_config.json         # Live config — sole source of truth once generated
│   ├── service_config/
│   ├── logs/runs/                  # Per-session debug logs
│   └── skills/                     # Project-local skills
├── memory/                         # Long-term memory storage
└── service_config/                 # Worker + service configs

Development

Adding New Features

• New Tools: Create a @lc_tool decorated function and add it to all_tools in main.py / api/server.py / core/mcp/mcp_server.py
• New LLM Providers: Add a branch in core/llm/llm.py's get_llm() factory
• New Models: No code change needed — add an entry to models.registry (or embedding_models.registry) in config, or register one at runtime via POST /engine/models / the agento-webui Models tab
• New Memory Backends: Implement ConversationHistoryBackend in core/memory/history.py
• New Skills: Create skills/<name>/SKILL.md — auto-discovered, no code changes needed

Testing

pytest
pytest --cov=core --cov-report=html

Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes with tests
4. Submit a pull request

Code style: black formatting, PEP 8, type hints, docstrings.

License

MIT License — see LICENSE file for details.

Changelog

v1.2.0

• Full Session Archive: Two-layer SQLite storage — sessions (windowed, LLM feed) + session_archive (append-only, never trimmed). Full history preserved even after summarisation.
• GET /history/sessions/{id}/full-export: New API endpoint to retrieve the complete, unabridged session conversation.
• Default Session: REPL now always persists — no --session flag defaults to "default" session. Use --session no for ephemeral (no persistence) mode.
• System Prompt Guarantee: System prompt is always position-0 in conversation history on resume, even if summarisation had previously trimmed it.
• mcp_server.py Bug Fixes: Fixed datetime.now() crash (was calling method on module, not class), fixed image tools loading inside except block (only loaded when memory failed), added archive support.
• Anthropic / Bedrock Support: Added provider support in core/llm/llm.py.

v1.1.0

• MCP Server Mode: Agent_head can now run as an MCP server (agento-mcp)
• Agent Networking: Multi-agent shared sessions with identity tagging
• 8 MCP Tools: orchestrate_task, chat, create_session, list_sessions, get_session_history, list_agents, get_status, close_session
• Configurable Progress Streaming: none / summary / full verbosity
• All MCP Transports: stdio, SSE, Streamable HTTP
• Image Tools: read, save, screenshot, OCR
• Audio Tools: transcribe, TTS, save, record, play, speak

v1.0.0

• Initial release
• Multi-agent orchestration
• MCP integration
• Memory system
• REST API
• Interactive REPL

---

For more information, see the orchestra system documentation.