agent-vision-mcp

agent-vision-mcp

English | 中文

Give MCP-compatible AI agents image analysis, metadata inspection, cropping, OCR, and image comparison through any OpenAI-compatible vision model.

Features

• Analyze screenshots, charts, documents, UI, objects, and general images.
• Inspect image dimensions and metadata without calling a model.
• Crop and zoom into regions using normalized coordinates.
• Extract visible text with a VLM or an optional dedicated OCR model.
• Compare two to four images.
• Accept public URLs, local files, data URLs, and Base64 images.
• Run locally over the standard MCP stdio transport.

Claude Code

Requirements

• Python 3.10 or newer
• uv
• An OpenAI-compatible vision API endpoint and API key

uvx downloads the published package from PyPI into an isolated environment and runs it. It does not use the source code in your current directory and does not permanently install the package into your system Python.

Add To Claude Code

The command below configures Claude Code to start agent-vision-mcp from PyPI:

claude mcp add --scope user agent-vision \
  --env UV_DEFAULT_INDEX=https://pypi.org/simple \
  VISION_API_KEY="your-api-key" \
  VISION_BASE_URL="https://your-provider.example/v1" \
  VISION_MODEL_ID="your-vision-model" \
  -- uvx agent-vision-mcp

Use UV_DEFAULT_INDEX=https://pypi.org/simple when your local PyPI mirror has not synchronized the latest release.

Verify the connection:

claude mcp get agent-vision
claude mcp list

Then start Claude Code and ask:

Use vision_capabilities to show the available vision tools.

Analyze a local image:

Use vision_inspect on /data/example.png, then use vision_analyze to describe it.

By default, local image access is limited to /data and /tmp. Add another directory with:

claude mcp remove --scope user agent-vision

claude mcp add --scope user agent-vision \
  --env UV_DEFAULT_INDEX=https://pypi.org/simple \
  VISION_API_KEY="your-api-key" \
  VISION_BASE_URL="https://your-provider.example/v1" \
  VISION_MODEL_ID="your-vision-model" \
  VISION_ALLOWED_PATHS="/data,/tmp,/home/your-user/Pictures" \
  -- uvx agent-vision-mcp

Dedicated OCR Model

Without dedicated OCR configuration, vision_extract_text uses the configured vision model. To use a separate OCR model:

claude mcp add --scope user agent-vision \
  --env UV_DEFAULT_INDEX=https://pypi.org/simple \
  VISION_API_KEY="your-vision-api-key" \
  VISION_BASE_URL="https://your-provider.example/v1" \
  VISION_MODEL_ID="your-vision-model" \
  OCR_ENABLED=true \
  OCR_API_KEY="your-ocr-api-key" \
  OCR_BASE_URL="https://your-provider.example/v1" \
  OCR_MODEL_ID="your-ocr-model" \
  -- uvx agent-vision-mcp

Never commit real API keys to Git.

Other MCP Clients

Use this stdio configuration with MCP clients that accept JSON configuration:

{
  "mcpServers": {
    "agent-vision": {
      "command": "uvx",
      "args": ["agent-vision-mcp"],
      "env": {
        "UV_DEFAULT_INDEX": "https://pypi.org/simple",
        "VISION_API_KEY": "your-api-key",
        "VISION_BASE_URL": "https://your-provider.example/v1",
        "VISION_MODEL_ID": "your-vision-model"
      }
    }
  }
}

Tools

Response format

Every tool returns a JSON string. Clients must json.loads the result before reading any field. All top-level keys are always present (even when empty), so consumers can iterate the envelope without dict.get(...) guards.

Success envelope

{
  "schema_version": "1.0",
  "ok": true,
  "tool": "vision_analyze",
  "task": "general",
  "model": "...",
  "source": null,
  "sources": [],
  "result": {},
  "warnings": [],
  "raw_model_output": null,
  "error": null
}

SourceMeta fields: type (url / file / data_url / base64), mime_type, width, height, size_bytes, source_ref (only when include_source_ref=true; redacted to host/path for URLs or basename for files; null for data URLs and base64).

Failure envelope

{
  "schema_version": "1.0",
  "ok": false,
  "tool": "vision_analyze",
  "task": "general",
  "model": "...",
  "source": null,
  "sources": [],
  "result": null,
  "warnings": [],
  "raw_model_output": null,
  "error": {
    "code": "INVALID_INPUT",
    "message": "Input is not a valid supported image",
    "retryable": false,
    "details": {}
  }
}

error.code values: INVALID_INPUT, IMAGE_TOO_LARGE, UNSUPPORTED_FORMAT, SECURITY_ERROR, PROVIDER_ERROR, TIMEOUT, INTERNAL_ERROR. retryable=true means the caller may try the same call again.

Per-tool result shape

Arrays that are not yet parsed from model output are returned as empty arrays (no fabricated structure). observations, inferences, and differences are empty in the current release; only summary carries the model's free-form text.

Multi-image input

vision_compare accepts 2–4 images. The envelope reports them in sources: [SourceMeta, ...] (one entry per input, in input order). source is null for multi-image tools. All other image tools accept a single image and use source; sources is [].

Opt-in flags

• include_raw: bool = False — when true, raw_model_output contains a sanitized subset of the provider response: {model, response_metadata: {model_name, finish_reason, system_fingerprint}, usage_metadata: {input_tokens, output_tokens, total_tokens}}. HTTP headers, request IDs, signed URLs, and raw exception text are dropped before reaching the envelope. Off by default to keep responses small and to avoid leaking auth material.
• include_source_ref: bool = False — when true, source.source_ref is populated with a redacted reference: host/path for URLs (query string stripped, including signed tokens) or basename for local files. data_url and base64 inputs always return null for source_ref. Off by default to avoid leaking paths and signed URLs.

URL Handling

VISION_URL_MODE controls remote-image handling:

• auto passes URLs through for analysis and comparison, but downloads them when inspection, cropping, or OCR requires image bytes.
• passthrough prefers URL passthrough, except for tools that require bytes.
• download always downloads and verifies remote images before model calls.

Downloads are streamed with byte limits, redirects are security checked, and downloaded or encoded inputs are verified as supported images.

Troubleshooting

If Claude Code cannot find the PyPI package:

UV_DEFAULT_INDEX=https://pypi.org/simple uvx --refresh agent-vision-mcp

If the MCP server does not connect:

claude mcp get agent-vision
uvx agent-vision-mcp

If you change the Claude Code configuration:

claude mcp remove --scope user agent-vision

Then add it again with the updated values.

Development

git clone https://github.com/idealizing/agent-vision-mcp.git
cd agent-vision-mcp
python -m venv .venv
.venv/bin/pip install -e ".[dev]"
cp .env.example .env
.venv/bin/python -m unittest discover -s tests -v

License

MIT