agentic-web3-rag

---

✨ Features

• Semantic search over Web3 docs using fastembed + Qdrant (no GPU required)
• AI-assisted answers with structured output and cited sources
• Consent-first ingestion — only indexes domains with explicit maintainer approval
• Display policy enforcement — respects license terms (link-only / snippet / fulltext) per domain
• FastAPI backend with OpenAPI docs at /docs
• Next.js web UI for interactive search
• CLI entry points — web3rag-api and web3rag-ingest
• Docker Compose stack for one-command local setup

---

📸 Screenshots

---

🏗 Architecture

┌─────────────────────────────────────────────────────────┐
│                      Web3 Sources                        │
│         (ethereum.org, geth.ethereum.org, …)            │
└────────────────────────┬────────────────────────────────┘
                         │  consent gate (consents.yaml)
                         ▼
┌─────────────────────────────────────────────────────────┐
│                   Ingest Pipeline                        │
│  ingest → preprocess → embed (fastembed) → index        │
│                         │                               │
│              data/processed/   data/vectors/            │
└────────────────────────┬────────────────────────────────┘
                         │
                         ▼
                  ┌─────────────┐
                  │   Qdrant    │  vector store
                  │  :6333      │  (Docker)
                  └──────┬──────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────┐
│                   FastAPI  :8080                         │
│   GET  /search   — dense vector search + policy filter  │
│   POST /assist   — retrieval + structured answer        │
│   GET  /health   — liveness check                       │
└────────────────────────┬────────────────────────────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │   Next.js Web UI     │
              │      :3000           │
              └──────────────────────┘

---

📦 Installation

Requirements: Python 3.11+, Docker

pip install agentic-web3-rag

With optional OpenAI-powered answers:

pip install "agentic-web3-rag[openai]"

For local development:

git clone https://github.com/VinitaSilaparasetty/agentic-web3-rag.git
cd agentic-web3-rag
pip install -e ".[dev]"

---

🚀 Quickstart

1. Configure environment

cp .env.example .env
# Edit .env and fill in your keys:
#   OPENAI_API_KEY=...   (optional — only needed for LLM-assisted answers)
#   GITHUB_TOKEN=...     (optional — raises GitHub API rate limits for discovery)

2. Start Qdrant

docker compose up -d qdrant

3. Run the ingest pipeline

# Ingest → chunk → embed → index (all four steps)
web3rag-ingest --sources data/sources.yaml
python -m pipelines.preprocess
python -m pipelines.embed
python -m pipelines.index

Or use Make:

make ingest   # runs ingest step
make dev      # creates venv + installs deps
make up       # starts Docker stack
make api      # starts API server
make test     # runs test suite
make eval     # runs retrieval smoke eval

4. Start the API

web3rag-api
# → http://localhost:8080
# → http://localhost:8080/docs  (OpenAPI)

5. (Optional) Start the Web UI

cd webui
npm install
npm run dev
# → http://localhost:3000

---

🔌 API Reference

GET /health

Liveness check.

curl http://localhost:8080/health
# {"ok": true}

---

GET /search

Dense vector search over indexed docs.

curl "http://localhost:8080/search?q=how+do+I+call+eth_getBalance&k=3&project=geth"

{
  "results": [
    {
      "url": "https://geth.ethereum.org/docs/interacting-with-geth/rpc",
      "title": "Rpc",
      "snippet": "JSON-RPC Server — Interacting with Geth requires sending requests...",
      "score": 0.82,
      "project": "geth",
      "source": "geth.ethereum.org"
    }
  ]
}

---

POST /assist

Retrieval-augmented answer with cited sources.

curl -X POST http://localhost:8080/assist \
  -H "Content-Type: application/json" \
  -d '{"q": "how do I call eth_getBalance in geth", "k": 3}'

Body parameters:

{
  "query": "how do I call eth_getBalance in geth",
  "answer": "### Enable JSON-RPC in geth\n...\n**References**\n- Rpc (geth.ethereum.org) → https://...",
  "results": [...]
}

---

⚙️ Configuration

All settings are read from environment variables (or .env). Copy .env.example to get started.

---

🙋 Web3 Maintainers — Opt In

If you maintain Web3 documentation and want it indexed, click the button below. It takes 2 minutes and you can revoke at any time.

By submitting the form you agree to the Consent to Index terms. Your GitHub account identity and submission timestamp are recorded as the consent record. You can revoke at any time by commenting "REVOKE" on your issue or emailing info@aevoxis.de — all indexed content is removed within 48 hours.

---

📋 Adding Your Own Sources

1. Get consent from the doc maintainer

Ask the maintainer to submit the opt-in form above, or raise an issue on their repo pointing them to it. Save the link to their consent issue as proof.

2. Add the domain to data/consents.yaml

consents:
  - status: approved
    domain: yourdocs.example.com
    project: yourproject
    proof: "https://github.com/yourorg/yourrepo/issues/123"
    scope:
      include_paths:
        - /docs/
      exclude_paths: []

3. Add the URL to data/sources.yaml

sources:
  - kind: website
    id: yourproject-docs
    project: yourproject
    url: https://yourdocs.example.com/docs/
    consent_proof: "https://github.com/yourorg/yourrepo/issues/123"

4. Re-run the pipeline

web3rag-ingest --sources data/sources.yaml
python -m pipelines.preprocess
python -m pipelines.embed
python -m pipelines.index

---

🐳 Docker

A full Docker Compose stack is included:

docker compose up -d          # starts Qdrant (+ Postgres)
docker compose down -v        # stops and removes volumes

To build and run the API in Docker:

docker build -f infra/docker/api/Dockerfile -t web3rag-api .
docker run -p 8080:8080 --env-file .env web3rag-api

---

🧪 Testing

pip install -e ".[dev]"
pytest

Run the retrieval smoke eval (requires a running Qdrant with indexed data):

python -m pipelines.eval_retrieval

---

📜 Consent, Governance & Compliance

This project operates on a deny-by-default consent model:

• Only domains listed as approved in data/consents.yaml are ever ingested
• Each entry requires a proof link (GitHub issue, email, PR) from the maintainer
• Display policy per domain is enforced at query time (link-only / snippet / fulltext)
• Takedown requests are honoured within 48 hours — see LEGAL.md
• Full policy details in GOVERNANCE.md

EU compliance

---

🤝 Contributing

Contributions are welcome. Please open an issue before submitting a large PR.

git clone https://github.com/VinitaSilaparasetty/agentic-web3-rag.git
cd agentic-web3-rag
pip install -e ".[dev]"
pytest

• Bug reports → open an issue
• Feature requests → open an issue

---

💼 Commercial Licensing

This software is licensed under AGPL-3.0. For commercial use, enterprise deployment, or white-label licensing:

📧 info@aevoxis.de

---

📄 License

Copyright © 2025 Vinita Silaparasetty, Aevoxis Solutions. Licensed under the GNU Affero General Public License v3.0.