0g-vector-search

ZeroG VectorSearch

Unified Vector Search on 0G Storage — a decentralized HNSW index built on 0G's sharded KV layer.

Transforms 0G Labs from a Data Repository into a Reasoning Engine by implementing approximate nearest neighbor search directly on 0G's modular storage stack.

Features

• Decentralized HNSW Index — O(log N) vector search with multi-layer graph traversal
• Deterministic Sharding — FNV-1a consistent hashing maps vectors to 0G shards
• Hybrid Search — BM25 keyword + semantic vector fusion (0gmem pattern)
• 0G KV Persistence — Hex-encoded key structure for native KV layer storage
• Security Layer — PoRA verification, data availability guard, alignment verification
• Drop-in SDK — Feels like a native extension of @0glabs/0g-ts-sdk

Quick Start

npm install 0g-vector-search @0glabs/0g-ts-sdk ethers

import { VectorStore } from "0g-vector-search";

// Local development (no 0G network required)
const store = new VectorStore({ dimensions: 768 });
await store.initialize();

// Insert vectors
await store.upsert(
  "doc_001",
  embedding,                          // Float32Array[768]
  { source: "arxiv", topic: "ml" },   // metadata
  "Attention is all you need..."      // text for keyword search
);

// Semantic search
const results = await store.search(queryEmbedding, { topK: 10 });

// Hybrid search (recommended)
const results = await store.search(queryEmbedding, {
  topK: 10,
  hybrid: {
    enabled: true,
    semanticWeight: 0.7,
    query: "transformer architecture",
  },
});

With 0G Network

const store = new VectorStore({
  dimensions: 768,
  zeroG: {
    endpoint: "http://3.101.147.150:6789",
    streamId: "your-stream-id",
    privateKey: "0x...",
    flowContract: "0x...",
    kvNodeCount: 1,
  },
});

Architecture

┌─────────────────────────────────────────────────┐
│                  VectorStore                     │
│  ┌──────────┐  ┌──────────┐  ┌───────────────┐ │
│  │  HNSW    │  │  BM25    │  │   Security    │ │
│  │  Index   │  │  Index   │  │   Layer       │ │
│  └────┬─────┘  └────┬─────┘  └───────┬───────┘ │
│       │              │               │          │
│  ┌────┴──────────────┴───────────────┴───────┐ │
│  │          Shard Router (FNV-1a)             │ │
│  └────────────────┬──────────────────────────┘ │
│                   │                             │
│  ┌────────────────┴──────────────────────────┐ │
│  │     0G KV Layer (via Batcher / KvClient)  │ │
│  └───────────────────────────────────────────┘ │
└─────────────────────────────────────────────────┘

KV Key Structure

All data is stored in 0G's KV layer with hex-encoded keys:

Vector Upsert Flow

1. ShardRouter.getShardId(id) → FNV-1a hash determines target shard
2. HNSWIndex.insert(id, vector) → builds graph links across layers
3. BM25Index.addDocument(id, content) → indexes text for keyword search
4. PoRAVerifier.generateCommitment() → creates proof for 0G validators
5. GraphPersistence.save*() → writes to 0G KV via Batcher

Vector Search Flow

1. DataAvailabilityGuard checks shard health
2. HybridSearchEngine.search() → HNSW + BM25 score fusion
3. AlignmentVerifier.verifyResults() → checks for injection attacks
4. Results returned sorted by fused score

Configuration

Development

npm install        # Install dependencies
npm test           # Run test suite (21 tests)
npm run build      # TypeScript compilation

Security

• PoRA Compatibility: Vector updates generate proof commitments for 0G validator verification
• Data Availability: Shard health monitoring with fallback routing for offline shards
• Alignment Verification: Detects suspicious result clustering indicating injection attacks

License

MIT