aferiq-eval

aferiq-eval

Quality observability for RAG and agents — vertical for the Brazilian market. PT-BR judge prompts hand-tuned for BR patterns (Lei nº, CNPJ/CPF, Receita Federal, INSS, BACEN, ANPD), claim-by-claim hallucination diagnosis, trajectory + tool-use evaluation for LangGraph / AgentExecutor / OpenAI Assistants.

pip install aferiq-eval, then import aferiq. The legacy namespace rageval is kept as a parallel alias — both import aferiq and import rageval work and expose the same API — but new code should lead with import aferiq.

Install

pip install aferiq-eval

# Optional integrations:
pip install 'aferiq-eval[langchain]'      # LangChain (RAG + AgentExecutor)
pip install 'aferiq-eval[langgraph]'      # LangGraph
pip install 'aferiq-eval[openai]'         # OpenAI SDK + Assistants
pip install 'aferiq-eval[anthropic]'      # Anthropic SDK

RAG eval (single-turn)

from rageval import evaluate

result = evaluate(
    queries=["Qual o prazo do CDC pra arrependimento?"],
    contexts=[["Art. 49 do CDC: o consumidor pode desistir em 7 dias."]],
    answers=["O CDC garante 7 dias contados do recebimento."],
    metrics=["faithfulness", "hallucination"],
    language="pt",  # default
)

print(result)            # rich-rendered table
result.to_dict()         # JSON-friendly
result.aggregate         # {"faithfulness": 0.95, "hallucination": 0.92}

Ship to the cloud — one line

Set AFERIQ_DSN in your env (copy it from the dashboard → API keys), then:

import aferiq
aferiq.start()  # reads AFERIQ_DSN, auto-patches openai/anthropic

# From here, every openai/anthropic call is captured automatically —
# no decorator, no callback. Just use the SDKs as you already do.

Set the env var before import openai runs, or call aferiq.start() first — the auto-patch instruments whatever is imported after it.

Capturing a custom RAG function with @trace

For your own RAG/agent function (not a raw SDK call), decorate it:

import aferiq

@aferiq.trace
def my_rag(query: str, context: list[str]) -> str:
    chunks = retriever.search(query)
    return llm.generate(query, chunks)

my_rag("Qual o aviso prévio mínimo na CLT?", ["..."])

With AFERIQ_DSN set in the env, the bare @aferiq.trace ships traces on its own — the first decorated call wires up cloud shipping from the env. No env set and no handler registered → the decorator is a zero-overhead pass-through. Want explicit control instead? Register your own handler:

from aferiq import register_trace_handler

captured: list = []
register_trace_handler(captured.append)  # collect locally, e.g. for batch eval

Agent eval (multi-step)

For LangChain AgentExecutor, LangGraph state machines, or OpenAI Assistants — no decorator needed. Run your application under aferiq-trace-run, attach the agent callback, and full trajectories (LLM calls + tool calls + tokens + latency) get captured automatically.

aferiq-trace-run python my_agent.py

from rageval.integrations.agent_callback import AferiqAgentCallbackHandler

handler = AferiqAgentCallbackHandler(goal="Qual o prazo do CDC?")
agent_executor.invoke(
    {"input": "Qual o prazo do CDC?"},
    config={"callbacks": [handler]},
)
# handler.last_trace is an AgentTrace; also broadcast to registered handlers

The dashboard at https://www.aferiq.com.br auto-detects trace shape: agent runs render as a step-by-step tree; RAG traces render claim-by-claim with PT-BR categorisation (lei_inventada, cnpj_fabricado, etc.).

CLI

export OPENAI_API_KEY=sk-...

# RAG eval
aferiq-eval evaluate --dataset rag_traces.jsonl --metrics faithfulness,hallucination

# Agent eval
aferiq-eval evaluate-agent --dataset agent_traces.jsonl \
    --metrics trajectory_quality,goal_completion,cost_efficiency

Examples

Runnable demos under examples/:

• langchain_agent_executor.py — calculator + search agent
• langgraph_agent.py — state-machine agent
• openai_assistants.py — Assistants API streaming

Supported judge models

Anything litellm supports — OpenAI, Anthropic, Google, Maritaca Sabiá, Ollama. Default: gpt-4o-mini (cheapest with acceptable quality on PT-BR judging).

Caching

By default, identical (prompt, model, temperature) tuples are cached on disk under ~/.cache/rageval. Repeated evals don't re-call the LLM.

from rageval.cache import DiskCache, MemoryCache, NullCache

DiskCache()                          # default location
DiskCache("/custom/path")            # custom location
MemoryCache()                        # in-process, tests
NullCache()                          # disable caching

Dev

git clone https://github.com/leo94pena/rag_eval
cd rag_eval/packages/lib
pip install -e ".[dev]"

pytest --cov=rageval         # tests
ruff check src tests         # lint
mypy --strict src            # type-check

Documentation

• Full docs: https://docs.aferiq.com.br (coming soon)
• Issues + roadmap: https://github.com/ileoh/aferiq/issues
• LGPD + data handling: auth in São Paulo (Supabase) · traces in ClickHouse Cloud us-east-1 under SCC + signed DPA per LGPD Art. 33 · redact_pii=True flag strips CPF/CNPJ/email/phone before transmission · TLS in transit · AES-256-GCM at rest for sensitive fields.

License

Proprietary — see LICENSE.