aice-crawler

aice-crawler

Python SDK for the Crawler Service — multi-tenant web crawler with a page budget and an approve-then-publish flow into the Knowledge Service.

from aice_crawler import CrawlerClient

client = CrawlerClient(base_url="https://crawler.example.com", api_key="sk-…")

job = client.jobs.create(
    start_urls=["https://docs.example.com"],
    page_budget=500,
)

client.jobs.start(job["id"])

for page in client.jobs.list_pages(job["id"]):
    ...

client.jobs.approve(job["id"])
client.jobs.publish(job["id"])

Endpoints covered (see crawler_service/app/api/routes.py):

• POST /jobs, GET /jobs, GET /jobs/{id}
• POST /jobs/{id}/start, GET /jobs/{id}/pages, GET /pages/{id}
• POST /jobs/{id}/approve, POST /jobs/{id}/publish
• PUT /jobs/{id}/schedule, GET /jobs/{id}/runs, POST /jobs/{id}/reclean
• GET /policy, PUT /policy