PkgFoundry
Catalog PyPI
Back to packageRELEASE DETAILS

0.0.2

PyPI

ai-db

Analyze your unstructured data

Published9 Nov 2023
Direct dependencies50
Transitive dependenciesNot calculated

README

Version 0.0.2 documentation

AIDB

Analyze unstructured data blazingly fast with machine learning. Connect your own ML models to your own data sources and query away!

Quick Start

In order to start using AIDB, all you need to do is install the requirements, specify a configuration, and query! Setting up on the environment is as simple as

git clone https://github.com/ddkang/aidb.git
cd aidb
pip install -r requirements.txt

# Optional if you'd like to run the examples below
gdown https://drive.google.com/uc?id=1SyHRaJNvVa7V08mw-4_Vqj7tCynRRA3x
unzip data.zip -d tests/

Text Example (in CSV)

We've set up an example of analyzing product reviews with HuggingFace. Set your HuggingFace API key. After this, all you need to do is run

python launch.py --config=config.sentiment --setup-blob-table --setup-output-table

As an example query, you can run

SELECT AVG(score)
FROM sentiment
WHERE label = '5 stars'
ERROR_TARGET 10%
CONFIDENCE 95%;

You can see the mappings here. We use the HuggingFace API to generate sentiments from the reviews.

Image Example (local directory)

We've also set up another example of analyzing whether or not user-generated content is adult content for filtering. In order to run this example, all you need to do is run

python launch.py --config=config.nsfw_detect --setup-blob-table --setup-output-table

As an example query, you can run

SELECT *
FROM nsfw
WHERE racy LIKE 'POSSIBLE';

You can see the mappings here. We use the Google Vision API to generate the safety labels.

Key Features

AIDB focuses on keeping cost down and interoperability high.

We reduce costs with our optimizations:

  • First-class support for approximate queries, reducing the cost of aggregations by up to 350x.
  • Caching, which speeds up multiple queries over the same data.

We keep interoperability high by allowing you to bring your own data source, ML models, and vector databases!

Approximate Querying

One key feature of AIDB is first-class support for approximate queries. Currently, we support approximate AVG, COUNT, and SUM. We don't currently support GROUP BY or JOIN for approximate aggregations, but it's on our roadmap. Please reach out if you'd like us to support your queries!

In order to execute an approximate aggregation query, simply append ERROR_TARGET <error percent>% CONFIDENCE <confidence>% to your normal aggregation. As a full example, you can compute an approximate count by doing:

SELECT COUNT(xmin)
FROM objects
ERROR_TARGET 5%
CONFIDENCE 95%;

The ERROR_TARGET specifies the percent error compared to running the query exactly. For example, if the true answer is 100, you will get answers between 95 and 105 (95% of the time).

Useful Links

Contribute

We have many improvements we'd like to implement. Please help us! For the time being, please email us, if you'd like to help contribute.

Contact Us

Need help in setting up AIDB for your specific dataset or want a new feature? Please fill this form.

Downloads & popularity

Awaiting observations

No metric observations yet

Download counts and popularity will appear here as the collector receives them.

Declared dependencies 50

Native requirements
50 direct targetsNot calculated transitive count
Package / targetRequirementRelationship
aiomysql
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
aiomysql ==0.2.0runtime
aiosqlite
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
aiosqlite ==0.19.0runtime
annotated-types
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
annotated-types ==0.5.0runtime
anyio
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
anyio ==3.7.1runtime
asyncpg
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
asyncpg ==0.28.0runtime
boto3
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
boto3 ==1.28.63runtime
certifi
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
certifi ==2023.7.22runtime
charset-normalizer
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
charset-normalizer ==3.2.0runtime
chromadb
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
chromadb ==0.4.11runtime
click
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
click ==8.1.7runtime
exceptiongroup
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
exceptiongroup ==1.1.3runtime
faiss-cpu
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
faiss-cpu ==1.7.4runtime
fastapi
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
fastapi ==0.99.1runtime
flatten-json
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
flatten-json ==0.1.13runtime
gdown
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
gdown ==4.7.1runtime
greenlet
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
greenlet ==2.0.2runtime
h11
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
h11 ==0.14.0runtime
idna
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
idna ==3.4runtime
iniconfig
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
iniconfig ==2.0.0runtime
nest-asyncio
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
nest-asyncio ==1.5.7runtime
networkx
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
networkx ==3.1runtime
numba
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
numba ==0.58.0runtime
numpy
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
numpy ==1.24.4runtime
packaging
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
packaging ==23.1runtime
pandas
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
pandas ==2.0.3runtime
Pillow
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
Pillow ==10.1.0runtime
pluggy
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
pluggy ==1.3.0runtime
pydantic
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
pydantic ==1.10.12runtime
PyMySQL
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
PyMySQL ==1.1.0runtime
pytest
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
pytest ==7.4.2runtime
python-dateutil
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
python-dateutil ==2.8.2runtime
pytz
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
pytz ==2023.3.post1runtime
PyYAML
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
PyYAML ==6.0.1runtime
requests
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
requests ==2.31.0runtime
scipy
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
scipy ==1.10.1runtime
six
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
six ==1.16.0runtime
sniffio
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
sniffio ==1.3.0runtime
SQLAlchemy
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
SQLAlchemy ==1.4.39runtime
SQLAlchemy-Utils
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
SQLAlchemy-Utils ==0.41.1runtime
sqlglot-aidb
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
sqlglot-aidb ==0.0.5runtime
starlette
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
starlette ==0.27.0runtime
statsmodels
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
statsmodels ==0.14.0runtime
sympy
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
sympy ==1.11.1runtime
tomli
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
tomli ==2.0.1runtime
tqdm
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
tqdm ==4.66.1runtime
typing-extensions
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
typing-extensions ==4.7.1runtime
tzdata
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
tzdata ==2023.3runtime
urllib3
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
urllib3 ==1.26.17runtime
uvicorn
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
uvicorn ==0.23.2runtime
weaviate-client
Environment
{
  "extras": "",
  "marker": null,
  "requires_python": ">=3.9",
  "evaluation": "NOT_EVALUATED"
}
weaviate-client ==3.24.1runtime

Distribution files

2 collected
ai_db-0.0.2-py3-none-any.whl

bdist_wheel · 64.92 KB · Python py3

Source file
ai-db-0.0.2.tar.gz

sdist · 47.3 KB · Python source

Source file

Metadata history

Meaningful changes
Current collected state Current

Observed since 21 Oct, 10:25 UTC

No superseded metadata states collected. Unchanged observations do not create new revisions.

Record identity

package_version:049064bd452b05c3319ae4374bbd6cab08fbdda8a2cb1619c19cd8e000a34dbe

Other collected versions

View all collected versions →