aethisdb

AethisDB

A lightweight, pure-Python document database with a custom binary file format (.aethisdb). Zero external dependencies — uses only the Python standard library.

Features

• Custom binary file format with a 96-byte header and append-friendly data section — no JSON on disk
• Unreadable files: records, collection names and the index are ciphered, so a text or hex editor shows noise
• Optional password protection (password=) with PBKDF2-HMAC-SHA256 key derivation
• Hash-based indexing for fast exact-match queries on indexed fields
• Full CRUD operations: insert, find, update, delete
• Rich query operators: $eq, $ne, $gt, $lt, $gte, $lte, $in, $nin, $exists, $and, $or
• Auto-commit or manual commit modes
• File compaction to reclaim space from deleted records
• Automatic compaction based on tombstone ratio — ideal for small/embedded storage
• zlib-compressed index block for minimal on-disk footprint
• Compact binary encoding — varint integers and a per-block string table beat compact JSON
• Space statistics reporting via db.stats()
• Context manager support (with Database(...) as db:)
• Persistent across sessions — collections and indexes survive reopens

Installation / Setup

AethisDB has no third-party dependencies. Simply place the aethisdb/ package directory on your Python path, or install it in development mode:

pip install aethisdb

Then import:

from aethisdb import Database, Record

Quick Start

Initializing a Database

from aethisdb import Database

# Creates a new .aethisdb file (or opens an existing one)
db = Database("mydata.aethisdb")

# Create a collection
users = db.create_collection("users")

# List collections
print(db.list_collections())  # ['users']

Inserting Records

# Insert a single record — returns the auto-assigned record ID
uid = users.insert({"name": "Alice", "email": "alice@example.com", "age": 30})

# Insert multiple records at once
ids = users.insert_many([
    {"name": "Bob",   "email": "bob@example.com",   "age": 25},
    {"name": "Carol", "email": "carol@example.com", "age": 35},
])

# Get a record by ID
alice = users.get(0)
print(alice["name"])  # Alice

Querying Records

# Find by exact match
results = users.find({"name": "Alice"})
print(results[0]["email"])  # alice@example.com

# Find one (returns None if no match)
bob = users.find_one({"email": "bob@example.com"})

# Range query
adults = users.find({"age": {"$gte": 30}})

# Multiple conditions (implicit AND)
results = users.find({"age": {"$gte": 25}, "age": {"$lte": 35}})

# $in operator
results = users.find({"name": {"$in": ["Alice", "Bob"]}})

# $or operator
results = users.find({"$or": [{"age": 25}, {"age": 35}]})

# Get all records
all_users = users.all()

# Count records
print(users.count())  # 3

Updating Records

# Update matching records
count = users.update({"name": "Alice"}, {"age": 31})
print(count)  # 1

# Update by record ID
users.update(record_id=1, update_data={"age": 26})

Note: Updates in AethisDB use a copy-on-write strategy — the old record is marked as deleted and a new record is appended with the updated data. Record IDs are preserved.

Deleting Records

# Delete by query
count = users.delete({"name": "Bob"})

# Delete by record ID
count = users.delete(record_id=2)

# Delete all records (use with care!)
users.delete()

# Count after deletion
print(users.count())  # 2

Indexing

Hash indexes provide O(1) exact-match lookups:

# Create an index on a field
users.create_index("email")

# Now queries on the indexed field use the index automatically
bob = users.find_one({"email": "bob@example.com"})  # fast

# List indexes
print(users.indexes)  # ['email']

# Drop an index
users.drop_index("email")

# Rebuild an index after bulk operations
users.create_index("email")
users.rebuild_index("email")

Compaction

Deleted records leave tombstones in the file. Use compact() to reclaim space:

# After many deletions
file_size_before = os.path.getsize("mydata.aethisdb")

db.compact()

file_size_after = os.path.getsize("mydata.aethisdb")
# File is smaller — tombstones removed

Transaction / Auto-commit Control

# Manual commit mode — changes are not persisted until commit()
db = Database("mydata.aethisdb", auto_commit=False)

users = db.get_collection("users")
users.insert({"name": "Alice"})
users.insert({"name": "Bob"})

# Verify data is not yet persisted
db.close()  # Without commit, changes are lost

# Correct usage:
db = Database("mydata.aethisdb", auto_commit=False)
users = db.get_collection("users")
users.insert({"name": "Alice"})
users.insert({"name": "Bob"})
db.commit()  # persist to disk
db.close()

Context Manager

with Database("mydata.aethisdb") as db:
    users = db.get_collection("users")
    users.insert({"name": "Alice"})
    alice = users.find_one({"name": "Alice"})
# db is automatically closed

Space Efficiency (Small Storage)

AethisDB includes several features to minimize disk usage, making it ideal for embedded or small-storage applications:

Compact Binary Encoding (no JSON)

Record documents and the index block are written with AethisDB's own binary codec:

• integers as zig-zag varints instead of ASCII digits
• one string table per block, so repeated keys are stored once
• varint element counts instead of braces, brackets, quotes and commas
• the index block packs record IDs and byte offsets as varints, with no field names

Measured against the previous JSON layout (2,000 records with one email index): 370,092 B → 337,292 B, about 9% smaller, and unreadable.

Unreadable by Default

Every record, collection name and the index block is sealed with a SHA-256 keystream before it is written, so opening the file in Notepad, VS Code or a hex editor shows binary noise. See Passwords and Encryption.

Compressed Index Block

The index block at the end of the file is automatically zlib-compressed when compression reduces its size. Compression is transparent — no configuration required.

Auto-Compaction

When records are deleted, they are soft-deleted (marked as tombstones). Over time, tombstones waste space. AethisDB can automatically compact the file on close when the tombstone ratio exceeds a configurable threshold:

from aethisdb import Database

# auto_compact=True: compact on close if deleted ratio > compact_ratio
# compact_ratio=0.3 (default): compact when 30%+ of records are deleted
db = Database("small.aethisdb", auto_compact=True, compact_ratio=0.25)

users = db.create_collection("users")
for i in range(10):
    users.insert({"name": f"user_{i}"})
for i in range(5):
    users.delete({"name": f"user_{i}"})
# 50% deleted > 25% threshold → file is compacted automatically on close
db.close()

Manual Commit Mode

For workloads with many writes, use manual commit mode to reduce index write frequency:

# Only write the index to disk when commit() or close() is called
db = Database("small.aethisdb", auto_commit=False)

users = db.create_collection("users")
for i in range(1000):
    users.insert({"name": f"user_{i}"})

db.commit()  # one index write instead of 1000
db.close()

Space Statistics

db = Database("small.aethisdb")
stats = db.stats()
print(stats)
# {
#   'file_size': 4122,
#   'header_size': 96,
#   'data_size': 3218,
#   'index_size': 808,
#   'record_count': 50,
#   'deleted_count': 0,
#   'deleted_bytes_estimate': 0,
#   'collections': 1,
#   'auto_compact': False,
#   'auto_commit': True,
#   'version': 3,
#   'cipher': 'obfuscated'
# }
db.close()

Manual Compaction

Force a compaction at any time:

db = Database("small.aethisdb")
users = db.get_collection("users")
users.delete({"age": {"$lt": 18}})
db.compact()  # immediately rewrite file, removing tombstones
db.close()

Full Example

from aethisdb import Database

with Database("company.aethisdb") as db:
    employees = db.create_collection("employees")
    employees.create_index("email")

    # Seed data
    employees.insert_many([
        {"name": "Alice",   "email": "alice@corp.com", "department": "Engineering", "salary": 90000},
        {"name": "Bob",     "email": "bob@corp.com",   "department": "Sales",       "salary": 65000},
        {"name": "Charlie", "email": "carol@corp.com", "department": "Engineering", "salary": 85000},
    ])

    # Query with index
    alice = employees.find_one({"email": "alice@corp.com"})

    # Range query
    high_earners = employees.find({"salary": {"$gte": 80000}})

    # Update
    employees.update({"email": "bob@corp.com"}, {"salary": 70000})

    # Delete
    employees.delete({"department": "Sales"})

    # Verify
    remaining = sorted(e["name"] for e in employees.all())
    print(remaining)  # ['Alice', 'Charlie']

---

File Format: .aethisdb

Overview

The .aethisdb file format is a custom binary layout designed for append-friendly record storage with a persistent binary index at the end of the file. Nothing is stored as JSON, and every block except the fixed header fields is ciphered — a text or hex editor shows noise. The format consists of three sections:

┌──────────────────┬───────────────────┬─────────────────────┐
│  Header (96B)    │  Data Section     │  Index Block        │
│  (fixed size)    │ (append-only)     │   (end of file)     │
├──────────────────┼───────────────────┼─────────────────────┤
│ Magic number     │ Collection        │ version, collection │
│ Version          │   markers         │   count, then per   │
│ Cipher mode      │ Data records      │   collection: name, │
│ KDF iterations   │                   │   counters, record  │
│ Index offset     │                   │   id → offset pairs,│
│ Index length     │                   │   and hash indexes  │
│ Data start       │                   │   (all varints)     │
│ Collection count │                   │                     │
│ Salt, key check  │                   │                     │
└──────────────────┴───────────────────┴─────────────────────┘

1. Header (exactly 96 bytes)

The header is a fixed-size binary structure at the beginning of the file:

All integers are little-endian (struct format character <).

2. Data Section

The data section starts at byte offset 96 (right after the header). It contains a sequence of block entries. Two block types are supported:

Collection Marker Block

Written when a collection is created. The name is ciphered, so collection names are not visible in the file.

Record Entry Block

Written for each record insert. Records are appended sequentially. The 14-byte entry header stays plaintext so that a soft-delete can flip the status byte in place; the document itself is ciphered.

3. Binary encoding

Documents use AethisDB's own codec (aethisdb/codec.py), never JSON:

Each block begins with a string table (varint count, then varint-prefixed UTF-8 strings), so keys and repeated values are stored once and referenced by index.

4. Index Block (end of file)

The index block is a hand-packed binary structure appended after the last data byte. The header's index_offset and index_length fields locate it. On every commit, the entire index is rewritten.

varint  format version
varint  collection count
  per collection:
    string-ref  name
    varint      next_id, record_count, deleted_count, marker_offset
    varint      position count
      varint    record ID
      varint    byte offset
    varint      index count
      per index: field name, then value key → list of record IDs

Key fields (in memory):

Before sealing, the index gets a one-byte flag: 0x00 raw binary, 0x01 zlib-compressed.

How Writes Work

AethisDB uses an append-friendly strategy:

1. Insert: A record is encoded, sealed and appended to the end of the data section. Its position is recorded in record_positions.
2. Update: The old record's status byte is set to 0 (deleted) in-place. A new record with the updated data is appended at the end. The record_positions map is updated to point to the new record. The old record remains as a tombstone.
3. Delete: The record's status byte is set to 0 (deleted) in-place. It is removed from record_positions and from all hash indexes.
4. Commit: The entire index block is packed, optionally compressed and sealed, then written after the last data byte. The 96-byte header is updated with the new index_offset and index_length.

How Reads Work

1. The header is read (first 96 bytes) to locate the index block and derive the key.
2. The index block is unsealed and unpacked to reconstruct collection metadata and hash indexes in memory.
3. For single-record lookups (get(id)), the byte offset is retrieved from record_positions, the file is seeked to that position, and the record is unsealed and decoded.
4. For indexed exact-match queries, the hash index directly maps field values to record IDs, avoiding any file scanning.
5. For non-indexed or range queries, all active records are scanned and filtered using the query engine.

How Compaction Works

When compact() is called:

1. A temporary file is created.
2. All active records (status = 1) are read from the current file and rewritten sequentially to the temp file, re-sealing each block at its new position.
3. Deleted records (tombstones) are skipped — they are not copied.
4. A fresh, compact index is written at the end of the temp file.
5. The temp file replaces the original file atomically (os.replace).
6. The result is a file with no wasted space from tombstones or orphaned index blocks.

Legacy files

Databases written by version 1.0.x (64-byte header, JSON payloads) are detected and migrated automatically the first time they are opened: records and indexes are re-encoded to the binary layout and rewritten using the current cipher mode.

Passwords and Encryption

.aethisdb files are unreadable without this library. Two modes, selected automatically:

from aethisdb import Database

# Obfuscated (default) — no password needed, raw file is still unreadable
db = Database("data.aethisdb")

# Encrypted — password required every time the file is opened
db = Database("secret.aethisdb", password="hunter2")
print(db.is_encrypted)   # True
print(db.cipher)         # "encrypted"

Under the hood, a per-file random salt plus PBKDF2-HMAC-SHA256 derives a key, and every block is sealed with a SHA-256 counter-mode keystream bound to its file position. An 8-byte HMAC check in the header rejects a wrong password before any record is read.

Add or change a password at any time — the file is rewritten with a fresh key:

db = Database("data.aethisdb")
db.change_password("hunter2")
db.close()

cipher="none" disables the cipher; cipher="obfuscated" and cipher="encrypted" force a mode for new files.

To look inside a file without opening a session:

from aethisdb import file_info, peek

file_info("secret.aethisdb")               # header only, no password needed
peek("secret.aethisdb", password="pw")     # dict of collections and records

Query Operators

Running Tests

cd /path/to/aethisdb
python -m unittest discover -s tests -v

Architecture

aethisdb/
├── __init__.py      Package exports (Database, Collection, Record, peek, ...)
├── database.py      Top-level Database class — file lifecycle, collections, passwords
├── collection.py    Collection class — CRUD operations & query optimization
├── storage.py       StorageEngine — binary file format I/O (header, records, index)
├── codec.py         Compact binary encoder/decoder (replaces JSON on disk)
├── crypto.py        Key derivation + block cipher (keeps files unreadable)
├── legacy.py        One-time reader for v2 (JSON) files, used by auto-migration
├── inspect_file.py  file_info() / peek() read-only helpers
├── index.py         Hash-based Index & IndexManager for fast lookups
├── record.py        Record class — dict-like document with an ID
├── query.py         Query engine — condition parsing, evaluation, update application
├── utils.py         Constants and struct helpers for the binary format
└── exceptions.py    Custom exception hierarchy