acmenra-ds

🚀 acmenra-ds

Production-ready, type-safe dataset management, YOLO parsing, caching, and in-memory augmentation for computer vision pipelines

# Install dataset toolkit (pulls acmenra-cv, no heavy ML stack)
pip install acmenra-ds

# OR install with development extras (pytest, ddt, black, mypy)
pip install "acmenra-ds[dev]"

---

📦 Overview

acmenra-ds is the data layer of the Acmenra stack. It turns a YOLO data.yaml plus a folder of images/labels into typed objects (Config, Dataset, Split, Sample, Instance) with strict validation, layout autodetection, RAM caching, visualization, and a composable augmentation pipeline.

It sits on top of acmenra-cv geometry (Box, Polygon, Obb, Instance) and rendering (Drawer, Style). The model itself stays out of this package.

All geometries stay in normalized coordinate space [0.0, 1.0]. Disk I/O is Split / LabelWriter. Augmentation never walks the filesystem.

---

✨ Key Features

🔹 Typed dataset contract

• Config.from_yaml() — Ultralytics-style YAML, autodetection of DIRECT (train/images) vs REVERSE (images/train)
• Leaf names — train_leaf / images_leaf / labels_leaf for non-standard folder names (trn, imgs, lbs)
• Fail-fast setters — relative paths only, no .., unique class names, bool rejected as int
• Mutation warnings — changing a field after construction is logged; prefer a new object

🔹 Dataset as one object

• Dataset owns train / val / test as Split instances — not three separate loaders
• health() — filesystem integrity across splits (missing images, missing labels)
• visualize() — Box / Polygon / OBB via acmenra-cv.Drawer
• Unlabeled frames — Sample.label_path is Optional; empty List[Instance] is valid

🔹 Augmentation that keeps labels in sync

• Single contract: (image, List[Instance]) → (image, List[Instance])
• p is rolled in __call__, not inside apply()
• Geometric steps rebuild Instance via map_instance (DETECT / SEGMENT / OBB independently)
• Photometric steps copy the label list, coordinates unchanged
• No source mutation — safe for DataLoader workers and offline export

🔹 Training-loop friendly

• LabelCache — parsed List[Instance] keyed by stem, optional max_size
• is_strict — raise on the first bad file, or log and skip
• Shared ImageParser / LabelParser / LabelWriter across splits

---

🧩 Module Documentation

🔷 data — Config, Dataset, Split, Sample, Cache

Single source of truth for paths, classes, and splits

Classes

Config — YOLO data.yaml container

• from_yaml(): Loads YAML, resolves root to the YAML parent, autodetects layout.
• get_train_images() / get_train_labels() (and val/test): layout-aware absolute paths.
• names, nc, task, layout: class map, TaskType, DatasetLayout.
• *_leaf: rename split / images / labels folders without rewriting YAML.

Dataset — Train + optional Val/Test

• __init__(config, category, is_cache, is_strict): builds all configured splits.
• train / val / test: Split accessors.
• health(): aggregated missing-file report.
• visualize(sample, style): draws the sample with the dataset task flags.
• from_yaml(): factory that parses YAML then constructs Dataset.

Split — One of TRAIN / VAL / TEST

• Iterable and indexable (for sample in dataset.train, dataset.train[i]).
• get_instances(idx): cache hit or parse-on-read.
• visualize(idx, style): draw by index.
• health(): per-split filesystem check.

Sample — Image + optional label

• stem, image_path, label_path, is_labeled().
• to_dict() / from_dict() for manifests.

LabelCache — RAM cache of List[Instance]

• store(), get(), __getitem__ (raises KeyError on miss), remove(), clear().
• Copy-on-read so callers cannot corrupt cache entries.

DataSplit & DatasetLayout

• DataSplit: TRAIN, VAL, TEST.
• DatasetLayout: DIRECT, REVERSE.

---

🔷 image — ImageParser

Atomic read: ndarray or None

Classes

ImageParser

• parse(path, strict): cv2.imread + min dimension checks.
• Strict mode raises; lenient mode logs and returns None.
• Same contract as LabelParser — one return value, no status tuples.

---

🔷 label — LabelParser / LabelWriter

Inverse pair for YOLO .txt

Classes

LabelParser

• Task-aware: DETECT (cx cy w h), SEGMENT (flat polygon), OBB (cx cy w h angle_rad).
• Validates class id in [0, nc) and coordinates in [0.0, 1.0].
• Builds acmenra_cv.Instance with Box / Polygon / Obb.

LabelWriter

• Inverse of the parser. Writes a List[Instance] to a .txt.
• Use this for offline augmentation export — not the transform classes.

---

🔷 augmentation — Transform, Compose, geometric, photometric

In-memory only. Disk is Split + LabelWriter.

Classes

Transform — ABC

• __call__(image, labels): validate → roll p → apply().
• apply(): subclasses implement the actual op; must not mutate inputs.
• Helpers: _as_image, _as_labels, _as_prob, _as_float, _as_int.

Compose

• Non-empty List[Transform]. Each step rolls its own p.
• Fail-fast: first invalid step raises.
• transforms getter returns a shallow copy.

Geometric (acmenra_ds.augmentation.geometric)

• HorizontalFlip — cv2.flip(..., 1), x' = 1 - x on Box / Polygon / OBB.
• VerticalFlip — cv2.flip(..., 0), y' = 1 - y.
• Resize — pixels to (H, W); normalized labels pass through.

Photometric (acmenra_ds.augmentation.photometric)

• Brightness, ChannelBrightness, Blur, Noise, Pixelate, Gray.
• Normalize — uint8 [0, 255] → float32 [0.0, 1.0]. Last step in a training loop. Do not imshow / JPEG it.

Geometry math (acmenra_ds.augmentation.geometry)

• Pure functions: hflip_box, vflip_obb, map_instance, rebuild_instance.
• No OpenCV. Used by geometric transforms.

---

💡 Quick Start

from enum import IntEnum
import cv2
import numpy as np

from acmenra_cv import Style, Stroke, Fill, Drawer, TaskType
from acmenra_ds import Config
from acmenra_ds.data import Dataset
from acmenra_ds.augmentation import (
    Compose,
    HorizontalFlip,
    VerticalFlip,
    Brightness,
    Blur,
    Noise,
)


class Defect(IntEnum):
    inclusion = 0
    scratch = 1
    crack = 2


# 1. YAML → Config (DIRECT / REVERSE is detected)
config = Config.from_yaml("/path/to/dataset/data.yaml")
print(config)  # DatasetConfig(root=..., nc=..., layout=direct, splits=[...])

# 2. Dataset owns train / val / test
dataset = Dataset(config=config, category=Defect, is_cache=True, is_strict=False)
print(dataset.health())

# 3. Iterate a split
style = Style(
    palette=np.random.randint(0, 255, (80, 3)).tolist(),
    stroke=Stroke(thickness=2, alpha=0.8),
    fill=Fill(alpha=0.2),
    show=True,
)

for sample in dataset.train:
    frame = dataset.visualize(sample=sample, style=style)
    cv2.imshow("sample", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

# 4. Online augmentation (training loop)
pipeline = Compose([
    HorizontalFlip(p=0.5),
    VerticalFlip(p=0.5),
    Brightness(min_factor=0.6, max_factor=1.4, p=0.8),
    Blur(max_ksize=5, p=0.4),
    Noise(max_ratio=0.03, p=0.4),
])

drawer = Drawer(style=style)
sample = dataset.train[0]
image = dataset.train.image_parser.parse(sample.image_path, is_strict=False)
labels = dataset.train.get_instances(0)

aug_image, aug_labels = pipeline(image, labels)

vis = drawer.draw_instances(
    frame=aug_image.copy(),
    instances=aug_labels,
    is_box=dataset.task == TaskType.DETECT,
    is_obb=dataset.task == TaskType.OBB,
    is_polygon=dataset.task == TaskType.SEGMENT,
)
cv2.imshow("augmented", vis)
cv2.waitKey(0)

Offline export (new directory only — never overwrite the source dataset):

from pathlib import Path
from acmenra_ds.label import LabelWriter

out_img = Path("/data/aug/train/images")
out_lbl = Path("/data/aug/train/labels")
out_img.mkdir(parents=True, exist_ok=True)
out_lbl.mkdir(parents=True, exist_ok=True)

writer = dataset.train.label_writer
for idx, sample in enumerate(dataset.train):
    image = dataset.train.image_parser.parse(sample.image_path, is_strict=False)
    labels = dataset.train.get_instances(idx)
    if image is None:
        continue
    aug_image, aug_labels = pipeline(image, labels)
    cv2.imwrite(str(out_img / f"{sample.stem}_aug.jpg"), aug_image)
    writer.write(out_lbl / f"{sample.stem}_aug.txt", aug_labels)

---

📋 Requirements

Installed with pip install acmenra-ds:

acmenra-cv>=0.3.2.0
PyYAML>=6.0
numpy>=1.21.0
opencv-python>=4.5.0

Development extras (pip install "acmenra-ds[dev]"):

pytest>=7.0.0
ddt>=1.6.0
black>=23.0.0
mypy>=1.0.0

Python >=3.9. No PyTorch in the core package — Split is a plain iterable; wrap it in torch.utils.data.Dataset yourself if you need a DataLoader.

---

🧪 Testing

# All tests
pytest tests/

# By module
pytest tests/data/
pytest tests/augmentation/

Coverage includes:

• Type / range validation (wrong type, invalid values, valid values)
• DIRECT and REVERSE layouts, custom leaves
• to_dict / from_dict round-trips
• Mutation logging after construction
• Compose construction and setter contract
• Geometric / photometric property lifecycle

---

🔐 License

This project is licensed under the GNU Affero General Public License v3 or later (AGPL-3.0-or-later). See the LICENSE file for details.

---

🌐 Links

• PyPI: https://pypi.org/project/acmenra-ds/
• Core library: https://pypi.org/project/acmenra-cv/
• YOLO plugin: https://pypi.org/project/acmenra-yolo/
• Documentation: https://docs.acmenra.studio/
• Source: https://github.com/Acmenra/acmenra-ds
• Issues: https://github.com/Acmenra/acmenra-ds/issues

---

acmenra.studio — Building reliable vision systems for the edge. Every sample, every label, every frame.