abstract-transfer

abstract_transfer

One transport for HTTP pulls, stdlib only. Generalised from hugpy's worker model pull (hugpy_storage.provision) so every project moves bytes the same way.

from abstract_transfer import fetch, chunk_sha256s
fetch("https://host/file", "/data/file", sha256="…")             # verified
fetch(url, dest, chunk_sums=sums, chunk_bytes=32 << 20)          # per-chunk verified, parallel
fetch(url, dest, validate=lambda p: open(p,"rb").read(4) == b"GGUF")

abstract-transfer URL DEST [--sha256 X] [--size N] [-H 'Authorization: Bearer …']

Guarantees

• Complete-or-raise. Data lands in DEST.part; only after size, sha256 and validate() pass is it os.replaced to DEST. A file under its final name is whole.
• Resume. Sequential pulls continue from the bytes already in .part (Range: bytes=N-); a server that answers 200 instead of 206 restarts cleanly.
• Parallel segments when the server honours Range and the file is ≥ 64 MiB (or chunk_sums is given): concurrency connections, one chunk each.
• Per-chunk SHA-256 (chunk_sums, from chunk_sha256s() on the server): each chunk is verified as it lands and recorded in DEST.part.state, so a crash resumes from proven content, not a byte offset.
• Single-flight per destination: a thread lock plus an fcntl lock file — two pulls of the same file never interleave.

Used by

• abstract-toolserver fileshare_fetch.
• hugpy_storage (planned): its private _download_* helpers map 1:1 onto fetch.