adbc-driver-quack

adbc-driver-quack

An Apache Arrow ADBC driver for DuckDB's Quack remote protocol.

Returns Apache Arrow RecordBatches directly from a remote DuckDB server speaking Quack. Supports the standard ADBC bulk-ingest path (Statement.BindStream → APPEND_REQUEST) for fast column-oriented loads.

Distributed as:

• a Go module — github.com/gizmodata/adbc-driver-quack
• a pip install adbc-driver-quack wheel for Python (macOS / Linux / Windows × x64 / arm64)

Status: Alpha — v0.1.0-alpha.1 is the first release. The companion gizmodata/quack-jdbc JDBC driver is the same protocol from the JVM and is at v0.1.0-alpha.1 on Maven Central.

Quickstart

1. Start a Quack server (any DuckDB v1.5.3+)

-- in any DuckDB session — Quack is a core extension as of v1.5.3,
-- so no `core_nightly` repository or `-unsigned` flag is needed.
INSTALL quack;
LOAD quack;
CALL quack_serve('quack:localhost:9494', token=>'my-secret-token');

The server stays running until the DuckDB session exits. Press Ctrl-C in the DuckDB REPL to stop it.

Note: quack_serve accepts shorter forms — 'quack:localhost' uses the default port, and a bare quack_serve() with no first arg uses localhost as the host. We keep the explicit localhost:9494 form throughout this README so the client-side URI maps obviously to what the server is bound to.

If localhost ever gives you a connection refused (rare, but it can happen on a system whose /etc/hosts is set up such that the server binds one address family and the client dials the other), use 127.0.0.1 on both sides.

2. Install the driver

Python:

pip install adbc-driver-quack

Go:

go get github.com/gizmodata/adbc-driver-quack@latest

3. Connect and query

import adbc_driver_quack.dbapi as quack
import pyarrow

with quack.connect(
    uri="quack://localhost:9494",
    db_kwargs={"adbc.quack.token": "my-secret-token"},
) as conn, conn.cursor() as cur:
    cur.execute("SELECT 42 AS answer, 'hello duckdb' AS greeting")
    table: pyarrow.Table = cur.fetch_arrow_table()
    print(table)

The result is a real pyarrow.Table — pass it straight to Polars, Pandas, DuckDB-in-process, ibis, or anything else that consumes Arrow:

import polars as pl
df = pl.from_arrow(table)

Alternative: drive adbc_driver_manager directly

If you prefer the adbc-quickstarts idiom — passing the driver to adbc_driver_manager.dbapi.connect rather than going through our wrapper — point at the bundled shared library via _driver_path():

from adbc_driver_manager import dbapi
import adbc_driver_quack

with dbapi.connect(
    driver=adbc_driver_quack._driver_path(),
    entrypoint="QuackDriverInit",
    db_kwargs={
        "uri": "quack://localhost:9494",
        "adbc.quack.token": "my-secret-token",
    },
) as conn, conn.cursor() as cur:
    cur.execute("SELECT 42 AS answer")
    table = cur.fetch_arrow_table()

Both styles work the same on the wire — pick whichever reads better for your codebase.

Streaming large result sets

Cursor.fetch_record_batch() returns a pyarrow.RecordBatchReader that pulls one server-side DataChunk per read_next_batch() call. Memory stays bounded by the server's chunk size (~2k rows) even when the result is millions of rows:

with conn.cursor() as cur:
    cur.execute("SELECT * FROM lineitem")  # arbitrary size
    reader = cur.fetch_record_batch()
    for batch in reader:
        process(batch)  # one ~2k-row Arrow batch at a time

Bulk ingest (Arrow → DuckDB)

import pyarrow as pa
import adbc_driver_quack.dbapi as quack

table = pa.table({"id": [1, 2, 3], "name": ["alice", "bob", "carol"]})
with quack.connect(
    uri="quack://localhost:9494",
    db_kwargs={"adbc.quack.token": "my-secret-token"},
    autocommit=True,  # ADBC connections are autocommit-OFF by default;
                      # opt in here so the ingest persists on close
) as conn, conn.cursor() as cur:
    # create_append: create "customers" from the Arrow schema if it
    # doesn't exist, then append. One APPEND_REQUEST per RecordBatch.
    cur.adbc_ingest(table_name="customers", data=table, mode="create_append")

Heads-up — autocommit is off by default. Per the Python DB-API, quack.connect() opens connections inside a transaction. Without the autocommit=True above (or an explicit conn.commit()), the CREATE + append run in a transaction that is rolled back when the connection closes — adbc_ingest still returns the row count it sent, but nothing persists. Prefer explicit transactions? Drop autocommit=True and call conn.commit() after adbc_ingest():

with quack.connect(uri="quack://localhost:9494",
                    db_kwargs={"adbc.quack.token": "my-secret-token"}) as conn, conn.cursor() as cur:
    cur.adbc_ingest(table_name="customers", data=table, mode="create_append")
    conn.commit()  # without this, the ingest is rolled back on close

mode accepts the four standard ADBC ingest modes:

Table DDL for the create-family modes is generated from the Arrow schema. Pass db_schema_name=... to target a non-default schema.

Transactions (autocommit off)

import adbc_driver_quack.dbapi as quack

with quack.connect(
    uri="quack://localhost:9494",
    db_kwargs={"adbc.quack.token": "..."},
    autocommit=False,
) as conn, conn.cursor() as cur:
    cur.execute("INSERT INTO orders VALUES (1, 'pending')")
    cur.execute("INSERT INTO order_items VALUES (1, 'widget', 2)")
    conn.commit()  # both inserts persist atomically

Connection URL

quack://host[:port]

Token precedence matches quack-jdbc: an explicit adbc.quack.token (or password) wins, then adbc.quack.token_env, then adbc.quack.token_file. The env/file indirections are accepted only as ADBC options, never as quack://...?tokenEnv=... URL query parameters — a pasted URL must not be able to read a local secret and send it to whatever host the URL names.

The URI is its own kwarg; everything else goes through db_kwargs:

import adbc_driver_quack.dbapi as quack

quack.connect(
    uri="quack://localhost:9494",
    db_kwargs={
        "adbc.quack.token": "my-secret-token",
        "adbc.quack.tls": "false",
    },
)

Extra HTTP headers

Gateways and load balancers in front of a Quack server often need their own auth. adbc.quack.http.header.<Name> options add static headers to every request the driver makes (mirroring the EXTRA_HTTP_HEADERS parameter of DuckDB's own quack secret):

quack.connect(
    uri="quack://gateway.example.com:443",
    db_kwargs={
        "adbc.quack.tls": "true",
        "adbc.quack.token": "my-secret-token",
        "adbc.quack.http.header.X-Proxy-Authorization": "Bearer abc123",
    },
)

Like the token indirections, header options are rejected as URL query parameters — a pasted URL cannot inject headers into your requests. The protocol-owned headers (Content-Type, Accept, Host, Content-Length) are reserved and cannot be overridden.

Connection profiles & driver manifests

ADBC connection profiles (adbc-driver-manager ≥ 1.11) let you keep a connection's driver + options in a reusable TOML file instead of code. Profiles resolve the driver by name, which requires a driver manifest on the search path. Install ours once per environment:

$ python -m adbc_driver_quack install-manifest
Wrote ADBC driver manifest: .../etc/adbc/drivers/quack.toml

(Inside a virtualenv/conda env this targets the environment's auto-searched etc/adbc/drivers/; otherwise the per-user ADBC config directory. --user, --venv, and --dir PATH override; the same is available programmatically as adbc_driver_quack.install_manifest().)

With the manifest in place, the driver manager finds the driver by name — no import of adbc_driver_quack needed:

from adbc_driver_manager import dbapi

# Resolve by URI scheme alone:
conn = dbapi.connect(uri="quack://localhost:9494")

And a profile bundles the whole connection. Drop this in ~/.config/adbc/profiles/quack_prod.toml (Linux; ~/Library/Application Support/ADBC/Profiles/ on macOS, or any directory named in ADBC_PROFILE_PATH):

profile_version = 1
driver = "quack"

[Options]
uri = "quack://prod.example.com:9494"
"adbc.quack.tls" = true
"adbc.quack.token" = "{{ env_var(QUACK_TOKEN) }}"

then connect from any ADBC driver-manager binding:

conn = dbapi.connect(profile="quack_prod")

The {{ env_var(...) }} substitution keeps secrets out of the file; options set explicitly in code still override profile values.

Query Quack from DuckDB or GizmoSQL (adbc_scanner)

The c-shared driver plugs into DuckDB's adbc_scanner community extension (see the GizmoSQL guide) — and therefore into GizmoSQL, which embeds DuckDB. Put the token in a DuckDB secret, ATTACH, and query the remote DuckDB with projection and filter pushdown (pushed-down filters bind parameters, which this driver renders as literals client-side because Quack has no wire-level parameters):

INSTALL adbc_scanner FROM community;
LOAD adbc_scanner;

CREATE SECRET quack_secret (
    TYPE adbc,
    SCOPE 'quack://quackhost:9494',
    driver 'quack',                     -- by name after `python -m adbc_driver_quack install-manifest`,
                                        -- or a path: '/path/to/libadbc_driver_quack.so'
    uri 'quack://quackhost:9494',
    extra_options MAP {'adbc.quack.token': '********'}
);

ATTACH 'quack://quackhost:9494' AS remote (TYPE adbc);
SELECT * FROM remote.main.orders WHERE order_date >= DATE '2024-01-01';

SET VARIABLE q = adbc_connect({'secret': 'quack_secret'});
SELECT * FROM adbc_scan(getvariable('q')::BIGINT, 'SELECT * FROM orders LIMIT 10');

Columnar's adbc extension (connection-profile based, see the GizmoSQL guide) currently refuses the DuckDB and Quack drivers by name; python/tests/test_duckdb_extensions.py tracks both extensions.

Why ADBC and not JDBC?

Both drivers speak the same protocol to the same kind of server. Pick the one that fits your runtime:

Repo layout

adbc-driver-quack/
├── go.mod, go.sum
├── internal/
│   ├── codec/       — BinaryReader/Writer for DuckDB BinarySerializer
│   ├── quacktype/   — Logical / physical / extra type system + codec
│   ├── message/     — DataChunk, DecodedVector, MessageCodec, VectorCodec
│   └── transport/   — QuackURI parser + net/http transport (IPv4/IPv6 fallback)
├── driver/quack/    — pure-Go ADBC Driver/Database/Connection/Statement impl
├── pkg/quack/       — cgo c-shared wrapper (produces libadbc_driver_quack.{so,dylib,dll})
├── python/          — Python wheel sources (adbc_driver_quack)
└── .github/         — CI: go test, python tests, cibuildwheel matrix, PyPI publish

The internal/ layer is a clean-room Go port of the matching Java packages in quack-jdbc.

Credits

• Wire format: DuckDB Quack protocol
• Codec design: ported from gizmodata/quack-jdbc (MIT), which itself was clean-room ported from tobilg/quack-protocol (MIT)
• ADBC framework: Apache Arrow ADBC (Apache-2.0)

License

MIT — see LICENSE for full attribution.