A few weeks ago, we published a preview of DuckDB v2.0.

Today, we branched off v2.0-cyanoptera from the main development branch in the duckdb/duckdb repository. This marks a feature freeze – in the coming weeks, development work will focus on testing, bugfixing and iterating toward the best DuckDB release to date: v2.0 Cyanoptera, projected for the second half of October.

The DuckDB ecosystem has duckdb/duckdb at its center: it's a dependency for most other work, so stabilizing it unlocks work across the other repositories. There are two main directions: clients and extensions.

Clients

Several DuckDB clients are already available as alpha versions.

CLI

To install the command-line client on Linux or macOS, run:

curl https://install.duckdb.org | DUCKDB_VERSION=alpha bash

~/.duckdb/cli/latest/duckdb -c "SELECT version() AS version;"

┌───────────────────┐
│      version      │
│      varchar      │
├───────────────────┤
│ v2.0.0-alpha39998 │
└───────────────────┘

On Windows, download the tarball, extract it and run duckdb.exe:

• DuckDB v2.0-alpha for Windows AMD64 (x86_64)
• DuckDB v2.0-alpha for Windows AArch64 (arm64)

Warning Extensions are not yet available for the Windows client. Stay tuned!

Python

To install the Python client, run:

pip install duckdb --pre --upgrade

python3 -c "import duckdb; print(duckdb.version())"

This prints the versions of both the Python client (currently 1.6-dev) and the underlying DuckDB library (2.0.0-alpha):

1.6.0.dev379 (with duckdb 2.0.0-alpha39998)

More Clients

We'll gradually add other clients to this list. Each new client that moves to 2.0 improves coverage for the DuckDB project, given that different clients will often have very different test environments and setups.

Extensions

In DuckDB v2.0, the quack extension will move from version 0.x to 1.0. The new version will unlock higher throughput for queries (both server to client and client to server) and better compatibility.

Other core extensions, such as httpfs, ducklake, iceberg or spatial, are also available in the alpha stage – please report any issues that you might encounter.

Community extensions can already be tested by providing a ref_next SHA, which builds them against the v2.0-cyanoptera branch before the release date. If no ref_next is provided, the ref SHA will be used once v2.0.0 is finalized.

Can I Help?

Yes, you can! DuckDB alpha clients are explicitly not production-ready, but you can already install them and throw some SQL queries at them to see whether your existing workloads work. Most are expected to be fine, some are expected to complete visibly faster, and some might result in an error. For the next few weeks, we are particularly interested in the latter group: are there environments, situations or queries where the DuckDB v2.0 alpha releases fall short?

If you find an issue, please open a bug report with a reproducible example, either in duckdb/duckdb or in the repository of the relevant client or extension.

DuckDB's main strength is its community: given enough eyeballs, all bugs are shallow, and your bug reports translate to stability gains for every user.

DuckDB, DuckLake, Quack, and all the other extensions will remain free and open source software under the MIT license under the stewardship of the non-profit DuckDB Foundation. DuckDB and related projects will continue to be developed at a very high pace. The Foundation will set up a stakeholder advisory board which can influence the direction of the projects. We are lifting the limitations of community support. Please consult the blog post on ducklabs.com and download the media kit for more information.

PS: Next week, we’ll host the DuckDB meetup in Boston. If you’re nearby, consider dropping by!

In the upcoming DuckDB v2.0, we assign those scopes explicitly. The query plan owns the physical operator tree, precomputed pipeline schedules and a pool of reusable pipeline executors. Each recursive invocation borrows executors from that pool and owns the accumulated recursive state and retained materializations whose lifetime spans the complete fixed-point computation, while each iteration owns only frontier-dependent state. This separation enables retained hash builds, per-iteration selection between inline and scheduled execution and direct probes into keyed state. The physical execution changes are accompanied by one semantic change: for USING KEY, UNION now makes new keys and keys whose finalized payload changed visible to the next iteration.

To illustrate the performance impact, we compare the original and new engines on a small reachability query. The table contains one million edges over 100,000 nodes. Ten identical edges leave each source node, and following them from node 0 visits 20,000 nodes:

CREATE TABLE edges AS
    SELECT (range % 100_000)::INTEGER AS src,
           ((range * 13 + 7) % 100_000)::INTEGER AS dst
    FROM range(1_000_000);

WITH RECURSIVE reachable(node) AS (
    SELECT 0
    UNION
    SELECT dst
    FROM edges, reachable
    WHERE src = node
)
SELECT count(*)
FROM reachable;

With the optimizations described below, median runtime falls from 4.051 seconds with DuckDB v1.5.5 to 0.095 seconds with the DuckDB v2.0 preview, a 42.6× speedup without changing the SQL.

Reusable Execution State Outlives the Frontier

Recursive CTE execution sits inside the query-plan lifetime. The physical operator tree belongs to that plan, and immutable schedule projections are built during physical pipeline construction. Within the plan, an invocation spans one complete fixed-point computation, while an epoch spans one evaluation of the recursive term. The original query plan already retained the recursive physical operator, but its runtime treated each epoch as a fresh execution: it recreated events and executors, reconstructed operator state and repeated scheduling work.

During pipeline construction, DuckDB derives immutable views of the pipeline dependency schedule, including the form that omits invocation-retained build pipelines after their first execution. We refer to these views as schedule projections. The runtime can therefore select the required dependency schedule without deriving it again for every epoch.

The Frontier Bounds Recursive Input

Every recursive epoch replaces the working table, often called the frontier, while the query remains fixed. The anchor term produces the first frontier. The recursive term reads it and produces candidates for the next one. With regular UNION, DuckDB rejects rows already seen; the survivors form the next frontier and also contribute to the logical union table. Evaluation stops when no rows survive.

The union table belongs to the semantics. DuckDB need not materialize it: result chunks can stream downstream while the engine retains the current frontier and, for regular UNION, the hash state required for duplicate elimination. It accumulates a complete copy only when the recursive term accesses recurring.cte_name.

For a monotone, linear recursive term, this frontier discipline is the operational core of semi-naive evaluation: an epoch reads only the preceding delta instead of reevaluating the recursive term from every row seen so far. The original operator already preserved this semantic invariant; the remaining mismatch concerned the lifetime of the physical machinery evaluating that frontier.

Epoch-Invariant State Belongs to the Invocation

The frontier scan, shown as REC_CTE_SCAN in an EXPLAIN plan, must read new input after an epoch boundary, and the runtime state of every operator that depends on that input must be reset. The query-plan-owned schedule projections remain valid. Executors checked out for the invocation and invocation-owned buffers can be reset for reuse, while a build exclusively over recursion-independent base tables can retain its materialized state if repeating the build would produce the same observable result. The old runtime failed to distinguish invocation-scoped materialized state from frontier-dependent state: materialized state whose validity spans the recursive invocation cannot be owned by one epoch.

An operation is epoch-invariant when its inputs do not depend on the current frontier and reevaluating it would reproduce the same observable result. Scanning a stable base table and building a hash table from it is the canonical example. The recursive runtime follows the same ownership distinction as loop-invariant code motion: the invariant build belongs outside the epoch loop, while probing it with the changing frontier remains inside.

The ownership invariant first failed in the recursive runtime. DuckDB executes a query as a graph of pipelines. Although the old physical operator retained its recursive meta-pipeline, every epoch recreated events and executors, scheduled the pipelines and reconstructed their state. Epoch-invariant pipelines consequently reread the same static input and rebuilt the same state.

A large epoch can amortize fixed scheduling and setup costs across many frontier rows. Rebuilding invariant state remains proportional to the static input size and can cost far more than probing it with the changing frontier.

DuckDB v1.5.5 rebuilds a hash table from each frontier and revisits edges as the probe input. The new engine reverses the join, builds the edge hash table once and probes it with each frontier.

The resulting ownership boundary is:

The hash join in the opening query requires both a join reorientation and retained state. DuckDB v1.5.5 builds a hash table from the current reachable frontier and scans edges as the probe input in every epoch. The new planner places the recursion-independent edges relation on the build side where valid. The first epoch scans edges and builds the hash table; later epochs rebind reachable as the probe and reuse that table.

EXPLAIN (ANALYZE, FORMAT JSON) makes the avoided input work concrete. Both versions return the expected 20,000 nodes. DuckDB v1.5.5 reports operator_rows_scanned: 19,718,328,320 on the edges scan, equivalent in row volume to about 19,718 complete scans, while the DuckDB v2.0 preview reports the table's one million rows once. Dynamic filtering skips parts of some v1.5.5 edge scans, but it cannot prevent the base relation from being revisited in every epoch.

Retention Requires a Repeatability Proof

We retain a build only when its observable result is repeatable. Recursion independence establishes one requirement. Repeatability adds another: a seeded sample may be retained; an unseeded sample, a volatile expression such as nextval(), DML, a side-effecting operator or an unknown extension operator must be rebuilt. The repeatability classifier owns that decision and rejects retention when it cannot prove safety. This conservative choice may forgo reuse while preserving query semantics.

Fully materialized CTE producers follow the same ownership rule: a recursion-independent result may survive while each consumer scan is reset. We do not retain streaming or hybrid producers because their consumption state does not have the same lifetime. Early source termination is another boundary case. A retained pipeline must not withhold partial output merely to fill a vector, because doing so could prevent a downstream LIMIT from stopping recursive execution early. Reuse therefore preserves that termination contract.

Exact Cardinalities Make Execution Adaptive

The epoch boundary provides information unavailable during ordinary query planning. By the time an epoch ends, DuckDB has produced the next frontier in full and therefore knows its row and chunk counts exactly. These observed cardinalities govern how the recursive runtime executes the next epoch; optimizer estimates no longer have to stand in for them.

Each Epoch Chooses Its Own Mode

Once the opening query has built and retained its edge hash table, an epoch carries one reachable node into one probe. Sending that work through a parallel scheduler would create more coordination than useful work. A traversal whose frontier expands to many chunks has the opposite shape: executing it on one thread would leave independent pipeline work idle. A static mode chosen for the complete query cannot serve both cases, and a single query may move between them as its frontier grows and contracts.

When the exact frontier and physical work shape do not justify task scheduling, an epoch runs inline. One thread walks the selected immutable schedule projection and drives its operators directly, without creating tasks or entering the general scheduler. When the work classifier finds enough independent work, scheduled execution instantiates the corresponding recursive event graph and distributes work across a bounded number of workers. Both modes use the same query-plan-owned schedule projections. Executors checked out for the invocation and invocation-owned buffers are reset for reuse; only the epoch-level execution policy changes.

Rows alone do not determine useful parallelism. The policy also considers frontier chunks, the number of recursive references, independent source tasks, configured threads and the pipelines that would actually execute. The same frontier can feed several pipelines, while an independent source can expose work not represented by frontier rows. Conversely, worker-local output and boundary combination introduce costs of their own. We therefore bound workers by both recursive input and physical work units, using private output only for broad regular UNION ALL and duplicate-eliminating UNION epochs where its combine cost can amortize.

Frozen Keyed State Enables Direct Probes

Torsten Grust and Björn Bamberg explain the original USING KEY feature and its applications in their “USING KEY in Recursive CTEs” blog post. Semantically, USING KEY operates the union table as keyed state: the declared key columns identify a row, while the remaining payload columns are maintained by declared aggregates or the default last aggregate. DuckDB physically represents this state in an aggregate hash table. The recursive term can therefore probe selected keys directly instead of scanning the complete recurring state.

Each Epoch Reads a Frozen Keyed State

Direct access must preserve one semantic invariant: every access through recurring.cte_name during epoch i observes the same keyed state Sᵢ. The epoch has three ordered phases: read Sᵢ, buffer its candidate bag and commit those candidates to obtain Sᵢ₊₁. Making a candidate visible during the read phase would let another candidate from the same epoch observe it and make the result depend on worker arrival order. The three phases preserve an epoch-at-a-time state transition while probes execute concurrently.

The original implementation preserved the invariant by materializing keyed state into a collection for recurring-state reads. That representation gave every access a broad physical path: an epoch touching a few keys could still copy or scan a state containing millions of keys.

The keyed-state owner now enforces those phases. While recursive pipelines execute, recurring.cte_name reads the frozen aggregate hash table and candidates accumulate separately. The owner commits them only at the epoch boundary. A probe cannot observe a half-applied update, and hash-table growth cannot invalidate an address held by a concurrent reader. After recursion ends, the source drains the final keyed state once.

Probe Eligibility Follows Join Shape

The frozen representation supports specialized physical lookups while preserving the semantic view. An inner join that compares every declared key with = or IS NOT DISTINCT FROM can use a recursive key join, shown as RECURSIVE_KEY_JOIN in EXPLAIN, and probe the aggregate hash table directly. Such comparisons on a proper subset of a composite key can use an epoch-stable secondary hash index through RECURSIVE_PARTIAL_KEY_JOIN. New complete keys extend an index only after their addresses are stable; payload-only updates do not require index maintenance.

We select the complete-key specialization only for an inner join against a direct recurring-state scan (REC_REC_CTE_SCAN) with a direct, exactly typed scalar = or IS NOT DISTINCT FROM comparison for every declared key. Residual predicates, wrapped scans, mismatched types, nested keys and other join types use the general path. An ordinary recursive reference is also ineligible because it denotes the frontier; the accumulated keyed state is available through the recurring reference. Treating those two identities as interchangeable would change the query.

Sparse Work Should Avoid Full-State Scans

The workload below retains one million keys and advances 1,000 of them through 20 epochs:

WITH RECURSIVE state(key, value) USING KEY (key) AS (
    SELECT key, CASE WHEN key < 1_000 THEN 0 ELSE 100 END
    FROM range(1_000_000) keys(key)
    UNION ALL
    SELECT frontier.key, recurring_state.value + 1
    FROM state AS frontier
    JOIN recurring.state AS recurring_state USING (key)
    WHERE frontier.value < 20
)
SELECT count(*) AS keys, sum(value)::BIGINT AS value_sum
FROM state;

Before direct probes, the join repeatedly scanned recurring.state. The runtime metrics counted roughly 20 million full-state scan rows for 20,000 useful matches. The specialized plan replaces them with 20,000 direct probes, a 1,000× reduction in recurring-state rows examined. In a comparison of builds immediately before and after the direct-probe work, median runtime fell from 0.401 to 0.040 seconds. The new path also removes the per-epoch collection previously used to materialize the complete keyed state.

Candidates Can Leave Keyed State Unchanged

The candidate bag can be nonempty even when keyed state does not change. If candidates automatically become the next frontier, recursion may continue after the state visible to the query has converged. Retaining execution state and probing it efficiently cannot repair that semantic mismatch.

USING KEY ... UNION now gives this distinction SQL-level semantics. Both our “A Fix for the Fixation on Fixpoints” CIDR paper and “How DuckDB is USING KEY to Unlock Recursive Query Performance” SIGMOD paper describe the original candidate-frontier design, in which the candidate bag becomes the next working table regardless of whether keyed state changes. Neither defines a changed-key delta. To our knowledge, DuckDB is the first and currently the only database system to distinguish changed-key recursion under UNION from candidate-frontier recursion under UNION ALL.

Finalized State Belongs to the Commit Layer

Before this change, DuckDB followed the original design and treated USING KEY ... UNION and USING KEY ... UNION ALL alike. Every candidate produced by the recursive term became input to the next epoch, even if applying it left the recurring table unchanged. Because plain UNION had the same candidate-frontier behavior as UNION ALL, DuckDB v1.5 deprecated it for USING KEY. The changed-key semantics introduced here give the two keywords distinct meanings, so the upcoming v2.0 retains both. The observable aggregate result for a key is determined only after every candidate for the epoch has been applied. Consequently, the epoch-commit layer owns the decision whether a key changed.

A shortest-path epoch demonstrates why candidates cannot decide this themselves. Several routes may reach the same node, and every route must participate in min(distance). Only the finalized minimum is observable in keyed state. Forwarding every inferior route multiplies the work of the next epoch; forwarding an unchanged minimum can keep candidate production alive after state convergence.

Consider keyed state {A:8, B:7} with a min payload and candidates [A:9, A:5, B:7]. Applying the complete candidate bag produces {A:5, B:7}. The A:9 candidate does not affect the finalized minimum, and B:7 reproduces an existing value. UNION ALL nevertheless forwards all three candidates. UNION forwards only the finalized row A:5 because it is the only observable state change.

UNION and UNION ALL Expose Different Frontiers

Let Cᵢ be the candidate bag produced in epoch i, Sᵢ the keyed state visible during that epoch and Wᵢ₊₁ the next working table. Let update(Sᵢ, Cᵢ) apply all candidates in the bag and finalize every affected payload aggregate, and let keys(Sᵢ) denote the keys present before the update. We define the two forms as follows:

USING KEY ... UNION ALL
Sᵢ₊₁ = update(Sᵢ, Cᵢ)
Wᵢ₊₁ = Cᵢ

USING KEY ... UNION
Sᵢ₊₁ = update(Sᵢ, Cᵢ)
Wᵢ₊₁ = { Sᵢ₊₁[k] | k ∉ keys(Sᵢ) OR Sᵢ₊₁[k] IS DISTINCT FROM Sᵢ[k] }

For the same input state and candidate bag, both forms compute the same next keyed state. They differ in what becomes recursively visible: UNION ALL forwards the bag unchanged, while UNION forwards one finalized row for each new or observably changed key. Recursion under USING KEY ... UNION therefore stops when keyed state stops changing.

The epoch-commit layer implements the rule by recording prior key existence and snapshotting the pre-epoch finalized payload of an existing key when that key is first touched. It then applies every candidate before finalizing and comparing each touched key once. Duplicate candidates and losing min or max inputs still participate in aggregate evaluation, but they do not become recursive work on their own.

For the same keyed state and candidate bag, both forms compute the same state update; the next frontier is either that bag or the changed-key delta.

We compare finalized values with SQL semantics, including NULL and collations. Key identity uses the same normalization as GROUP BY, and collated scalar keys remain eligible for direct and partial-key probes. The implementation also covers multi-column and nested payloads, NaN and signed zero.

The two frontiers also have different types. A UNION ALL frontier contains raw candidate rows, so its payload columns use aggregate input types. A UNION frontier contains finalized keyed rows, so its payload columns use aggregate result types. This distinction is visible through the SQL contract and constrains the internal representation. Because the keyword selects the frontier's contents, types and termination condition, we never infer one form from the other as an optimization.

Changed-Key Deltas Still Pay for Candidate Work

A changed-key delta reduces the next frontier only after all candidates have been applied. Millions of duplicate candidates can therefore still make the recurring hash-table update expensive even when few keys ultimately change. For eligible duplicate-heavy epochs, we first combine candidates in a temporary keyed hash table, then combine those aggregate states into recurring state.

Preaggregation Requires Combinable Aggregate State

Early combination must be observationally equivalent to applying candidates directly. We enable preaggregation only when every payload aggregate provides a state-combine operation and none is order-dependent. The default last aggregate and extension aggregates without a combine callback therefore remain on the direct update path. Without a combine operation, preaggregation lacks a valid state transition. For an order-dependent aggregate, it could change the observed input order and therefore change the result.

Cardinality Evidence Governs Preaggregation

Eligibility establishes correctness; observed cardinality determines whether the valid transformation is worthwhile. Epochs smaller than one standard vector bypass classification, as do non-expanding epochs whose candidate count does not exceed the preceding frontier. Larger eligible epochs build a HyperLogLog sketch over candidate keys. We preaggregate only when the sketch's error-inflated cardinality estimate is below one quarter of the candidate count, and stop sketching early once distinct-key evidence is sufficient to reject the additional hash table. The decision therefore adapts to the duplication observed in each epoch.

The cost of proving an unchanged value constrains this policy. For a wide workload with 102,400 unique existing-key updates, median runtime increases by 0.913 milliseconds, or 6%, under the new UNION semantics. The executor must compare finalized values where the previous implementation forwarded candidates without proving a change. Aggregate-specific shortcuts for min and max would place aggregate semantics in the recursive executor, so we do not use them. An aggregate-level change-reporting contract could put that responsibility in the aggregate interface. The current interface provides no such contract.

The large motivating workload was an LDBC SF100 pathfinding query. It generated about 21 million aggregate candidates, but epoch-level aggregation produced only 3.7 million observable keyed results. The adaptive path preaggregated 17 million candidates. On matched Release builds immediately before and after the changed-key work, query time fell from a median of 19.319 to 2.948 seconds, a 6.55× speedup, while peak resident memory fell from 3.918 to 2.663 GB.

The wider regression suite bounds that result. In the same build comparison, the geometric mean across 63 recursive benchmarks improved by 5.5%, and 60 were within ±2%. Duplicate fan-in and duplicate-heavy preaggregation improved substantially; the wide unique-key case above is the one stable loss. For this reason, we classify work per epoch instead of applying the strategy that wins the motivating query to every workload.

Recursive CTEs Now Execute as One Adaptive Computation

The new engine resolves the lifetime mismatch I wanted to remove after implementing DuckDB's original recursive operator. The query plan owns the physical operator tree, schedule projections and reusable executor pool. Within each invocation, the runtime checks out executors and retains proven repeatable, recursion-independent state across epochs, while each epoch rebinds and resets frontier-dependent state. This separation makes retained invariant work and per-epoch execution policy compatible. Keyed recursion uses the epoch boundary to freeze recurring state for direct probes, then commits the candidates and exposes only observable changes as the next frontier under UNION. The implementation landed across #22211, #24031, #24565 and #24647.

That matters because recursion multiplies every physical cost placed inside the epoch loop. A scan, hash build or trip through the general scheduler that seems modest in isolation may run tens of thousands of times. The new engine pays eligible invariant costs once per invocation, avoids scheduler overhead when the observed work cannot amortize it and distributes epochs that expose sufficient independent work. Eligible keyed joins can access the required state directly, while USING KEY ... UNION recursion terminates when that state converges. This makes recursive SQL a more practical execution model for graph traversals, pathfinding and state machines, especially when the frontier remains small while the static input is large or when many candidates update the same keys.

The next step is to extend this foundation to more recursive plans. Future work can extend invocation-scoped reuse to more eligible operator states and use cardinalities observed at epoch boundaries for further execution decisions. Each extension must retain the same frontier and state-transition semantics. I intend to keep working along that boundary: reduce the physical cost of iteration while keeping recursive SQL predictable.

Teams increasingly add DuckDB to these environments for fast single-node analytics. Because the surrounding infrastructure is Java, they use the DuckDB Java client (the JDBC driver). But a query engine is only as useful as the data it can reach, and in these environments much of that data is accessible only through Java client libraries.

Bringing External Data into DuckDB

Before writing a table function to reach a new source, it is worth asking what DuckDB can already read on its own, and the answer is almost everything. It reads Parquet and other files directly from data lakes. It connects to relational databases such as PostgreSQL, MySQL, and SQLite directly. The ODBC extension reaches proprietary databases such as Oracle, SQL Server, and DB2. JSON functions can query REST-like services by treating them as remote JSON files, and community extensions cover many more systems.

For a source that none of those cover, the data typically ends up in application code, processed by hand on top of a DuckDB result set. It is often more convenient, and frequently more performant, to bring that data into DuckDB's SQL directly, where it can be filtered and joined like any other table. Table functions are meant for exactly that. DuckDB has always let you plug in a custom source this way, but until now that meant writing the function in C++, which in practice was the only realistic choice.

Extending DuckDB in Pure Java

The DuckDB Java client can now register table functions written in pure Java. Any data source you can reach from Java can be exposed as a SQL table function, reusing the same streaming, asynchronous client libraries your infrastructure already runs and trusts. You can then query it alongside local Parquet and CSV files, wildcard globs included, joining and filtering across all of them in a single SQL statement.

Until now, adding a custom source in native code meant building and shipping a DuckDB extension. That involved a C++ toolchain, notoriously complex build systems, and the operational risk that a crash in native code is a segfault that takes down the whole JVM. A pure-Java table function is an ordinary Maven dependency that runs on the JVM alongside the rest of the application, with far less risk. And because it is just a Java API, any JVM language works, so the same function could be written in Kotlin, Scala, or Clojure.

This post walks through an example project that adds a mongo_query() function to DuckDB, and uses it to explain the mechanism.

MongoDB is a document store with a robust Java API. The mongo_query() function below is a deliberately minimal, illustrative example, not a production connector. There are better ways to reach MongoDB in particular, including the ODBC extension and the MongoDB community extension. We use MongoDB as an example because it is a popular system with a good Java client. The same pattern applies to any source reachable from the JVM, whether through a JDBC driver, one of the most common Java clients, a SOAP service, or a proprietary vendor SDK.

The Goal

The function should be callable from ordinary SQL:

FROM mongo_query(
    'tab1',
    '{ "col1": "foo", "col2": { "$lt": 42 } }',
    columns = '["col1", "col2", "col3"]',
    hostname = 'localhost',
    port = 27017,
    database = 'db1'
);

The first argument is a MongoDB collection name. The second is a MongoDB filter document, in native MongoDB query syntax, passed through unchanged. The named columns parameter specifies which fields to project and the shape of the result. Because the filter is a MongoDB BSON document rather than a full SQL query, the collection name and the desired result columns cannot be read out of the query itself and are passed as separate parameters instead. Run against a seeded collection, the function returns the documents that match the filter:

┌─────────┬───────┬─────────┐
│  col1   │ col2  │  col3   │
│ varchar │ int32 │ varchar │
├─────────┼───────┼─────────┤
│ foo     │    10 │ match-a │
│ foo     │    41 │ match-b │
└─────────┴───────┴─────────┘

There is no export to Parquet step and no intermediate table. The rows are read from a MongoDB cursor during query execution. And because the result is an ordinary relation, it can be joined with anything else DuckDB can read. Here it joins a remote collection against a directory of local CSV files in one statement:

SELECT
    c.region,
    count(*)                     AS orders,
    sum(o.amount::DECIMAL(10,2)) AS revenue
FROM mongo_query(
         'orders',
         '{ "status": "shipped" }',
         columns = '["customer_id", "amount"]',
         database = 'app'
     ) AS o
JOIN 'customers/*.csv' AS c
  ON c.customer_id = o.customer_id
GROUP BY c.region;

That is the heterogeneous join. A remote system reached through its Java driver is queried together with a glob of local files in one SQL statement on a single node.

Table Functions in Java

Adding a new data source no longer requires a native extension. Through the DuckDB Java client, a table function is registered entirely in Java using the DuckDBFunctions.tableFunction() builder. The builder declares the function name, its positional and named parameters and their types, and the implementation class. Registration in the example is done in MongoExample.java:

DuckDBFunctions.tableFunction()
    .withName("mongo_query")
    .withParameter(String.class)   // collection name
    .withParameter(String.class)   // Mongo filter (JSON)
    .withNamedParameter("columns", String.class)
    .withNamedParameter("hostname", String.class)
    .withNamedParameter("port", Integer.class)
    .withNamedParameter("database", String.class)
    .withNamedParameter("username", String.class)
    .withNamedParameter("password", String.class)
    .withFunction(new MongoQueryFunction())
    .register(connection);

This completes registration. mongo_query(...) is then available as a table function on that DuckDB connection and can be used in FROM clauses, joins, CTEs, and subqueries like any built-in function.

This example takes username and password parameters and opens a fresh connection on every call. In a real, non-toy implementation it is often preferable to open a connection to the remote server once and run several queries on it. That can be done with a set of scalar functions. A mongo_connect() opens a connection and returns its handle as a DuckDB value, a mongo_query() reads through that handle, and a mongo_close() closes it. That approach is outside the scope of this post and could be the subject of a follow-up. For a similar example, see odbc_connect in the ODBC extension.

The implementation is a class that implements DuckDBTableFunction, which follows DuckDB's native table-function lifecycle with three callbacks:

• bind runs at prepare time. It reads the parameters, determines the output schema, and declares each result column with addResultColumn().
• init runs once before execution and sets up shared state, such as opening a cursor or issuing the remote request.
• apply is called repeatedly, each time filling a chunk of the output, until it returns 0.

The API also exposes an initLocal callback for per-thread local state used in multithreaded execution. This post keeps to single-threaded execution and does not use it, though it may be covered in a future post.

The example implements these in MongoQueryFunction.java, with named parameters defined in MongoQueryParameters.java.

Bind: Declare the Output Schema

bind reads the call parameters and declares the output columns. Because MongoDB documents have no fixed schema, this example asks the caller to list the columns rather than inferring them from a prepared statement, and declares each one to DuckDB:

public MongoQueryBindData bind(DuckDBTableFunctionBindInfo info)
    throws Exception {
    String collectionName = info.getParameter(0).getString();
    String queryJson      = info.getParameter(1).getString();
    String columnsJson    = info.getNamedParameter("columns").getString();
    // ... read hostname / port / database / credentials ...

    MongoClient client = MongoClients.create(settings.build());
    MongoCollection<Document> collection =
        client.getDatabase(database).getCollection(collectionName);

    List<String> columns = new ArrayList<>();
    for (BsonValue bv : BsonArray.parse(columnsJson)) {
        String name = bv.asString().getValue();
        columns.add(name);
        info.addResultColumn(name, String.class); // declare it to DuckDB
    }

    Document query = Document.parse(queryJson);
    return new MongoQueryBindData(client, collection, columns, query);
}

Two aspects here generalize beyond MongoDB. DuckDB never parses or interprets the source's query language. The filter is handed to the source's own driver essentially unchanged, apart from a JSON-to-BSON conversion, so the full source syntax stays available. The object returned from bind, here a MongoQueryBindData, then carries that state forward to the next callback. Note that bind also runs when a query is EXPLAINed, since DuckDB needs the output schema to plan the query.

Init: Open the Cursor

init runs once and prepares the execution state that apply will consume. In the example it issues the query and stores the resulting cursor in a MongoQueryInitData. It also pins execution to a single thread with setMaxThreads(1), because this post does not cover multithreaded execution:

public MongoQueryInitData init(DuckDBTableFunctionInitInfo info)
    throws Exception {
    info.setMaxThreads(1);
    MongoQueryBindData bindData = info.getBindData();
    FindIterable<Document> iter = bindData.collection.find(bindData.query);
    return new MongoQueryInitData(iter.cursor());
}

Apply: Stream Rows into Vectors

apply reflects DuckDB's vectorized execution model. Rather than returning rows, it writes them into the output data chunk column by column, up to the chunk's capacity of 2048 rows, and returns the number of rows produced:

public long apply(DuckDBTableFunctionCallInfo info,
    DuckDBDataChunkWriter output) throws Exception {
    MongoCursor<Document> cursor = info.getInitData().getResultCursor();

    long row = 0;
    for (; row < output.capacity() && cursor.hasNext(); row++) {
        Document doc = cursor.next();
        for (long col = 0; col < output.columnCount(); col++) {
            copyValueFromResultSetToVector(
                doc,
                output.vector(col),
                row,
                columns.get((int) col)
            );
        }
    }
    return row; // 0 signals "no more data"
}

Values are written into typed vectors through the vector API, using setString, setInt, setDouble, setNull, and related methods. DuckDB calls apply repeatedly, draining the source one vector-sized batch of 2048 rows at a time, so the full result does not have to be materialized in memory on either side. In the example, the BSON-to-vector conversion lives in MongoTypes.java.

Writing rows into a DuckDBDataChunkWriter here is the mirror image of reading them out. The previous post covered consuming query results through the DuckDBDataChunkReader API, where the driver hands you chunks to read. As shown in the code snippet above, a table function uses the writer side of the same columnar model to produce them.

Implications

This example is a MongoDB connector, but the broader point is that DuckDB can now be extended in pure Java, putting the rich ecosystem of Java libraries within reach as a data source:

• No native build. The project contains no C++ code. It is a Maven project with two dependencies, the DuckDB Java client and the MongoDB driver. Adding a new source is ordinary Java work, done with the tools a team already uses and without touching a native toolchain.
• Reuse of the official client. The example does not reimplement a wire protocol or query language. It uses the vendor's Java driver and passes filters through unchanged. The same applies to any system with a JDBC driver or a Java SDK, such as a REST API, a message queue, or a SOAP service.
• In-process access. The data is not exported and reloaded. It is read from a cursor during query execution. Because the result is an ordinary table function, it can be joined with Parquet files, CSV globs, or attached databases in a single SQL statement, so DuckDB acts as the query layer over the source rather than a copy of it.
• Predicates run at the source. The filter is written in the source's own query language and handed to its driver unchanged, so it executes at the source, against its indexes, and only matching rows cross the wire. The selective work happens remotely, and DuckDB streams back only what it needs.

Current Limitations

These are limitations of the current Java table-function API:

• A Java table function cannot yet be packaged as a DuckDB extension. A DuckDB extension is a native shared library loaded into the DuckDB process, whereas a Java table function requires a JVM to run. As a result, the function can only be used from the DuckDB Java client. Packaging JVM code as a loadable extension is technically possible, for example through a JNI or FFM shim that embeds the JVM, or by compiling to a GraalVM native image, but neither is supported today.

• Bind and init objects are not managed automatically. In the current release, the caller is responsible for the lifecycle of the objects returned from bind and init, for example closing the MongoDB client and cursor after the query completes. A DuckDBTableFunctionState mechanism (contributed by a community member) manages that lifecycle for you. The init object then becomes:

  public class MongoQueryInitData
      implements AutoCloseable /* DuckDBTableFunctionState */ {
  
      final MongoCursor<Document> cursor;
  
      // ...
      @Override
      public void close() {
          cursor.close();
      }
  }
  

• Composite DuckDB types are not yet supported. The vector API currently covers scalar types, but STRUCT, LIST, and other nested types are planned for the future. Until then, nested source data has to be flattened or serialized to a scalar column.

Conclusion

For many analytical workloads, the data does not have to move to a cluster, and you do not need a distributed query engine to join across systems. With pure-Java table functions, the client libraries already running in your infrastructure expose remote data sources as SQL tables, and DuckDB joins them against each other and against local files in a single query on one node, using the Java client libraries you already have.

The table-function API is documented under Defining Functions. The complete example is available in the duckdb_mongo_example repository, and the DuckDB team is always happy to talk Java in the #java channel on the DuckDB Discord.

JDBC, on the other hand, was designed in 1997 around a very different idea. The ResultSet API hands you one row at a time (next()) and one value at a time (getInt(1), getString(2) and so on). It is a solid, familiar API, supported across the Java ecosystem, but it forces the data into a form that DuckDB never used internally.

The DuckDB Java driver has to bridge the two. It embeds the native DuckDB library and talks to it over JNI. When the engine has already produced a columnar chunk of 2,048 rows, the JDBC specification requires the driver to slice it back into rows and cells before you can read it. If your application is going to put those values right back into columnar form, such as arrays, Arrow buffers or a machine learning feature matrix, the data is split into rows only to be reassembled into columns, wasting work on both sides.

Version 1.5.3.0 of the Java driver adds an alternative.

Tip We have thoroughly updated the Java client documentation, which now covers the driver's full API surface across dedicated pages: defining connections, running queries, handling results (including Arrow methods and streaming results), importing data with the appender and batch writer, and defining functions. The chunked query results discussed in this post are documented under Chunked Results.

Reading Results Row by Row

A typical JDBC read loop looks like this:

try (ResultSet rs = stmt.executeQuery("SELECT a, b FROM measurements")) {
    while (rs.next()) {
        long a = rs.getLong(1);
        double b = rs.getDouble(2);
        // ...
    }
}

Calling next() advances the cursor, and a get* call then locates the value inside the current native chunk and converts it to a Java representation. For some non-primitive types that conversion has to cross the JNI boundary on every value, and a few conversions are more involved, such as fetching a timestamp against a caller-supplied java.util.Calendar (the pre-java.time class the JDBC specification still mandates here). None of these steps is expensive on its own, but multiplied by millions of rows times a handful of columns, they add up. This is exactly the per-value interpretation overhead that vectorized execution is designed to avoid.

The loop also carries fixed overhead. So that a single connection can keep several result sets open at once, result streaming is off by default, which means a query's entire result is read into memory unless you set jdbc_stream_results, an option most users never set. The driver also assembles a full set of result metadata. That happens once per result rather than once per row, so it is cheap next to the row loop, but it is not free, and most queries never look at it.

The ResultSet path is the standard your ORM and the rest of the ecosystem already use, and for the vast majority of queries, the ones returning hundreds or thousands of rows, it is the right choice. The chunked API is for the other case: a query returns a lot of data, you control both ends of the pipe, and reading it row by row provides no advantage.

Reading Chunks with DuckDBChunkedResult

The Java driver now exposes the engine's native chunk stream directly, via the same mechanism the C API offers as duckdb_fetch_chunk. A query result becomes a lazily fetched sequence of data chunks, and you read column vectors out of each chunk in batches, the way the engine produced them, avoiding the per-row overhead required by the JDBC specification.

Putting that together:

try (DuckDBConnection conn = DriverManager
        .getConnection("jdbc:duckdb:")
        .unwrap(DuckDBConnection.class);
     DuckDBPreparedStatement ps = conn.prepare(
         "SELECT l_orderkey, l_linenumber, l_shipmode "
             + "FROM 's3://my-bucket-name/lineitems.parquet'")) {

    try (DuckDBChunkedResult res = ps.query()) {

        // advance to the next chunk, returns true on success
        while (res.nextChunk()) {

            // get the current chunk from the result
            DuckDBDataChunkReader chunk = res.chunk();

            // vectors are addressed by 0-based column index
            DuckDBReadableVector orderKeys = chunk.vector(0);
            DuckDBReadableVector lineNumbers = chunk.vector(1);
            DuckDBReadableVector shipModes = chunk.vector(2);

            // read each column with the getter for its type
            for (long row = 0; row < chunk.rowCount(); row++) {
                long orderKey = orderKeys.getLong(row);
                int lineNumber = lineNumbers.getInt(row);
                String shipMode = shipModes.getString(row);
                System.out.println(orderKey + " " + lineNumber + " " + shipMode);
            }
        }
    }
}

This approach has a few notable properties:

• Laziness. nextChunk() pulls one chunk at a time from the engine. The full result is never materialized on the native side or the Java side, so you can stream through results far larger than your heap, the same way the engine itself produces them.

• Columnar access. Inside a chunk you work vector by vector. If your destination is columnar too, such as a long[], an Arrow VectorSchemaRoot or a Parquet writer, you copy values in tight, monomorphic loops instead of switching columns on every row.

• Enough metadata to dispatch. Each result still reports its column count and column types, so you can pick the right get* call per vector without carrying a separate schema description alongside the data.

• A familiar API if you have written a UDF. Chunk contents are accessed through the same DuckDBDataChunkReader API that the driver's user-defined functions use to read their input vectors. So the same code that reads a function's arguments inside a UDF also reads query results outside one.

• Zero-based indexing. Chunk columns and rows are 0-based, matching the C API and the UDF interfaces.

Inside a vector you are free to iterate however suits your code, a Java Stream included. A parallel stream over a single chunk rarely helps, though: 2,048 values is small enough that fork/join coordination usually costs more than it saves. If you want parallelism, apply it across chunks rather than within a single one.

Current Limitations

This is the first iteration of the API, and there are a few limitations you should know about before adopting it:

• Basic data types only, for now. The scalar types are supported. Composite types (LIST, STRUCT) are not yet readable through the chunked interface. Support for them is planned for a future release.

• Prepared statements only. query() currently exists on DuckDBPreparedStatement, and there is no query(String) convenience overload yet. Preparing the statement first adds one line.

• A small reader surface, for now. The reader covers the essential types today, with more coverage planned for future releases. One addition already planned is reading VARCHAR values directly as UTF-8 byte[], so text-heavy workloads can skip materializing a Java String per value.

If any of these blocks a use case you care about, please open an issue. Your feedback is what drives the priorities for the next iteration.

Conclusion

The JDBC ResultSet remains the right default for most applications. But the engine produces its results in columnar chunks, and DuckDBChunkedResult lets your Java code read them in that same form: directly and lazily, with little overhead at the JNI boundary.

The feature is provided by duckdb-java#682 and is available in the current duckdb_jdbc releases on Maven Central. Try it on your largest result set and let us know how it performs. Issues and pull requests are welcome, and the DuckDB team is always happy to talk Java in the #java channel on the DuckDB Discord.

What is the Role of a Parser?

At a high level, DuckDB processes a SQL query through the following stages:

In this blog, we focus on the tokenizer, parser, and transformer:

• Tokenizer: This is the first step and is responsible for splitting up the raw input string into tokens. These can be of various categories, for example: KEYWORD, NUMBER, or IDENTIFIER. It is also where comments, in SQL denoted with either -- or /* */, are recognized and skipped.
• Parser: The parser determines whether these tokens follow DuckDB's grammar and produces a ParseResult tree.
• Transformer: Converts the generic parse results into DuckDB’s internal abstract syntax tree (AST), forming structures such as SQLStatement, TableRef, and ParsedExpression. The resulting AST is passed on to the binder.

The parser determines whether a query is syntactically valid, while the binder determines whether the tables, columns, and functions it refers to actually exist.

Consider the following query:

SELECT *
WHERE true
FROM range(1);

Parser Error:
syntax error at or near "FROM"

LINE 3: FROM range(1);
        ^^^^

Every individual token in this query is valid, but the clauses occur in an order that DuckDB’s grammar does not accept. Friendly SQL allows both SELECT-first and FROM-first syntax, but it does not allow the clauses to appear in an arbitrary order.

By comparison, the following query is syntactically valid, so it passes the parser and transformer. However, it fails later in the binder because the table missing_table does not exist.

FROM missing_table;

Catalog Error:
Table with name missing_table does not exist!

LINE 1: FROM missing_table;
             ^^^^^^^^^^^^^

The DuckDB SQL Dialect

Although a SQL standard exists, every database system supports different parts of the standard and adds its own syntax and behavior. The resulting variants are commonly referred to as SQL dialects. Examples include the dialects supported by PostgreSQL, Oracle, GoogleSQL for BigQuery, MySQL, MariaDB, SQLite, Spark SQL, and, of course, DuckDB.

DuckDB’s SQL closely follows PostgreSQL conventions, but it has evolved considerably over the years. We have added features of our own, such as GROUP BY ALL, as well as features inspired by other database systems. At the same time, DuckDB does not implement every aspect of PostgreSQL’s behavior. DuckDB therefore speaks its own SQL dialect, which we will refer to as DuckSQL in this post, even though it remains strongly influenced by PostgreSQL.

This distinction is important when talking about the parser. The SQL dialect that DuckDB accepts and the implementation used to parse that SQL are two separate things. For DuckDB v2.0, we are replacing the parser implementation and rewriting its grammar. What we are not replacing is DuckSQL itself.

Outgrowing the PostgreSQL-Derived Parser

When DuckDB started out, it made a lot of sense to use the PostgreSQL-derived parser and grammar. This parser was already part of the first commit to DuckDB in 2018. It gave DuckDB a mature, battle-tested SQL grammar based on syntax that many users were already familiar with. We adapted the parser to our needs and added a Transformer that converted the resulting PostgreSQL-style parse tree into DuckDB’s internal AST.

However, over the years this parser also came with some downsides. Extending DuckSQL meant modifying the underlying YACC/Bison grammar. Because Bison generates an LALR(1) parser, seemingly small additions to the grammar can interact with existing rules and introduce shift/reduce or reduce/reduce conflicts. As DuckSQL grew, making changes to the grammar therefore became increasingly difficult.

This was one of the motivations behind our earlier blog post on runtime-extensible SQL parsers. In that post and the accompanying CIDR paper, we explored whether Parsing Expression Grammars (PEGs) could provide a better foundation for an extensible database parser. At the time, the PEG parser was still an experimental prototype capable of parsing only a subset of SQL.

A Primer on PEG Parsers

Before looking at how we turned the prototype into a production parser, let us briefly revisit how a PEG describes a language.

A PEG consists of named rules that describe how an input should be matched. Consider the following rules from DuckDB’s new grammar:

SelectFrom <- SelectFromClause / FromSelectClause
SelectFromClause <- SelectClause FromClause?
FromSelectClause <- FromClause SelectClause?

The <- operator defines a rule, / specifies a choice between alternatives, and ? makes an element optional. Together, these rules state that DuckSQL accepts both a traditional SELECT-first query:

SELECT *
FROM range(1);

And DuckDB’s Friendly SQL FROM-first equivalent:

FROM range(1)
SELECT *;

A PEG evaluates alternatives in order. When matching SelectFrom, the parser first attempts SelectFromClause. If that does not match, it attempts FromSelectClause. The first successful alternative is selected. As a result, PEG grammars do not have the same shift/reduce and reduce/reduce conflicts as LALR grammars. Instead, alternatives are ordered explicitly, and that order forms part of the grammar’s behavior.

We are not the only ones changing to a PEG-based parser. Python switched from its LL(1) parser to a PEG-based parser in Python 3.9, also motivated by the additional flexibility PEG provides to evolve the language.

In DuckDB, these rules operate on the tokens produced by the tokenizer. The matcher applies the grammar rules to those tokens and constructs a generic ParseResult tree, which is subsequently transformed into DuckDB’s internal AST.

Going from Prototype to Production

The research prototype demonstrated that a PEG-based SQL parser was feasible. Replacing DuckDB’s existing parser, however, required considerably more than parsing a subset of SQL. The new parser had to accept all of DuckSQL and produce the same AST expected by DuckDB’s binder.

The PEG grammar was first introduced in DuckDB v1.2, where it handled autocomplete in the CLI. Later, in DuckDB v1.5, we introduced the complete PEG parser as an experimental, opt-in feature. We also used it for an April Fools' joke that made DuckDB speak Dutch. Since then, the grammar, matcher, and transformer have been steadily improved to make the PEG parser the default for DuckDB v2.0.

Among other things, the parser had to support:

• Every statement and expression type: Supporting the complete DuckSQL dialect includes both common syntax as well as the less frequently used statements and expressions.
• Operator precedence and associativity: For example, SELECT true OR true AND false; must be interpreted as (true OR (true AND false)), because AND binds more tightly than OR.
• Correct keyword classification: Some keywords, such as SELECT, are RESERVED and cannot be used as unquoted table or column names. Other keywords may be used as identifiers depending on their context.
• Compatibility with DuckDB’s internal AST: The PEG transformer must produce the same DuckDB AST structures as the transformer for the PostgreSQL-derived parse nodes wherever the language behavior is intended to remain unchanged.
• Correct error reporting: For an invalid query, the parser should report where parsing failed and, where possible, provide context and a useful indication of what went wrong. Ideally, it should do so without pointing to a manual.
• Performance on unusual inputs: Besides keeping normal parsing fast, we also had to make sure that malformed queries do not suddenly take a long time to parse.

Avoiding Repeated Work with Packrat Parsing

One issue we encountered was repeated work during backtracking. A naïve PEG matcher can evaluate the same grammar rule at the same token position many times while trying different alternatives. For certain malformed inputs, the amount of repeated work can grow exponentially.

We encountered this with queries containing a large number of unmatched opening parentheses:

SELECT ((((((((((((((((((;

With the experimental PEG parser shipped in v1.5, adding one more opening parenthesis approximately doubled the parsing time:

18 opening parentheses:  5.303 seconds
19 opening parentheses: 10.640 seconds

We addressed this using packrat parsing, a memoization technique commonly used with PEG parsers. For each memoized matcher, we store the result of applying it at a particular token position. If the parser later attempts the same matcher at the same position, it reuses the cached result instead of evaluating it again.

With packrat parsing enabled, the same malformed query was rejected almost instantly:

19 opening parentheses: 0.001 seconds

As a result, a memoized matcher is evaluated at most once at a particular token position, removing the repeated work that caused the exponential behavior in this example. This requires additional memory while parsing, but that is a worthwhile trade-off for avoiding cases such as this.

Turning the prototype into a production parser involved much more than translating the grammar. The new parser had to cover the complete DuckSQL dialect, preserve DuckDB’s existing AST, remain compatible with existing queries, and handle both valid and malformed input efficiently.

The resulting architecture replaces the PostgreSQL-derived parser front end, while the binder and the remainder of DuckDB’s query-processing pipeline continue to operate on the same internal AST.

DuckDB Parser architecture. Key idea: replace the PostgreSQL-derived parse front end while keeping the rest of DuckDB's execution pipeline the same.

Evolving DuckSQL

With the PEG parser now in place for DuckDB v2.0, we have also continued to extend DuckSQL with new syntax.

One example is the new expression-statement syntax. Until now, executing a query consisting only of expressions always required writing a SELECT:

SELECT date: current_date(), time: current_localtime();

With an expression statement, the SELECT can be omitted:

date: current_date(), time: current_localtime();

As a bonus, this also works with prefix aliases.

Another example is the new CONNECT statement, introduced for Quack. It allows you to connect to a remote database and route subsequent queries to it until you run DISCONNECT:

CONNECT 'postgres://localhost/mydb';
SELECT count(*) FROM orders; -- Runs on the PostgreSQL server
DISCONNECT;

There will also be new syntax for working with external resources. This will allow you to manage resources that live outside DuckDB through an extension. You will be able to create, register, inspect, connect to, or destroy a resource all from within DuckDB:

CREATE EXTERNAL RESOURCE '<resource-type>' AS <name> (...);
REGISTER EXTERNAL RESOURCE '<resource-type>' AS <name> FROM <handle>;

SHOW EXTERNAL RESOURCES;

CONNECT TO EXTERNAL RESOURCE <name>;

DESTROY EXTERNAL RESOURCE <name>;

We have also extended COPY TO with PARTITION BY and ORDER BY syntax:

COPY orders TO 'orders'
(
    FORMAT parquet,
    PARTITION BY (year, month),
    ORDER BY (order_date)
);

These additions would also have been possible with the old PostgreSQL-derived parser, but adding them would have been considerably more cumbersome. The PEG grammar makes it easier for us to continue evolving DuckSQL.

So far, these rules are all part of DuckSQL itself. The next step is allowing extensions to add rules of their own.

Extending the Parser

Extensions are a central part of DuckDB. They can already add scalar and table functions, optimizer rules, query-plan rewrites, and even custom physical operators.

Extensions that add new syntax already exist, such as psql and duckpgq, but under the hood they work as fallback parsers. DuckDB first tries to parse the query itself and only calls the extension if that fails. This works well for self-contained syntax, but an extension that wants to add syntax inside SQL also has to parse the surrounding SQL itself. These fallback parsers also make it impossible to combine the syntax of multiple extensions.

With the PEG parser, extensions can instead extend individual parts of DuckDB’s parser. They can extend the tokenizer, add grammar rules, and register custom matchers while continuing to reuse the rest of DuckSQL.

Warning The API shown below is still a preview and may change before DuckDB v2.0. You can follow the ongoing development on GitHub.

To make this concrete, we use Google’s pipe query syntax. This is an extension to SQL that adds piped data flow syntax. Pipe syntax expresses a query as a sequence of operators, where each operator consumes the result of the previous one.

FROM produce
  |> WHERE
        item != 'bananas'
        AND category IN ('fruit', 'nut')
  |> AGGREGATE COUNT(*) AS num_items, SUM(sales) AS total_sales
     GROUP BY item
  |> ORDER BY item DESC;

A simplified PEG grammar for this needs a handful of rules:

PipeSelectAtom <- PipeSource PipeStage+
PipeSource <- FromClause / SelectStatementType / SelectParens
PipeStage <- '|>' PipeOperator
PipeOperator <- PipeAggregate / PipeAggregateGroupOnly / PipeWhere / PipeSelect / PipeExtend / PipeDistinct / PipeOrderBy / PipeLimit
PipeWhere <- WhereClause
PipeSelect <- 'SELECT' TargetList
PipeExtend <- 'EXTEND' TargetList
PipeDistinct <- 'DISTINCT'
PipeOrderBy <- OrderByClause
PipeLimit <- LimitClause OffsetClause?
PipeAggregate <- 'AGGREGATE' TargetList GroupByClause?
PipeAggregateGroupOnly <- 'AGGREGATE' GroupByClause

Here, + means that PipeStage must occur one or more times, so a pipe query must contain at least one pipe operator.

This grammar can reuse existing rules, such as GroupByClause, to reduce the amount of grammar the extension needs to define. An extension can still define its own rule where DuckDB’s existing syntax does not fit.

Registering the Grammar

Defining just the PEG rules does not yet make them part of DuckDB’s grammar. The extension must also specify (1) the existing grammar rule it wants to extend and (2) the transformer rules that convert the new syntax into DuckDB’s AST.

In the current prototype, certain grammar rules expose extension points. Pipe SQL registers PipeSelectAtom as an additional alternative for SelectAtom, together with the new keywords AGGREGATE and EXTEND.

static void LoadInternal(ExtensionLoader &loader) {
    ParserExtension extension;
    extension.grammar_extension.grammar = PIPE_SQL_GRAMMAR;
    extension.grammar_extension.select_atom_rule = "PipeSelectAtom";

    extension.grammar_extension.RegisterSelectAtomTransformer(
        "PipeSelectAtom",
        TransformPipeSelectAtom
    );

    loader.RegisterKeyword(
        "aggregate",
        ExtensionKeywordCategory::RESERVED
    );
    loader.RegisterKeyword(
        "extend",
        ExtensionKeywordCategory::RESERVED
    );

    loader.RegisterParserExtension(std::move(extension));
}

By registering this alternative, the resulting grammar is effectively:

SelectAtom <-
    PipeSelectAtom /
    SelectParens /
    SelectStatementType

The extension alternative is now tried first. If no pipe syntax is present, it fails without consuming any tokens and the query is parsed with the built-in alternatives.

Transforming the Result

Adding a grammar rule only gets us as far as a ParseResult. The extension still needs to transform that result into the DuckDB AST that is expected by the binder. Since PipeSelectAtom extends SelectAtom, its transformer returns a SelectStatement:

static unique_ptr<SelectStatement>
TransformPipeSelectAtom(PEGTransformer &transformer, ParseResult &parse_result) {
    auto &pipe = parse_result.Cast<ListParseResult>();

    // PipeSelectAtom <- PipeSource PipeStage+
    auto statement = TransformPipeSource(transformer, pipe.GetChild(0));

    auto &stages = pipe.Child<RepeatParseResult>(1);
    for (auto &stage : stages.GetChildren()) {
        ApplyPipeStage(transformer, stage.get(), *statement);
    }

    return statement;
}

The shape of the ParseResult follows the grammar rule we defined earlier. PipeSelectAtom contains a PipeSource and one or more PipeStages. We first transform the PipeSource into a DuckDB SelectStatement. Each PipeStage is then applied to that statement in order. The resulting SelectStatement is then returned and can continue through the rest of the parser's pipeline and eventually on to the binder.

This is where reusing DuckDB's existing grammar becomes especially useful. The extension only needs to transform the new syntax it introduced. When it reuses an existing DuckDB grammar rule, such as GroupByClause, it can also reuse the corresponding transform function instead of having to implement GROUP BY itself.

This is an important difference from the fallback parsers that are available today. An extension no longer needs to implement expressions, table references, GROUP BY clauses, and the rest of SQL itself. Instead, it can add only the syntax it needs and reuse DuckDB’s grammar and transformations for everything else.

Executing Pipe SQL

With the extension registered, we can now execute queries using the new pipe syntax. For example, we can combine the pipe operators added by the extension with existing DuckSQL features such as range() and prefix aliases:

FROM range(6) t(i)
  |> WHERE i % 2 = 0
  |> SELECT i, doubled: i * 2
  |> ORDER BY i DESC;

┌───────┬─────────┐
│   i   │ doubled │
│ int64 │  int64  │
├───────┼─────────┤
│     4 │       8 │
│     2 │       4 │
│     0 │       0 │
└───────┴─────────┘

The extension only defines the pipe-specific syntax. Expressions, table references, WHERE, SELECT, ORDER BY, and other reused rules are still parsed and transformed by DuckDB itself. This means that new syntax can be combined with DuckSQL without the extension having to implement the rest of SQL again.

To Conclude

With DuckDB v2.0, we are replacing the PostgreSQL-derived parser with a new PEG parser. Existing DuckSQL queries should continue working as before. Under the hood, however, the new parser gives us something that is easier to evolve and designed for runtime extensibility.

The runtime grammar extension API shown in this post is still a preview and may change before v2.0 is released. However, the underlying idea is already working. Extensions can add their own syntax directly to DuckDB's grammar while reusing its existing rules and transformations. This means they no longer need to parse the rest of SQL themselves.

We are excited to see what new syntax the community will create. In the meantime, we will continue evolving DuckSQL and improving the parser.

If you do find an existing query that behaves differently with the PEG parser, please let us know by filing an issue.

Guest blog post by Mustafa Khan (Atlan).

DuckDB's JSON extension already covers reading JSON, extracting paths, applying RFC 7396 merge patches, and the usual scalar accessors. The upcoming v2.0 release will extend it with four new scalar functions: json_merge_patch_diff computes the inverse of an RFC 7396 merge patch, json_deep_merge applies patches with skip-on-null semantics, json_normalize canonicalizes key order, and json_strip_nulls recursively removes null-valued keys. You can try these primitives today by installing the preview version of DuckDB v2.0-dev.

These primitives address a handful of recurring problems in data pipelines that reconcile state between systems. The examples in this post come from Atlan. Atlan is a context layer for AI: a platform that gathers metadata, lineage, and semantics from a company's data systems so that data teams and AI agents can find and understand the data. Keeping that layer up to date means constantly reconciling entity documents from many upstream sources. Two upstream services may describe the same entity with different key orderings. Partial updates may carry null in a field to indicate either deletion or absence of data. Most events change only one or two fields out of dozens. Handling these cases in SQL previously required leaving the database. The four new functions allow you to perform the work in a single query.

In the rest of this post, we walk through each function, then chain them into an end-to-end reconciliation example. Finally, we present the results of a simple benchmark against equivalent Python implementations on 500,000 synthetic change data capture (CDC) events.

json_merge_patch_diff: The Inverse of json_merge_patch

json_merge_patch_diff(orig, modified) returns the minimal RFC 7396 patch such that json_merge_patch(orig, patch) = modified. json_merge_patch already exists in DuckDB. It applies an RFC 7396 patch to a document: a null value in the patch deletes the key, nested objects are merged recursively, and any other value overwrites the original. Deleted keys appear as null in the patch, changed and added keys appear with their new values, and unchanged keys are omitted entirely.

SELECT json_merge_patch_diff('{"a":1,"b":2,"c":3}', '{"a":1,"b":99,"d":4}');

{"c":null,"b":99,"d":4}

a is unchanged, so it is omitted. b changed. c was removed, so it becomes null. d is new.

The function recurses into nested objects, producing patches that only touch the changed paths:

SELECT json_merge_patch_diff(
    '{"user":{"name":"Alice","age":30}}',
    '{"user":{"name":"Alice","age":31}}'
);

{"user":{"age":31}}

The round trip with json_merge_patch holds by construction:

SELECT json_merge_patch(
    '{"a":1,"b":2,"c":3}',
    json_merge_patch_diff('{"a":1,"b":2,"c":3}', '{"a":1,"b":99,"d":4}')
) = '{"a":1,"b":99,"d":4}' AS round_trips;

true

Equality between subtrees is computed with yyjson_equals, which compares two yyjson values structurally without serializing either side. Subtrees that compare equal contribute nothing to the output and short-circuit the recursion:

SELECT json_merge_patch_diff(
    '{"a":{"b":{"c":{"d":1}}}}',
    '{"a":{"b":{"c":{"d":1}}}}'
);

{}

This function is particularly useful for CDC pipelines, where most events on a metadata catalog touch one or two fields out of dozens. Shipping the output of json_merge_patch_diff(prev_state, new_state) downstream, instead of the full new state, cuts the change payload to a small fraction of its previous size. The diff is the change itself, so there is nothing extra to filter on the consumer side, and a json_merge_patch on the other end reconstructs the new state from prev_state and the patch.

json_deep_merge: Recursive Merge where null Means “Skip”

The patches from the previous section are applied with json_merge_patch, which strictly follows RFC 7396: null in the patch deletes the key. That is the right rule for the round trip with json_merge_patch_diff, but it is the wrong rule when you are reconciling fragments from multiple upstreams that emit null to mean “I do not have a value for this field on this message.” json_deep_merge covers that case.

json_deep_merge follows the same recursive merge structure as json_merge_patch, with one difference: a null value in the patch means “keep the original value” instead of “delete the key”. Everything else, including handling of non-null values, nested objects, and the variadic argument shape, matches json_merge_patch. In the following example, we apply a patch with the key b set to null with json_merge_patch and json_deep_merge, respectively, resulting in two different results:

SELECT json_merge_patch('{"a":1,"b":2}', '{"b":null}') AS rfc_7396;

{"a":1}

SELECT json_deep_merge('{"a":1,"b":2}', '{"b":null}') AS deep_merge;

{"a":1,"b":2}

Both functions replace the original value with the patch value when the patch contains a non-null value, and both recurse into nested objects when the original and patch are both objects. The handling of null is the only divergence:

SELECT json_deep_merge(
    '{"a":{"x":1,"y":2}}',
    '{"a":{"y":null,"z":3}}'
);

{"a":{"x":1,"y":2,"z":3}}

The nested y is preserved because the patch contained null. z is added because the patch had a real value. The same patch through json_merge_patch deletes y instead:

SELECT json_merge_patch(
    '{"a":{"x":1,"y":2}}',
    '{"a":{"y":null,"z":3}}'
);

{"a":{"x":1,"z":3}}

The function is variadic: pass any number of patches and they apply left to right.

SELECT json_deep_merge(
    '{"a":1}',
    '{"a":null}',
    '{"a":2}'
);

{"a":2}

The first patch skips (keeps a=1), the second overwrites (a becomes 2).

The use case that motivates the skip-on-null rule is reconciling fragments from multiple sources. One source knows the column name (columnName) but not its parent table (parentColumn). Another knows the parent but not the column name. Each emits a fragment with the unknown fields as null:

SELECT json_deep_merge(
    '{"columnName":"user_id","parentColumn":null}',
    '{"columnName":null,"parentColumn":"accounts.id"}'
);

{"columnName":"user_id","parentColumn":"accounts.id"}

Note that passing SQL NULL as an argument and using JSON null values result in different behaviors. A SQL NULL patch makes the result NULL, and a SQL NULL original is ignored, matching the behavior of json_merge_patch. JSON null inside a patch means “keep the original value for this key”.

json_normalize: Canonical Form for Hashing

The third primitive answers the first of the questions this post opened with: when two services emit the same JSON object with different key orders, are they the same document? Lexically they are not, which means comparing the content hashes of the raw strings will conclude they differ. json_normalize fixes that by recursively sorting the keys of every object, including objects nested inside arrays. Array elements keep their order, since element order is meaningful in JSON and key order is not. Scalar values are returned unchanged.

SELECT json_normalize('{"z":1,"a":2,"m":3}');

{"a":2,"m":3,"z":1}

Recursion descends into nested objects, and into objects nested inside arrays:

SELECT json_normalize('{"c":{"b":{"z":1,"a":2},"a":3},"a":4}');

{"a":4,"c":{"a":3,"b":{"a":2,"z":1}}}

SELECT json_normalize('[{"z":1,"a":2},{"y":3,"b":4}]');

[{"a":2,"z":1},{"b":4,"y":3}]

Once two semantically-equal documents normalize to the same bytes, hashing them collapses duplicates trivially:

SELECT
    md5(json_normalize('{"z":1,"a":2}')) =
    md5(json_normalize('{"a":2,"z":1}')) AS same_hash;

true

A GROUP BY md5(json_normalize(payload)) collapses duplicate emissions to a single row, deduplicating entities that differ only in key ordering. This is useful for a metadata catalog where the same entity may be described by two different upstream sources with slightly different key orderings.

Internally, the function walks the mutable doc, sorts each object's key-value pairs with std::sort, and rebuilds the object in order.

json_strip_nulls: Recursively Drop Null-Valued Keys

json_strip_nulls is a cleanup helper. It removes every key whose value is JSON null, recursively. Array elements are not modified (since null is a valid array element), and scalar values are returned unchanged.

SELECT json_strip_nulls('{"a":1,"b":null,"c":null,"d":2}');

{"a":1,"d":2}

It descends into nested objects:

SELECT json_strip_nulls('{"a":{"x":1,"y":null},"b":2}');

{"a":{"x":1},"b":2}

And into objects inside arrays, while leaving the array structure intact:

SELECT json_strip_nulls('{"a":[{"x":1,"y":null},{"z":null}]}');

{"a":[{"x":1},{}]}

The motivating case is cleaning patches before storing them. After computing a json_merge_patch_diff, the patch contains null entries for deleted keys, which is correct under RFC 7396. If you only want to ship the additions and updates, json_strip_nulls(json_merge_patch_diff(orig, modified)) removes those null entries from the patch. The resulting patch can add and update fields but never delete them. The same idea applies to trimming verbose API responses before hashing or storing them.

Under the hood, this uses yyjson_mut_obj_iter_remove, which deletes keys during iteration without invalidating the iterator. Calling json_strip_nulls on an already-clean document reduces to a parse, one pass, and a serialize.

Composing Them: An End-to-End Reconciliation

The four functions compose into a small reconciliation workflow. Suppose an upstream system emits partial JSON documents on each change. Some fields are present with new values, some are absent because the upstream did not include them, and some are explicitly null because the upstream has no data for them on this event. The goal is to clean the event, compute the minimal patch against the previous state, ship that patch, and store a canonical hash of the resulting document so identical entities collapse on deduplication.

Let's see this workflow through a concrete example:

CREATE TABLE catalog (id INTEGER, state JSON);
INSERT INTO catalog VALUES (
    1,
    '{"typeName":"Column","name":"user_id","description":"primary key","dataType":"BIGINT","ownerEmail":"alice@example.com"}'
);

CREATE TABLE incoming_events (id INTEGER, event JSON);
INSERT INTO incoming_events VALUES (
    1,
    '{"name":"user_id","description":"primary key for the users table","dataType":"BIGINT","ownerEmail":null,"team":null}'
);

Step 1: strip the null placeholders out of the incoming event so they do not get treated as deletes:

SELECT json_strip_nulls(event) AS clean_event
FROM incoming_events
WHERE id = 1;

{"name":"user_id","description":"primary key for the users table","dataType":"BIGINT"}

Step 2: produce a minimal patch against the current state. Deep-merge the cleaned event onto the original (so missing fields keep their previous value), then diff back against the original:

SELECT json_merge_patch_diff(
    c.state,
    json_deep_merge(c.state, json_strip_nulls(e.event))
) AS patch
FROM catalog c
JOIN incoming_events e USING (id);

{"description":"primary key for the users table"}

Only the changed field appears. dataType was unchanged, ownerEmail and team were null and got stripped, so neither are returned. The patch is what you would ship downstream.

Step 3: apply the patch back and hash the normalized result:

SELECT
    md5(json_normalize(
        json_merge_patch(
            c.state,
            '{"description":"primary key for the users table"}'
        )
    )) AS content_hash
FROM catalog c
WHERE id = 1;

21ceaced24d8958d71d537f6fcebe8a2

Each function performs a single transformation, and the whole chain runs in a single query.

Benchmarks: DuckDB vs. Python on 500,000 CDC Events

The benchmark below compares each function and the full composed chain against equivalent Python implementations on 500,000 synthetic catalog-event pairs (roughly 600 bytes per document). The Python implementations parse with json.loads, apply the equivalent transformation in pure Python (the Python output matches DuckDB's exactly on 100 sample documents), and serialize back with json.dumps. That is the closest apples-to-apples comparison to what DuckDB does end-to-end on a JSON column.

The benchmark methodology is as follows. Each measurement is the median of five iterations. Python timings include parse, transform, and serialize per row, with data pre-loaded into memory so disk I/O is excluded. For DuckDB, we timed the whole run from the command line and subtracted a no-op SELECT json run, so only the transformation is measured. Hardware is an Apple M3 Pro (arm64) with 18 GB RAM, Python 3.11. DuckDB runs multi-threaded by default, while the Python script is single-threaded. The benchmark code and synthetic data generator are available on GitHub.

The speedup comes from two sources: DuckDB operates on yyjson trees in place, while the Python equivalent spends most of its cycles in json.loads and json.dumps. The result is a speedup of about 1-2 orders of magnitude, depending on how much transformation work the function does. json_strip_nulls is the extreme case, since the work after parsing reduces to a single iterator pass with an in-place delete. On the composed chain, the per-function gap narrows because the absolute work goes up on both sides, but DuckDB still finishes the full reconciliation of 500,000 event pairs in roughly one second.

Performance Notes

There are a few characteristics worth pointing out:

• All four functions operate on yyjson mutable values directly.
• json_merge_patch_diff uses yyjson_equals for deep equality, and subtrees that compare equal stop the recursion.
• json_normalize's sort is per-object, not global. The cost scales with the size of each object, not the document.
• All four are plain scalar functions that process one vector of rows at a time, so they fit DuckDB's vectorized execution model.

Trying the Functions

The four functions will ship in DuckDB v2.0. To try them today, install a v2.0-dev preview build, then load the JSON extension:

LOAD json;

SELECT json_normalize('{"z":1,"a":2}');

Wrapping Up

These four functions extend the DuckDB JSON extension with primitives for diffing, merging, normalizing, and stripping null values. They compose into reconciliation pipelines that previously required transformations outside SQL.

That's not all. The functions json_set(json, path, value), json_remove, json_insert, and json_replace are also available in DuckDB v2.0 preview (#23786), completing the set of edit primitives defined in the SQL/JSON specification. For the implementations, see the JSON extension source in extension/json.

Feedback and bug reports are welcome on Discord or GitHub.

A major version bump is not something we do lightly, and it is not just ceremony: v2.0 ships a new SQL parser, a new default storage format, a reworked C API, and a small number of carefully chosen breaking changes. But above all, it is a feature release, built from over 10,000 commits since we released v1.5 in March. Where last year was the year of the lakehouse, this release kicks off the year of DuckDB as a server. We previewed many of these features in the “State of the Duck” talk at DuckCon #7, if you prefer to watch instead of read.

DuckDB is moving rather quickly, and we can only cover a small fraction of the changes here. Condensing all new features down to a shortlist is always a fight over what gets in, and yes, we know that what follows is technically a listicle (Ten Things Coming to DuckDB v2.0, Number Eight Will Shock You). We are not proud of the format, but it works, so here it is, starting with the SQL-level features and working down into the engine.

1. DuckDB as a Server: Quack and CONNECT

DuckDB has been an in-process database since day one. But people have asked us – very persistently – for a client/server mode, and we have finally caved. The quack extension implements DuckDB's native protocol for talking to other DuckDBs. It was released as a preview shortly before DuckCon #7, graduates to stable in v2.0, and it is a big part of where DuckDB is headed: any DuckDB process can serve its databases over the network, and any other DuckDB can attach to it and route queries there using the new CONNECT statement. For example:

CALL quack_serve(
    token = 'my_token'
);

ATTACH 'quack:server.example.com'
    AS qk (TOKEN 'my_token');

CONNECT qk;
SELECT count(*) FROM events;
-- executes on the server,
-- results stream back
DISCONNECT;

CONNECT is the successor to the remote.query($$...$$) workaround we showed when Quack was first revealed – we looked at that syntax and said: no, this cannot be it. And CONNECT is not limited to Quack: it points your session at any remote database that supports it, and the new remote pushdown optimizer (#22914) ships SQL directly to PostgreSQL and MySQL instead of pulling tables over the wire:

CONNECT 'postgres://localhost/mydb';
SELECT count(*) FROM orders; -- runs on the PostgreSQL server
DISCONNECT;

If you have worked with analytical systems in the past, you may assume that DuckDB cannot handle transactional workloads. But DuckDB has been built as a transactional, multi-connection database with full MVCC and transaction isolation since day one. Most users just never needed that in a single-user scenario. It turns out DuckDB handles transactions well: it's fast enough to compete with general-purpose databases like PostgreSQL on quite a few workloads, and the client/server pattern finally lets that machinery shine in multi-tenant, long-running deployments.

Running DuckDB long-term also comes with new challenges, which is why v2.0 pushes on better metrics, logs, and observability (see, e.g., the metrics layer rework in #22799) that let you look at a DuckDB instance and see what it is actually doing. People even built standalone clients for the Quack protocol within weeks of the preview. We thought we were extending DuckDB to talk to other DuckDBs; the world said no, no, no, and built their own clients. Who would have thought.

2. VARIANT Becomes a First-Class Citizen

The VARIANT type shipped in DuckDB v1.5, and the way to think about it is JSON on steroids. Basically, imagine if JSON were fast. Like JSON, a VARIANT column can store differently-shaped data in every row. Unlike JSON, it is not a text format: DuckDB automatically detects the common structure hidden in your semi-structured data and “shreds” it, so it compresses well in storage and executes fast in queries, all without you ever declaring a schema. This makes VARIANT a natural fit for real-time log ingestion, where streams of JSON-ish records share structure but evolve over time.

In v2.0, this pipeline works end to end: shredded execution straight from storage (#20912), extraction pushdown into scans (#22478), shredded VARIANT reading and writing for Parquet, and a family of variant_* functions:

CREATE TABLE events (payload VARIANT);
INSERT INTO events
VALUES ('{"user": {"id": 42, "tags": ["a", "b"]}}'::JSON::VARIANT);

SELECT variant_type(payload), variant_keys(payload)
FROM events;

SELECT *
FROM events
WHERE variant_contains(payload, {'user': {'id': 42}}::VARIANT);

Longer term, likely soon after v2.0 (but don't hold us to it), we plan to back the regular JSON type with VARIANT, so existing JSON workloads get all of these benefits without changing a single query.

3. Triggers

Triggers have been a long-standing feature request, and DuckDB v2.0 delivers them in full: BEFORE and AFTER triggers, FOR EACH ROW and FOR EACH STATEMENT, transition tables via REFERENCING OLD/NEW TABLE, multiple triggers per event, RETURNING on triggered tables, and DROP TRIGGER.

The classic use case is audit tables: something happens in the system, and a trigger records what changed. For example:

CREATE TABLE target (id INTEGER, val INTEGER);
CREATE TABLE audit (id INTEGER, old_val INTEGER, new_val INTEGER);

CREATE TRIGGER trg_audit AFTER UPDATE ON target
REFERENCING OLD TABLE AS o NEW TABLE AS n
FOR EACH STATEMENT
    INSERT INTO audit
    SELECT n.id, o.val, n.val
    FROM o
    JOIN n ON o.id = n.id;

INSERT INTO target VALUES (1, 10), (2, 20);
UPDATE target SET val = val * 10 WHERE id <= 2;
SELECT * FROM audit;

Triggers fit naturally with long-running DuckDB services, and we are also planning to use them internally to build several upcoming features. They are fully exposed at the SQL level too, so you can build your own cool stuff with them.

4. SQL Dialect Additions

As always, DuckDB's SQL dialect keeps growing. A few favorites from this release cycle:

With NEAREST joins (#24137), top-k similarity search becomes a join clause, handy for vector and embedding workloads:

SELECT q.user_id, t.product_id
FROM users q
    INNER JOIN products t APPROX NEAREST 2
    BY SIMILARITY array_cosine_similarity(q.embedding, t.embedding);

DML inside CTEs (#21634, #21997, #24217) lets you use INSERT, UPDATE, DELETE, and COPY as pipeline steps:

WITH moved AS MATERIALIZED (
    DELETE FROM staging RETURNING *
)
INSERT INTO archive SELECT * FROM moved;

Nested schemas (#23492, #24222) allow schemas within schemas:

CREATE SCHEMA finance;
CREATE SCHEMA finance.reports;
CREATE TABLE finance.reports.q3 (revenue DECIMAL);

The new variable syntax (#21194) lets you write $x anywhere an expression is allowed, no more getvariable(...) verbiage:

SET VARIABLE threshold = 100;
SELECT * FROM orders WHERE amount > $threshold;

The JSON mutation functions json_set, json_insert, json_replace, and json_remove (#23786) finally let you modify JSON documents in place:

SELECT json_set('{"a":1}', '$.b', '2');

And recursive CTEs with USING KEY aggregation (#19481) enable iterative algorithms in pure SQL, backed by the rewritten recursive CTE engine described below:

WITH RECURSIVE tbl(a, b) USING KEY (a, avg(b)) AS (
    SELECT 1, 5
    UNION ALL
    SELECT a, b - 1 FROM tbl WHERE b > 0
)
TABLE tbl;

There is more: SQL-standard FETCH FIRST 2 ROWS ONLY (#23533), OVERLAY() (#22456), UNNEST in GROUP BY (#23644), and well-defined MERGE / UPDATE ... FROM semantics for multi-matched rows (#24058).

5. Asynchronous I/O

Interacting with object stores like S3 is central to the DuckDB experience: your data has to come from somewhere, and it often sits in object storage. DuckDB has long been able to read from object stores in parallel, but synchronous access placed a limit on how fast this could go. DuckDB v2.0 introduces asynchronous I/O throughout the engine. We described the design in detail in a dedicated blog post.

Thanks to asynchronous access, the I/O layer now scales independently from the query processing layer, which means far more parallelism for remote reads and dramatically faster queries on network storage. Parquet support came first (#23662), with CSV (#23961) and DuckDB's own file format (#24654) following, along with asynchronous Parquet writes (#23283) and new MMAP and DIRECT_IO modes (#22988). Local storage benefits a little too, but network storage is where you will see the big gains.

6. Faster Queries Across the Board

As with every release, a lot of work went into making your existing queries faster without you doing anything. To pick some highlights: partial aggregates are now pushed below joins (#22572) and redundant aggregations are reused (#24543), the recursive CTE engine has been rewritten (#22211), aggregations now spill to disk when they outgrow memory (#24499), and the Windows CLI got approximately 2.2× faster at multi-threaded result materialization (#24036).

How much faster can this get? Here is a microbenchmark you can run on a laptop: single-source reachability over a graph with one million edges, written as a plain recursive CTE.

CREATE TABLE edges AS
    SELECT (range % 100_000)::INTEGER AS src,
           ((range * 13 + 7) % 100_000)::INTEGER AS dst
    FROM range(1_000_000);

WITH RECURSIVE reachable(node) AS (
    SELECT 0
    UNION
    SELECT dst FROM edges, reachable WHERE src = node
)
SELECT count(*) FROM reachable;

As you can see, DuckDB v2.0 is about 40× faster (!) for the same recursive query.

Update We published a blog post on how we are speeding up recursive CTEs in DuckDB v2.0.

Row-group pruning has been massively expanded: min-max indexes (zone maps) and Parquet Bloom filters now skip data for structs, lists, decimals, UUIDs, IN filters, and even function predicates:

-- these now prune row groups instead of scanning them:
SELECT * FROM logs WHERE contains(message, 'ERROR');
SELECT * FROM t WHERE substr(code, 1, 3) = 'NL-';
SELECT * FROM 'data/*.parquet' WHERE id IN (1, 5, 9);

Query planning also becomes partition-aware (#22336). Lakehouse formats (DuckLake, Iceberg and plain Hive-partitioned Parquet on S3) are all partitioned, and exploiting that partitioning is often the difference between scanning a dataset and skipping most of it. In v2.0, the planner and optimizer take full advantage of existing partitioning, and partitioned writes have been reworked as well (#22225, #22620).

7. Storage Format v2.0

DuckDB v2.0 bumps the default storage format version to v2.0.0 (#22875).

Column metadata is now loaded lazily (#22333), so wide tables open faster too. The DICT_FSST string compression method is enabled by default (#23733), deletes are stored compactly (#24336), and the storage layer performs much stronger corruption validation on read. In short: databases with big indexes and wide tables open faster and use far less memory. The new storage format also allows checkpoint vacuuming to compact tables with ART indexes by incrementally remapping entries whose row IDs change, avoiding a full index rebuild (#23653).

Later this year, ART indexes will be buffer-managed (#21458, #23605). This will allow ART index buffers to be evicted under memory pressure, and remove the restriction on the ART index fitting completely within memory.

8. A Brand New SQL Parser

DuckDB has famously always used a parser derived from PostgreSQL's. We have decided that enough is enough: v2.0 ships our own modern, extensible PEG-based parser (#22194), an idea we first explored in our 2024 post on runtime-extensible parsers. This change ties into the extension ecosystem: extensions can now hook into the grammar itself, so expect extensions that expose entirely new SQL syntax. It also brings better error messages with precise source locations.

You should not actually notice anything from the parser swap as we designed it to be compatible with the old one. If you do notice, please file an issue.

Update We published a new blog post on the PEG parser.

9. Timezones, Calendars, and Collations Without ICU

Timezone-aware timestamps, calendars, and collations in DuckDB have always been powered by the ICU library. ICU is a fine library, but we only ever used a small slice of it, while still carrying it around in every DuckDB distribution. In v2.0, the ICU library is gone entirely: the icu extension now implements timezones, calendars, and collations itself (#24463, #24403), with the timezone data built directly from the IANA database and compressed down to around 45 kB. Everything keeps working exactly as before:

SELECT '2026-08-14 12:00:00'::TIMESTAMPTZ AT TIME ZONE 'Europe/Paris';
SELECT * FROM names ORDER BY name COLLATE de;

Besides being much smaller and easier to keep up to date, the new implementation is also simply faster. Here's a quick microbenchmark on a MacBook that converts 25 million timestamps to a timezone and filters 5 million strings with a German collation:

10. Write Extensions Once, Host Them Yourself

Extensions are one of the best things about DuckDB, but today, most of them, including our own, build against the unstable C++ API. Extension authors are required to rebuild and publish their extensions for every DuckDB release, even if the extension itself does not change. This is often a non-trivial operation for the extension author and a non-trivial coordination effort for the DuckDB team. Not rebuilding an extension means it can't be installed on the latest version of DuckDB.

Developing Extensions

DuckDB v2.0 will ship with a revamped C API (see #24702 for part 1). The API will have a versioned specification expressed in YAML that uses an also versioned specification schema, and tooling for code generation (#24135). The schema lets us tag every symbol with its lifecycle and stability guarantees (#24435). A large part of the API will be marked stable and frozen, providing a stable ABI across DuckDB versions.

The API itself improves over the pre-2.0 C API in a number of ways. It provides coherent error handling and a unified set of conventions around naming and ownership. It covers a much larger feature surface, including scalar, aggregate, table, cast, and copy functions with named parameters and varargs, parsing and inspecting SQL statements, prepared statements, replacement scans, custom filesystems, direct access to vector buffers, values and types, and more. It supports streaming query result consumption. With this API we want to provide the primitives to access DuckDB's most powerful features.

You won't need to program against the C API directly, though. We also provide a C++ API on top of it. It is a thin layer that compiles into your extension and talks only to the stable C ABI, so the binary you ship stays independent of DuckDB versions. It gives you a convenient, typed, well-documented way to access DuckDB's API. We're also working on bindings that provide the same for extension writers using Rust.

So what will an extension using the C++ API look like? Here is a complete example, though the details may still change: a single file that registers a vectorized scalar function.

#include "duckdb_cpp.hpp"

using namespace duckdb_api;

DUCKDB_CPP_EXTENSION_ENTRYPOINT(extension) {
    // add_numbers(a BIGINT, b BIGINT DEFAULT 2): adds two BIGINTs, one vector at a time
    ScalarFunction function;
    function.SetName("add_numbers")
        .SetSignature(FunctionSignature::Create()
                        .AddParameter("a", LogicalType::BIGINT())
                        .AddParameterDefault("b", LogicalType::BIGINT(), Value::Bigint(2))
                        .SetReturnType(LogicalType::BIGINT()))
        .SetExecCallback([](ScalarFunction::ExecInput &input) {
            auto chunk = input.GetInputChunk();
            auto a = chunk.GetVector(0).GetView();
            auto b = chunk.GetVector(1).GetView();
            auto out = input.GetResultVector().GetDataMutable<int64_t>();
            for (idx_t i = 0; i < chunk.GetRowCount(); i++) {
                out[i] = a.Data<int64_t>()[a.SelAt(i)] + b.Data<int64_t>()[b.SelAt(i)];
            }
        })
        .Register(extension);
}

You can use it as follows:

LOAD add_numbers;
SELECT add_numbers(40, 2);           -- 42
SELECT add_numbers(40);              -- 42: b falls back to its default
SELECT add_numbers(b := 2, a := 40); -- 42: named arguments work too
SELECT add_numbers(40, NULL);        -- NULL, without the function doing anything

You do not need re-target or rebuild it every time a new DuckDB version comes out. And nowadays, with all the AI tooling around, building an extension has never been easier.

Distributing Extensions

So you have written your extension. But how should you distribute it? Until now, DuckDB could only install extensions from the built-in repositories (core, core_nightly, community, …). In v2.0, you will be able to register your own trusted repositories (#24777, currently work-in-progress), so an organization can host and sign its own extensions and have them install and load just like the built-in ones:

SET allow_extension_repositories = 'allowed';

CREATE EXTENSION REPOSITORY my_repo
    WITH PREFIX 'https://extensions.example.org';

INSTALL my_ext FROM my_repo;
LOAD my_repo/my_ext;

A repository is a name, a URL prefix, and one or more RSA public keys that are trusted to sign the extensions served from it. The prefix can point at anything DuckDB can read: a local path, https, s3, you name it. At CREATE time, DuckDB fetches the repository's public keys and pins them into the repository definition, printing each key's SHA-256 fingerprint so you can compare it against one published out of band. If you would rather not trust the network at all, you can pass the key directly:

CREATE EXTENSION REPOSITORY my_repo
    WITH PREFIX 'https://extensions.example.org';
    USING PUBLIC KEY '-----BEGIN PUBLIC KEY----- ...';

Pinned repositories survive restarts, support key rotation by trusting multiple keys, and can be audited at any time through the duckdb_extension_repositories() table function, or removed again with DROP EXTENSION REPOSITORY. Together with the stable C API, the extension story rounds out nicely: write your extension once, sign it, host it wherever you like, and INSTALL it anywhere.

Update This feature is now available in the preview build.

Bonus: DuckDB Foundation – Advisory Board

Starting this fall, we will add a stakeholder advisory board to the DuckDB Foundation. The advisory board will provide input on the development roadmap of DuckDB, DuckLake, and Quack. This allows key stakeholders to have a say in the projects' direction.

Final Thoughts

These are only a few highlights, and this post is only a preview. Some details may still shift before the release this fall, and there are many more features and improvements that we could not cover here. DuckDB v2.0 will also come with a small set of breaking changes, including the new default storage format and the completed lambda syntax transition, which we will cover in detail in the release announcement.

There have been more than 10,000 commits by many contributors since we released v1.5. We would like to thank our community for the detailed issue reports, feedback, and contributions that shaped this release. If you want a taste before the fall, the preview builds have most of these features today, and if something breaks, you know where the issue tracker is.

We’d like to use this milestone to stop for a moment and revisit recent developments in the “Duck Stack” ecosystem since last summer (2025), when we surpassed 30 000 stars.

Releases

• Since the last milestone, the DuckLabs team has released DuckDB 1.4.0, our first long-term support release, and DuckDB 1.5.0. Both releases are packed with many new features, performance optimizations, and bugfixes.
• The DuckLabs team also published the DuckLake 1.0 standard, our production-ready format for SQL-as-a-lakehouse.
• Finally, we released the Quack remote protocol, which lets you run DuckDB in a client-server setup with multiple concurrent writers. That’s right: DuckDB instances can now talk to each other!

Community Metrics

Besides the GitHub stars, the DuckDB community has also experienced a lot of growth in other metrics:

• Each month, duckdb.org receives traffic from over 8 million unique visitors – more than double last summer’s numbers. We also see over 2 PB (!) in traffic from millions of extension downloads. Thanks again to Cloudflare for sponsoring the project.
• DuckDB now has 50M+ monthly downloads in PyPI, more than double the 20M we reported last time.

(As usual, we’d like to emphasize that while we’re happy to see these metrics grow, we are not glorifying them and they are not a target per se in accordance with Goodhart’s law.)

Events

Since 30 000 stars, we ran DuckDB meetups in Amsterdam, Berlin, and London. We also hosted a Developer Meeting focused on DuckDB internals and extension development.

Finally, we hosted DuckCon #7, our largest DuckCon so far, which we also streamed online. These events allowed us to connect with our community of engineers, builders, and scientists, and learn about the many ways they are using DuckDB.

Next up, we'll host the DuckDB Paris meetup with Altertable and we are setting up a meetup in Boston. Stay tuned!

DuckDB in the Wild

Here are just a few of the projects, extensions, and announcements that happened in the DuckDB community since the last 10 000-star milestone:

• Blog: Petrica Leuca shows you how to use DuckDB for keyword, full-text, and semantic similarity search with embeddings for lightweight text analytics.
• Video: GizmoEdge: A distributed DuckDB engine for IoT (from the first DuckDB Developer Meeting) by Philip Moore.
• Blog: DuckDB for streaming analytics? Yes, says Guillermo Sanchez, you can use DuckDB to refresh your data at near real-time speed.
• Use Case Walkthrough: Uncovering Financial Crime with DuckDB and SQL/PGQ graph syntax that's part of SQL:2023 by Daniël ten Wolde.
• For Fun: DuckDB co-creator Hannes Mühleisen shows you how to store and even process video in DuckDB.
• Blog: Iceberg in the Browser: learn how to read and write tables in Iceberg catalogs without needing to manage any infrastructure – directly from your browser!
• Presentation: Denis Hirn from University of Tübingen introduces you to DuckPL: A procedural language in DuckDB.
• Hardware benchmarks: Hannes Mühleisen tests DuckDB on a Loongson CPU and Gábor Szárnyas put DuckDB to work on a MacBook Neo.
• Video: Barry Smart walks you through 20 years of wind data while auditing UK energy policy without a cluster (DuckCon).
• Blog: Interested in building DuckDB extensions in C#? Giorgi Dalakishvili shows you how to build DuckDB extensions in C# with DuckDB.ExtensionKit.
• Talk: Kian Mehrabani takes you on a tour of how Spotify is using DuckDB as a SQL layer over user listening history in this video from DuckCon.
• Preview: In DuckDB v2.0, scheduled for fall 2026, DuckDB will support async reads of Parquet and CSV files. Pedro Holanda walks you through the details of asynchronous I/O.

Closing Thoughts

We would like to extend a heartfelt thank you to the DuckDB community of users and contributors. We look forward to growing the community around the “DuckStack” and hope the GitHub stars follow. We'll report back when we reach 50 000 stars.

This worked particularly well because DuckDB primarily ran locally, with its main use case being as a quick-draw database engine for querying data directly from your machine's SSD. We could split the data into several partitions, such as row groups for Parquet files or fixed-size buffers for CSV files, and load them with low latency and high bandwidth. As a result, the main bottlenecks were elsewhere: subqueries, joins, aggregations, and so on. The actual data access path received less attention because synchronous access was perfectly suitable for this use case.

As usual, things changed. We realized that DuckDB's architecture was a great fit for querying remotely stored large-scale datasets, such as data lakes (e.g., DuckLake). Since May this year, we can even run DuckDB as a server using the Quack protocol. The original expectation of data files sitting on a local SSD therefore no longer always holds.

The practical implication of these changes is that many current DuckDB setups need to transfer files from remote storage to the machine that will actually process them. For data lakes, for example, a typical setup is to store the data in blob storage, such as S3, and process it on an EC2 machine in the same region. In this setup, latency and bandwidth play a much more significant role. If we cannot issue enough concurrent requests to use the available network bandwidth, performance can suffer drastically, with threads spending a large amount of their time waiting for remote reads instead of processing data.

As an example, let's consider a simple query over a remote Parquet file. For simplicity, let's assume we only have a single thread executing.

FROM read_parquet('s3://bucket/file.parquet');

A Parquet scan is partitioned into row-group-based jobs, with each job containing one or more fetch tasks that issue byte-range requests. With synchronous I/O, the worker thread will be blocked, waiting for the data to arrive at the machine before performing actual work, such as decoding, aggregating, and so on. You can see a visual depiction in the figure below, where the thread is blocked from doing any work while it waits for the read to finish.

Synchronous read

To address this, we have been implementing asynchronous I/O pipelines in DuckDB. They are currently implemented for Parquet and for uncompressed, seekable UTF-8 CSV files, with support for other formats, such as DuckDB's native format and JSON, still to come. In the remainder of this blog post, we will give a simple explanation of how asynchronous I/O is implemented in DuckDB and provide benchmarks for both Parquet and CSV files.

If you would like to try asynchronous I/O now, you can do so by using DuckDB's v2.0.0-dev preview builds. Asynchronous I/O will be used by default from the next major DuckDB version, v2.0, released in the fall.

Asynchronous I/O

The conceptual idea of asynchronous I/O is rather simple: we should be able to start an I/O operation without blocking the worker thread that requested it. Applied to our Parquet example, the same picture would look like the following:

Asynchronous read

In this example, we have two ASYNC threads and one regular worker thread. The ASYNC threads keep fetch tasks in flight while the worker thread decodes data. During the initial warm-up, the scan task parks, leaving the worker thread free to run other pipeline tasks. Once the first job is ready, fetching and decoding can overlap.

In DuckDB, we implemented something similar. We have two separate thread pools:

• REGULAR – This pool contains our worker threads (by default: one for each available CPU thread). These are the ones that do real work, like decoding, joins, and aggregations. They prioritize regular work but can also perform I/O tasks when idle.
• ASYNC – A pool of threads intended for asynchronous tasks, primarily blocking I/O.

The main reason we have these two different pools is that, for remote I/O, these threads can spend almost all their time blocked, waiting for an HTTP response, for example, and hence have very little CPU utilization. Because of that, we have many more ASYNC workers than system threads, with the default setting being 4 * system threads and the total being capped at 256.

It's of utmost importance to keep as many of our ASYNC threads busy as possible. To ensure that, we implement a read-ahead strategy instead of issuing reads on demand. This means scheduling fetch tasks ahead of what our regular worker threads currently need.

One thing we need to be attentive to is that read-ahead buys throughput by holding memory. If decoding is slow and the network is fast, prefetched data can accumulate and lead to out-of-memory issues. To mitigate this, we also implemented asynchronous memory governance. Both read-ahead and memory governance will be explained in more detail in the following sections.

Read-Ahead Queue

The idea of read-ahead is also straightforward. Instead of starting a read at the exact moment a regular worker needs the data, we schedule fetch tasks for work that lies further ahead. While a regular worker decodes the current job, the ASYNC threads are already pulling in data for the next jobs. The goal is to keep enough fetch tasks in flight to hide the latency of remote storage.

The jobs are units of work that can be scheduled and processed independently, and they can be different depending on the underlying file format. For a Parquet file, a job is one row group of one file. For a CSV file, a job is a scan boundary that generally covers a fixed byte range within the file.

A Parquet job might be broken down into multiple fetch tasks depending on the query projections, filter pushdowns, physical column locations, and which nearby byte ranges can be combined. The two fetch tasks in the figure below are illustrative, as their exact grouping and sizes depend on the file and query.

For CSV files, we don't have the same granularity of information as we do for Parquet files. A job's fetch tasks load its starting buffer if it is not already in memory and, when the scan boundary reaches the end of that buffer, the following buffer as well (e.g., to handle lines that are split across two buffers).

Jobs

Filling the queue requires no dedicated producer thread. Any regular worker that comes looking for scan work first tops up the queue as far as it is allowed to. The limit is either given by a user-specified number of slots or by a memory budget. If there is space, a job and its fetch tasks are created. The fetch tasks are scheduled immediately on the ASYNC pool, while the job is admitted to the read-ahead queue in batch order.

ASYNC threads execute individual fetch tasks independently of the job queue's claim order. Fetch tasks from the same job can run concurrently, although no particular assignment to ASYNC threads is guaranteed. All fetch tasks of a job share a countdown, and the fetch task that brings it to zero completes the job's I/O.

A worker thread claims the oldest job in the queue and checks that countdown. If I/O is done, the worker starts decoding the job. If not, it parks the scan task and is free to run other pipeline tasks. The last fetch task then unblocks the scan task, which may resume on any regular worker.

Claiming the job also immediately frees a queue slot, allowing any regular worker looking for scan work to produce a replacement job at the back of the queue. The figure below depicts this cycle:

Read-ahead cycle

Memory Management

Keeping more fetch tasks in flight consumes more memory. To determine a budget and avoid out-of-memory issues, we introduced the read_ahead_depth configuration option. It can have three types of values:

• -1 (default): unlimited depth, bounded by memory.
• N > 0: at most N jobs ahead, with no memory budget.
• 0: read-ahead is off, each scan task schedules I/O only for its own job.

To configure it, use the SET clause, e.g.:

SET read_ahead_depth = 5;

In the default mode, the budget is negotiated with the temporary memory manager, which is the same manager that splits memory between concurrent joins, sorts, and window operators. When there is a lot of memory pressure, for example, because an operator is using a large amount of memory, queue reservations might instantly be over budget. In practice, this means that the queue will only allow one job at a time, and the scan will behave close to a synchronous scan.

When the memory-heavy operator finishes, the memory manager has more budget to give, and the queue fills back up.

Benchmarks

Asynchronous I/O should have the largest effect when the latency of synchronous requests prevents us from using the available remote bandwidth. To measure this effect, we ran TPC-H Query 6 at SF100, with the data sitting on S3, and compared the results against DuckDB v1.5.5, our latest stable release. The SF100 dataset was written as a single file per table for both the Parquet and CSV benchmarks, with the lineitem table containing 600,037,902 rows.

For compute, we used an EC2 r7i.16xlarge machine (64 vCPUs and 512 GB of RAM), with both the machine and the S3 bucket with the data located in the same region. We executed the query five times and report the mean execution time. The files were never cached (i.e., SET enable_external_file_cache = false;), meaning that every execution read the data straight from S3.

Parquet

The Parquet file is approximately 22 GB and has around 4,880 row groups, with each row group containing approximately 122,880 rows. With asynchronous I/O, the mean runtime drops from 8.230 seconds to 2.844 seconds, making the query almost 3× faster.

Below we also show the network throughput over the course of the query:

Network throughput

In it, we run DuckDB v1.5.5 and two variations of DuckDB v2.0.0-dev. One with the read-ahead depth determined by the memory governor, and one tuned for this machine, where we cap the read-ahead at 64 in-flight jobs and adjust the I/O settings (SET async_threads = 48; SET http_retries = 8; SET http_retry_wait_ms = 50; SET http_retry_backoff = 2). We can see that v2.0.0-dev uses the available bandwidth much more effectively, approaching the network limit and reaching it at several points. The tuned version goes further. With fewer, hotter connections and cheap retries, the throughput variance drops to a minimum and the 25 Gbit/s network stays almost fully saturated. Its query time was 2.227 seconds, reducing the runtime of the untuned v2.0.0-dev run by 21.7% and making it about 3.7× faster than DuckDB v1.5.5. In comparison, v1.5.5 stays around 5 Gbit/s because its synchronous reads do not keep enough requests in flight to saturate the network.

One other detail worth noting is that, in all experiments, a few hundred milliseconds pass before the first bump in network traffic, followed by another few hundred milliseconds before the main data transfer begins. The first gap is the time needed to open a DuckDB connection, perform the first TLS handshake, and open the file. The bump corresponds to downloading the file footer, while the second gap comes from processing the information in the footer before executing the query. We believe this is an area we can investigate and optimize further before the v2.0 release.

We sampled the NIC's received-byte counter every 50 ms and calculated the throughput from the change in bytes between samples. We independently confirmed that the machine can access the network at 25 Gbit/s with both a DuckDB full-file read and the s5cmd tool.

Local Disk

Remote storage is the main target for asynchronous I/O, but cold local reads give us a useful contrast. To measure them, we ran TPC-H Query 6 over the SF100 Parquet file, this time with the file sitting on the local disk of a MacBook Pro (Apple M4 Max, 14 cores, and 36 GB of RAM). Since the benefit of asynchronous I/O on local disks comes from cold reads, we cleared the OS caches (with the macOS purge command) between runs, making sure every execution actually read the file from disk.

We can see that, for cold runs, asynchronous I/O is approximately 1.5× faster, reducing runtime by about 33%. The performance difference is much smaller than in the cases presented above, due to the SSD having much lower latency and much higher bandwidth than the EC2/S3 network. For hot runs, the difference is negligible, as there is no disk access happening if data is properly cached.

Small Files

Partitioned datasets are a particularly relevant use case here, as partitioning can easily spread the data across many small files. To see how asynchronous I/O behaves in this setup, we also performed a Parquet run using the same TPC-H SF100 dataset. Instead of using one file, we generated 976 files with five row groups each. Each file contains approximately 615,000 rows and is around 22 MB in size.

We can see that v2.0.0-dev delivers a similar performance improvement here as it does for the single-file benchmark, running around 3× faster. This shows that read-ahead can also parallelize across multiple files without becoming bottlenecked by opening files or fetching their footers.

Large Row Groups

We also wanted to see what happens at the other extreme, when a Parquet file has only a few very large row groups. For this experiment, we generated six versions of the same TPC-H SF100 lineitem table as a single file, changing only the requested row group (RG) size, and ran Q6 using DuckDB v2.0.0-dev. The table below reports the runtime for each version.

At first, larger row groups decrease query times. As we increase the row-group size, request latency is amortized over much larger transfers. However, beyond a certain point, the available parallelism starts to fall. A row group is DuckDB's unit of Parquet scan parallelism, so ideally a scan should expose at least one row group per system thread. On this 64-vCPU machine, the version with 64 row groups provides exactly that and finishes in 2.27 seconds, while the fastest run comes from the version with 306 row groups, at 2.11 seconds.

However, with fewer row groups than threads, we lose parallelism and can no longer saturate the network. For Q6, the projections and the physical location of the columns result in two fetch requests per row group. Four row groups therefore expose only about eight concurrent S3 streams, raising the runtime to 8.01 seconds. For the largest configuration, the file contains a single row group, and its I/O is effectively reduced to two giant streams, pushing the runtime to 25.26 seconds. This happens even though better compression makes the file a little over half the size of the version with 4,886 row groups. In this case, the extra bandwidth required by smaller row groups is cheaper than the parallelism lost with extremely large ones.

Concurrent Queries

The effect becomes even clearer when several queries run at the same time. For this experiment, we ran TPC-H queries 1, 6, 9, and 18 concurrently against the same SF100 Parquet dataset on S3, using a single DuckDB instance. We picked these queries because they cover a mix of scans, aggregations, and joins, with different CPU and memory requirements. We repeated the experiment with the default memory configuration and with memory limits of 16 GB and 8 GB. The total runtime is the wall-clock time until all four queries finish. We report the average and peak CPU utilization (number of cores utilized), the peak bandwidth (bw.) and the peak resident set size (RSS).

With the default memory configuration, DuckDB v1.5.5 keeps an average of only about 6 of the 64 cores busy. In other words, around 90% of the machine sits idle waiting for synchronous S3 reads. DuckDB v2.0.0-dev, on the other hand, averages 48 busy cores, reaches all 64 at its peak, and saturates the 25 Gbit/s network. As a result, all four queries finish in less than half the time.

The memory results are also interesting. As we lower the limit, the memory governor reduces the read-ahead backlog, while memory-heavy operators such as those in Q18 can spill to disk. This lowers the peak physical memory used by the DuckDB process (i.e., RSS) of DuckDB v2.0.0-dev from 20.1 GB with the default configuration to 15.7 GB with a 16 GB limit and 11.5 GB with an 8 GB limit. The additional spilling and reduced read-ahead also lower average CPU utilization and increase the runtime, but v2.0.0-dev continues to saturate the network and remains substantially faster than v1.5.5 in both cases.

One might notice that the 8 GB result still peaks at 11.5 GB of RSS. This is because jemalloc keeps recently freed pages resident for about one second so they can be reused. This memory is no longer counted by DuckDB's memory manager, and v1.5.5 shows the same allocator behavior.

CSV

The effect is larger on CSV files. The CSV file is 80.89 GB, and asynchronous I/O reduces the mean runtime from 878 seconds to just 45 seconds, making the query almost 20× faster. CSV is row-oriented, so the scan transfers substantially more data and performs fixed-size buffer reads, making concurrent remote reads especially valuable.

As in the other experiments, we used the default memory-governed read-ahead depth, so this run was not tuned to keep the 25 Gbit/s network saturated on average.

Conclusion

In this blog post, we presented the recent work on asynchronous I/O for Parquet and CSV files. Most of its benefit comes from accessing remote data, but cold local reads can also benefit, albeit less. Next, we plan to add async reads for JSON and DuckDB-native files, as these are the two other formats most relevant to DuckDB core. Formats that live in out-of-tree extensions are not on the roadmap yet. We will also investigate io_uring, Linux's asynchronous I/O interface, which could reduce system-call overhead and the number of threads blocked on I/O. If it proves beneficial in practice, we will integrate it into DuckDB. One important thing to notice is that any of the data lake solutions supported in DuckDB can already benefit from asynchronous I/O automatically as long as the underlying data format is Parquet (or CSV, if you are brave enough).