DuckDB Async I/O Engine: The Secret Behind 1.7x Parquet Read Performance

In June 2026, the DuckDB core team merged a remarkable PR #23142 — introducing asynchronous I/O support for the Parquet reader, delivering up to 1.68x performance improvement. This article provides an in-depth analysis of how this technology works, benchmark results, and how you can leverage these improvements in your data pipelines.

Why I/O Becomes the Bottleneck

In modern data analytics, Parquet has become the de facto columnar storage format. Its high compression ratio, predicate pushdown support, and suitability for large-scale data analysis make it indispensable. But when data reaches tens or hundreds of gigabytes, the traditional synchronous I/O model encounters a hidden performance bottleneck:

The CPU sits idle waiting for disk or network I/O, while the disk/network waits for the CPU to issue read commands.

This “wait-execute-wait” pattern becomes especially severe in cloud environments (S3, GCS, OSS), where network latency far exceeds local SSD access latency.

How DuckDB’s Async I/O Engine Works

Limitations of Synchronous I/O

In DuckDB’s previous architecture, the Parquet reader worked synchronously:

1. The main thread requests to read a Parquet file
2. The main thread waits for data to arrive from disk/network
3. Once data arrives, the main thread parses and filters it
4. Repeat

This means every I/O operation blocks the entire Worker thread, even when the CPU has no work to do at that moment.

New Architecture with Async I/O

DuckDB’s async I/O engine redesigns the entire read workflow:

┌─────────────────────────────────────────────────────┐
│ DuckDB Query Engine │
│ │
│ ┌──────────┐ ┌──────────────┐ ┌──────────┐ │
│ │ Worker │───▶│ Task │───▶│ Result │ │
│ │ Thread │ │ Scheduler │ │ Buffer │ │
│ │ (Compute)│◀───│ (Schedule) │ │ (Result) │ │
│ └──────────┘ └──────┬───────┘ └──────────┘ │
│ │ │
│ ┌──────▼───────┐ │
│ │ Async I/O │ │
│ │ Thread Pool │ │
│ │ (Disk/Net) │ │
│ └──────────────┘ │
└─────────────────────────────────────────────────────┘

The core innovation lies in introducing a dedicated Asynchronous I/O Thread Pool:

• Task Scheduler: The main thread schedules I/O tasks to the async thread pool, then immediately returns a BLOCKED state
• Async I/O Pool: A dedicated thread pool handles all disk/network read operations, without blocking compute threads
• Multi-strategy Support: Supports multiple prefetch strategies including WHOLE_GROUP, COLUMN_WISE_EAGER, and PREFETCH_FILTERS

Three Prefetch Strategies Explained

Benchmark Results

Let’s analyze the benchmark data provided by DuckDB’s official team. Test environment: AWS c6in.4xlarge instance, reading from S3 in the same region — 8.2 GB, 16 Parquet files, 640 row groups, 17 columns, 64 million rows.

Test 1: Full Column Scan

-- Test full column scan performance
INSTALL httpfs;
LOAD httpfs;

SELECT COUNT(*) AS total_rows
FROM read_parquet('s3://bucket/large_dataset/*.parquet');

-- Full column scan with filter condition
SELECT category, AVG(price) AS avg_price
FROM read_parquet('s3://bucket/large_dataset/*.parquet')
WHERE region = 'us-east-1'
GROUP BY category
ORDER BY avg_price DESC;

Key Findings

1. 32-thread async I/O achieves the biggest gain in full column scans (1.68×), because I/O bandwidth utilization is highest in this scenario
2. High-selectivity filtering scenarios see the most significant improvement (1.73×), because the PREFETCH_FILTERS strategy maximally reduces redundant I/O
3. Aggregation queries benefit from improved CPU cache efficiency due to reduced I/O

Test 2: Real-World Scenario Replication

Let’s replicate an equivalent local benchmark using DuckDB’s built-in capabilities:

import duckdb
import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd
import numpy as np

# Generate test data
np.random.seed(42)
n_rows = 10_000_000

data = {
 'id': range(n_rows),
 'category': np.random.choice(['A','B','C','D','E'], n_rows),
 'region': np.random.choice(['us-east-1','us-west-2','eu-west-1','ap-south-1'], n_rows),
 'timestamp': pd.date_range('2024-01-01', periods=n_rows, freq='s'),
}

# Add numeric columns
for i in range(10):
 data[f'value_{i}'] = np.random.randn(n_rows)

# Save as Parquet
df = pd.DataFrame(data)
table = pa.Table.from_pandas(df)
pq.write_to_dataset(table, '/tmp/test_dataset', partition_cols=['category', 'region'])

# Test synchronous read
con = duckdb.connect(':memory:')
result_sync = con.execute("""
 SELECT category, region, COUNT(*) AS cnt, AVG(value_0) AS avg_val
 FROM read_parquet('/tmp/test_dataset/**')
 WHERE region = 'us-east-1'
 GROUP BY category, region
""").fetchdf()
print("Sync result:", result_sync)

Comparison with Traditional Synchronous I/O

How to Enable in Production

DuckDB’s async I/O is enabled by default in the latest version. You can fine-tune the behavior through the following configurations:

-- Check current configuration
SHOW all WHERE name LIKE '%io%';

-- Adjust async I/O thread count (based on CPU cores)
SET async_io_threads = 32;

-- Adjust prefetch buffer size
SET io_thread_buffer_size = '2GB';

-- Verify configuration
SELECT * FROM duckdb_settings() WHERE name LIKE '%async%';

Additional Optimizations for DuckLake and Iceberg

If you’re using DuckLake or Iceberg engines, the benefits of async I/O are even greater:

-- Iceberg table query (async I/O enabled automatically)
SELECT event_type, COUNT(*) AS cnt
FROM iceberg_scan('s3://warehouse/events/*.parquet')
WHERE event_date >= '2026-01-01'
GROUP BY event_type;

-- DuckLake table query
SELECT product_category, SUM(revenue) AS total
FROM ducklake_scan('s3://lake/sales/*.parquet')
WHERE quarter = 'Q1'
GROUP BY product_category;

Performance Tuning Recommendations

1. Choose the Right Prefetch Strategy for Your Query Pattern

-- Full aggregation: use WHOLE_GROUP (default)
SET parquet_prefer_whole_group_scan = true;

-- Select few columns: use COLUMN_WISE_EAGER
SET parquet_column_wise_eager = true;

-- High-selectivity filtering: use PREFETCH_FILTERS
SET parquet_prefetch_filters = true;

2. Set Async Thread Count Appropriately

CPU cores ≤ 8 → async_io_threads = 8
CPU cores 8-16 → async_io_threads = 16
CPU cores 16-32 → async_io_threads = 24
CPU cores 32+ → async_io_threads = min(cpu_cores - 4, 64)

Core principle: reserve a few cores for CPU-intensive computation, allocate the majority to async I/O.

3. Optimize Data Sharding and Row Group Size

-- Set appropriate row group size when creating Parquet
SET parquet_row_group_size = 128 * 1024 * 1024; -- 128MB per row group

-- Partition large files by date, combined with predicate pushdown
CREATE TABLE events_parquet AS
SELECT * FROM read_parquet('s3://bucket/events/2026-*/data/*.parquet')
WHERE event_date = '2026-06-01';

Comprehensive Tool Comparison

High-performance ETL example

import duckdb

con = duckdb.connect(’:memory:’) con.execute(“SET async_io_threads = 32”) con.execute(“SET parquet_prefetch_filters = true”)

Read from S3, filter, aggregate, write results — all in SQL

result = con.execute(""" COPY ( SELECT date_trunc(‘day’, ts) AS day, category, COUNT() AS orders, SUM(amount) AS revenue FROM read_parquet(‘s3://data-lake/sales/**/.parquet’) WHERE ts >= ‘2026-01-01’ GROUP BY 1, 2 ) TO ‘s3://report-bucket/daily/’ (FORMAT PARQUET); “”").fetchdf()

Package this solution as a SaaS tool and charge enterprise clients handling 10-100GB of data **** in subscription fees.

### 3. Technical Consulting and Training

- Share async I/O benchmark results and optimization insights on tech communities (Medium, Dev.to, LinkedIn)
- Create an online course "DuckDB Performance Tuning in Practice," covering async I/O tuning, query plan analysis, and more
- Provide DuckDB consulting for teams with big data needs but limited budgets, charging per consultation session

### 4. Cloud Cost Optimization Service

The performance gains from async I/O mean **the same data processing tasks can be completed with lower-tier instances**:

- Help enterprises downgrade from c6in.4xlarge to c6in.2xlarge, saving approximately 40-50% in compute costs
- Reduce S3/GCS API call counts and transfer latency, lowering storage expenses
- Provide cloud cost audit and optimization reports, charging 10-20% of the savings amount

> **Core logic**: Async I/O is not just a technical optimization — it's a **cloud cost reduction tool**. For enterprises spending over on data processing, your optimization service can pay for itself within 1-2 months.

## Summary

DuckDB's async I/O engine is one of the most significant performance improvements in 2026. It delivers a substantial 1.5-1.7x performance boost for Parquet reads, especially in cloud storage scenarios (S3, OSS). For teams using DuckDB to process large-scale Parquet data, upgrading to the latest version and correctly configuring async I/O parameters represents the most direct, lowest-cost optimization available.

Try enabling async I/O in your production environment today and observe how much your query performance improves!

---

## 本文信息

| 项目 | 内容 |
|------|------|
| DuckDB 版本 | v1.5.x（部分功能基于 v2.0 Preview） |
| 最后验证 | 2026-09-12 |
| 测试环境 | Linux / x86_64 / 16GB RAM |
| 官方文档 | [DuckDB Documentation](https://duckdb.org/docs/) |
| GitHub | [pengzz9527/duckdb-blog](https://github.com/pengzz9527/duckdb-blog) |

如发现错误，欢迎通过 [GitHub Issue](https://github.com/pengzz9527/duckdb-blog/issues) 或邮件 contact@duckdblab.org 反馈。