Building a Lakehouse with DuckDB and Apache Iceberg

A Practical Guide to Modern Data Architecture

Author: Rajesh Kumar Meher
Category: Data Engineering
Reading Time: 5–7 minutes

1. Introduction

Modern data platforms need to process large volumes of data efficiently while keeping infrastructure costs under control. The lakehouse architecture addresses this challenge by combining open data storage with reliable table management and analytical processing.

Two technologies that complement each other are DuckDB and Apache Iceberg.

• DuckDB executes analytical SQL queries efficiently without requiring a separate database server.

• Apache Iceberg manages analytical tables, metadata, schema evolution, and historical snapshots.

Together, they provide a flexible foundation for modern data engineering.

2. How Do They Work Together?

DuckDB and Iceberg serve different purposes.

Iceberg organizes the data into managed tables, while DuckDB queries those tables through supported integrations.

3. Architecture and Query Execution

The architecture follows this logical flow:

Data Sources → Ingestion → Object Storage → Iceberg Tables → DuckDB → Analytics

When an application submits a query, DuckDB resolves the table through the configured catalog, reads the relevant Iceberg metadata, identifies the required data files, executes the SQL, and returns the results.

Sequence Ladder Diagram

Insert the generated DuckDB and Iceberg sequence diagram image here.

The diagram should illustrate these interactions:

1. An application submits an SQL query to DuckDB.

2. DuckDB resolves the table through the Iceberg catalog.

3. The catalog provides the table metadata location.

4. DuckDB reads the metadata and identifies relevant data files.

5. DuckDB retrieves the required data from object storage.

6. DuckDB executes the query and returns the results.

This approach separates table management from query execution.

Note: Direct path-based Iceberg scans in DuckDB are read-only. Supported write operations require an appropriate catalog integration.

4. Practical Example: Revenue Analysis

Suppose an Iceberg table contains transaction records with region, amount, transaction date, and status.

Once the catalog is configured, DuckDB can execute a query like this:

SELECT
    region,
    COUNT(*) AS transactions,
    SUM(amount) AS total_revenue
FROM lakehouse.sales.transactions
WHERE status = 'COMPLETED'
GROUP BY region
ORDER BY total_revenue DESC;

SELECT
    region,
    COUNT(*) AS transactions,
    SUM(amount) AS total_revenue
FROM lakehouse.sales.transactions
WHERE status = 'COMPLETED'
GROUP BY region
ORDER BY total_revenue DESC;

SELECT
    region,
    COUNT(*) AS transactions,
    SUM(amount) AS total_revenue
FROM lakehouse.sales.transactions
WHERE status = 'COMPLETED'
GROUP BY region
ORDER BY total_revenue DESC;

This query calculates the number of completed transactions and total revenue for each region.

DuckDB handles the SQL execution, while Iceberg provides the managed table structure.

5. Key Benefits

Performance: DuckDB supports vectorized execution, parallel processing, and query optimization.

Open architecture: Iceberg tables can be accessed by multiple compatible processing engines.

Reliable table management: Iceberg provides snapshots, schema evolution, and transactional table operations.

Flexible analytics: DuckDB can support data exploration, reporting, data quality checks, and suitable transformation workloads.

Cost efficiency: Embedded query execution can reduce infrastructure requirements for workloads that do not need a distributed engine.

Actual performance and cost savings depend on the workload and deployment.

6. When Should You Use This Architecture?

Consider DuckDB and Iceberg when building:

• Analytical applications and reporting pipelines.

• Data quality validation workflows.

• Data preparation pipelines for machine learning.

• Open lakehouse platforms that support multiple query engines.

• Lightweight SQL analytics over managed analytical datasets.

For very large distributed workloads or high-concurrency analytical services, evaluate whether a distributed processing engine or centralized data warehouse is more appropriate.

Conclusion

DuckDB and Apache Iceberg provide complementary capabilities for modern data engineering.

Apache Iceberg manages analytical tables, while DuckDB executes analytical queries.

Together, they enable flexible lakehouse architectures based on open data formats without requiring every workload to use a large distributed processing environment.

The key is to choose the right architecture for the data volume, concurrency, and operational requirements.

References

• DuckDB Documentation

• DuckDB Iceberg Extension

• Apache Iceberg Documentation