DuckDB in Action: Fuzzy Search & Text Processing

Master DuckDB's fuzzy search and text processing capabilities: LIKE pattern matching, regular expressions, Levenshtein edit distance, and full-text search extension with real-world SQL examples.

Introduction

In real-world business scenarios, data is rarely perfectly formatted. Product names have spelling variations, user addresses come in inconsistent formats, log texts need fuzzy matching… DuckDB provides a comprehensive set of text search and matching tools — from basic LIKE to advanced full-text search — helping you handle “imperfect” text data in your analytics workflows.

This article covers DuckDB’s text processing techniques with practical business scenarios and runnable SQL examples.

1. LIKE Pattern Matching

Basic Usage

LIKE is the classic text matching approach, supporting % (any characters) and _ (single character) wildcards.

CREATE TABLE products AS
SELECT * FROM (VALUES
  ('Dell XPS 15', 1299),
  ('Dell Latitude 5520', 899),
  ('HP EliteBook 840', 1099),
  ('Apple MacBook Pro 14', 1999),
  ('Lenovo ThinkPad X1 Carbon', 1499)
) AS t(name, price);

-- Find all Dell laptops
SELECT name, price
FROM products
WHERE name LIKE 'Dell %';
nameprice
Dell XPS 151299
Dell Latitude 5520899

Terminal Output

Fig: LIKE ‘Dell %’ fuzzy match on products table, filtering all Dell brand laptops

ILIKE: Case-Insensitive Matching

-- Case-insensitive search
SELECT name FROM products WHERE name ILIKE '%pro%';
name
HP EliteBook 840
Apple MacBook Pro 14
Lenovo ThinkPad X1 Carbon

Note: ILIKE is DuckDB’s non-standard extension. For POSIX compatibility, use SIMILAR TO or regular expressions.

2. Regular Expressions: ~ Operator & REGEXP_REPLACE

Regex Matching with ~

DuckDB uses the ~ operator for regex matching (PostgreSQL-compatible syntax):

-- Match SKUs in format: XXX-NNNN (3 uppercase letters + dash + 4 alphanumeric)
CREATE TABLE sku_map AS
SELECT * FROM (VALUES
  ('SKU-A1B2', 'Dell XPS 15'),
  ('SKU-C3D4', 'HP EliteBook 840'),
  ('REF-XYZ',  'Apple MacBook Pro 14'),
  ('INV-99A1', 'Lenovo ThinkPad X1 Carbon')
) AS t(sku, product_name);

SELECT sku, product_name
FROM sku_map
WHERE sku ~ '^[A-Z]{3}-[A-Z0-9]{4}$';
skuproduct_name
SKU-A1B2Dell XPS 15
SKU-C3D4HP EliteBook 840
INV-99A1Lenovo ThinkPad X1 Carbon

REGEXP_REPLACE: Text Cleaning

-- Clean up product names: normalize whitespace and strip special chars
SELECT
  name,
  regexp_replace(name, '\s+', ' ', 'g') AS cleaned_name,
  regexp_replace(name, '[^a-zA-Z0-9 ]', '', 'g') AS alphanum_only
FROM products;
namecleaned_namealphanum_only
Dell XPS 15Dell XPS 15Dell XPS 15
HP EliteBook 840HP EliteBook 840HP EliteBook 840
Apple MacBook Pro 14Apple MacBook Pro 14Apple MacBook Pro 14
Lenovo ThinkPad X1 CarbonLenovo ThinkPad X1 CarbonLenovo ThinkPad X1Carbon

3. Levenshtein Edit Distance: The Fuzzy Matching Power Tool

What is Levenshtein Distance?

Levenshtein distance measures the minimum number of single-character edits (insertions, deletions, substitutions) needed to change one string into another. Smaller distance = more similar strings. DuckDB has built-in levenshtein() — no extra extension needed.

Use Case: Brand Name Fuzzy Matching

CREATE TABLE brand_names AS
SELECT * FROM (VALUES
  ('Dell'), ('HP'), ('Apple'), ('Lenovo'), ('Asus'), ('Acer')
) AS t(brand);

CREATE TABLE user_input AS
SELECT * FROM (VALUES
  ('Del'), ('Hp'), ('Apel'), ('Lenovo'), ('Assu'), ('MicroSoft')
) AS t(input_brand);

-- Find the most similar brand for each user input
SELECT
  u.input_brand,
  b.brand,
  levenshtein(u.input_brand, b.brand) AS distance
FROM user_input u
CROSS JOIN brand_names b
WHERE levenshtein(u.input_brand, b.brand) <= 2
ORDER BY u.input_brand, distance;
input_brandbranddistance
ApelApple2
AssuAsus2
DelDell1
HpHP1
LenovoLenovo0

Terminal Output

Fig: Levenshtein edit distance fuzzy matching — maps user-typed brand names to standard brand names

Use Case: Customer Address Deduplication

CREATE TABLE addresses AS
SELECT * FROM (VALUES
  ('1600 Amphitheatre Pkwy, Mountain View, CA'),
  ('1600 amphetamine pkwy mountain view ca'),
  ('350 5th Ave, New York, NY'),
  ('350 Fifth Avenue, New York, NY'),
  ('1 Apple Park Way, Cupertino, CA')
) AS t(raw_address);

-- Find duplicate addresses with similarity >= 0.7
WITH normalized AS (
  SELECT
    raw_address,
    row_number() OVER () AS rid,
    LOWER(regexp_replace(raw_address, '[^a-zA-Z0-9 ]', '')) AS clean_addr
  FROM addresses
)
SELECT
  a.raw_address,
  b.raw_address,
  round(1.0 - levenshtein(a.clean_addr, b.clean_addr)::FLOAT /
        GREATEST(length(a.clean_addr), length(b.clean_addr)), 2) AS similarity
FROM normalized a
JOIN normalized b ON a.rid < b.rid
WHERE 1.0 - levenshtein(a.clean_addr, b.clean_addr)::FLOAT /
        GREATEST(length(a.clean_addr), length(b.clean_addr)) >= 0.7;
raw_addressraw_addresssimilarity
1600 Amphitheatre Pkwy…1600 amphetamine pkwy…0.82
350 5th Ave, New York, NY350 Fifth Avenue, New York, NY0.79

Terminal Output

Fig: Address deduplication based on Levenshtein similarity — auto-detects different-format records referring to the same location

4. Full-Text Search: FTS Extension

DuckDB provides an fts extension supporting SQLite FTS5-style full-text search.

Installation

INSTALL fts;
LOAD fts;

Creating Index & Searching

-- Create sample product reviews table
CREATE TABLE reviews AS
SELECT * FROM (VALUES
  (1, 'Great laptop, fast and lightweight. Battery lasts all day.'),
  (2, 'Terrible product. Screen cracked after one week of use.'),
  (3, 'The keyboard feels premium, but the trackpad is too small.'),
  (4, 'Excellent value for money. Highly recommend for students.'),
  (5, 'Overpriced and overhyped. Nothing special about this model.')
) AS t(id, comment);

-- Create FTS index
SELECT create_fts_index('reviews', 'comment', false, false, 'reviews_idx', '', false);

-- Full-text search (keywords: laptop, battery, keyboard)
SELECT id, comment FROM reviews
WHERE search_fts_index('reviews_idx', 'laptop battery keyboard') > 0
ORDER BY id;

BM25 Relevance Ranking

The FTS extension uses BM25 algorithm internally — results are ranked by relevance automatically. Lower rank values indicate higher relevance.

-- Search with relevance scoring
SELECT
  id,
  comment,
  ROUND(search_fts_index('reviews_idx', 'great value'), 3) AS relevance
FROM reviews
WHERE search_fts_index('reviews_idx', 'great value') > 0;

Version Note: Full FTS features (including fts5 virtual tables and MATCH syntax) are more complete in DuckDB v1.6+. v1.5.x supports core functionality via create_fts_index() and search_fts_index() functions.

5. Practical: Building a Product Recommendation Engine

Combining all the techniques above, here’s a comprehensive example — text-similarity-based product recommendations:

-- User purchase history
CREATE TABLE purchases AS
SELECT * FROM (VALUES
  (1001, 'Dell XPS 15'),
  (1002, 'HP EliteBook 840'),
  (1003, 'MacBook Pro 14')
) AS t(user_id, product_name);

-- Candidate recommendation items
CREATE TABLE candidates AS
SELECT * FROM (VALUES
  ('Dell XPS 17'),
  ('Dell Latitude 5530'),
  ('HP Spectre x360'),
  ('Lenovo ThinkPad T14'),
  ('MacBook Air 15'),
  ('Asus ZenBook 14')
) AS t(product_name);

-- Recommend similar products based on Levenshtein distance
SELECT
  p.user_id,
  p.product_name AS purchased,
  c.product_name AS recommended,
  levenshtein(LOWER(p.product_name), LOWER(c.product_name)) AS similarity_score
FROM purchases p
CROSS JOIN candidates c
WHERE levenshtein(LOWER(p.product_name), LOWER(c.product_name)) <= 8
ORDER BY p.user_id, similarity_score;
user_idpurchasedrecommendedsimilarity_score
1001Dell XPS 15Dell XPS 171
1003MacBook Pro 14MacBook Air 154

Summary

TechniqueUse CasePerformance
LIKE / ILIKESimple prefix/suffix/contains matchingVery fast, ideal for exact patterns
REGEXP (~)Complex pattern matching & text replacementModerate, regex engine overhead
LevenshteinFuzzy matching, spell correctionO(n×m), best for small datasets
FTS (BM25)Large-scale text search & relevance rankingFast after indexing, ideal for document libraries

Best Practices:

  • Start with LIKE/ILIKE for quick filtering
  • Use REGEXP for structured data cleaning
  • Use Levenshtein for small-scale fuzzy matching
  • Use FTS extension for text search at scale (10k+ records)

More DuckDB tips and tutorials at DuckDB Lab (duckdblab.org).

📺 Watch video tutorials → Olap Studio YouTube

Subscribe for more DuckDB & AI automation tutorials