duckdb-normalized

A DuckDB extension that adds is_normalized(str) — a fast check for Unicode NFC, equivalent to PostgreSQL’s str IS NORMALIZED (NFC is the default form).

Unicode can encode the same text in more than one way. Without normalization, strings that look identical do not compare equal. This function answers whether a value is already in NFC, without allocating a normalized copy on the common path.

SELECT count(*) FROM hits WHERE is_normalized(url);

┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│     99997420 │
└──────────────┘

Function

Signature	Returns	Nulls
is_normalized(VARCHAR)	BOOLEAN	NULL in → NULL out

true if the string is Unicode NFC, false otherwise. DuckDB VARCHAR is valid UTF-8, so the check assumes well-formed input.

SELECT is_normalized('https://example.com');           -- true  (ASCII is always NFC)
SELECT is_normalized('caf' || chr(233));               -- true  (precomposed é)
SELECT is_normalized('e' || chr(769));                 -- false (e + combining acute)
SELECT is_normalized(chr(8486));                       -- false (Ω OHM SIGN → Ω in NFC)
SELECT is_normalized(chr(44032));                      -- true  (Hangul syllable 가)
SELECT is_normalized(chr(4352) || chr(4449));          -- false (Hangul L+V jamo)
SELECT is_normalized(NULL);                            -- NULL

It agrees with DuckDB’s built-in nfc_normalize:

SELECT is_normalized(s) = (nfc_normalize(s) IS NOT DISTINCT FROM s);

Why it is fast

1. ASCII fast path. If every code point is one byte, the string is pure ASCII, which is always NFC. On ClickBench hits, that is ~85% of URLs.
2. Vectorized code-point counting. Code points are counted with NEON / AVX2 / SSE2 (SWAR fallback), not a per-character callback. byte_len == code_point_count is the ASCII check.

Non-ASCII strings take an allocation-free NFC quick check (combining-class order, Hangul composition, singleton replacements). Only the rare “maybe composes” cases fall through to utf8proc NFC + compare. On ClickBench hits, 77 of ~100M URLs are not already NFC.

ClickBench

Query, fully cached in memory (URL column only):

SELECT count(*) FROM hits WHERE is_normalized(url);

Engine	Threads	Time
PostgreSQL 18 IS NORMALIZED	2 workers	40s
DuckDB is_normalized	14 threads	0.31s
DuckDB is_normalized	2 threads	1.52s

DuckDB numbers were measured on an Apple M4 Pro (14 cores) against ClickBench hits_raw.parquet (~99,997,497 rows, 85.04% ASCII, 77 not NFC), fully cached in memory. The PostgreSQL figure is a published ClickBench url IS NORMALIZED result with the default max_parallel_workers_per_gather = 2 (Zen 5 laptop, 12 cores / 24 threads) — not a same-machine comparison.

A synthetic 10M-row bench with the same 85/15 mix lives in benchmark/is_normalized.benchmark.

Building

Needs a C++17 compiler, CMake, and Ninja (recommended). Clone with submodules:

git clone --recurse-submodules https://github.com/<you>/duckdb-normalized.git
cd duckdb-normalized
GEN=ninja make

That produces:

./build/release/duckdb
./build/release/test/unittest
./build/release/extension/normalized/normalized.duckdb_extension

The duckdb shell has the extension preloaded:

./build/release/duckdb

D SELECT is_normalized('café');
┌─────────────────────┐
│ is_normalized('café') │
│        boolean        │
├─────────────────────┤
│ true                  │
└─────────────────────┘

To load the shared extension from another DuckDB build (unsigned):

duckdb -unsigned

LOAD 'build/release/extension/normalized/normalized.duckdb_extension';

Built against DuckDB v1.5.5.

Tests

make test

SQLLogic tests are in test/sql/is_normalized.test.

License

MIT. See LICENSE.