Fix Unicode case-insensitive search for keyword queries - #6224
Fix Unicode case-insensitive search for keyword queries#6224sadorlovsky wants to merge 2 commits into
Conversation
f16e026 to
46ae680
Compare
|
I think we might want to consider adding/changing name indexes to accommodate this. |
37997aa to
09339ae
Compare
|
I checked this — there is nothing a name index can accommodate here. Keyword search wraps every term as The existing name indexes were not used for keyword search in the first place — they only serve uniqueness and exact-name lookups, which this PR does not touch. I also tested expression indexes on Benchmarks (SQLite 3.45.1 as bundled by
The extra cost comes from evaluating the application-defined This keeps the common search path at the current performance level while fixing the non-Latin case-insensitive search issue. If we want keyword search to become indexed in the future, that would require a different search approach rather than changing name indexes. SQLite FTS5 with I also merged the latest changes with |
Use custom lower_unicode() function for proper Unicode case folding instead of SQLite's LIKE operator which only supports ASCII.
The built-in LIKE is already case-insensitive for ASCII, so pure-ASCII search terms keep the plain LIKE predicate and pay no per-row function call cost. Only terms containing non-ASCII characters go through lower_unicode(). The decision is made per parsed term, so mixed queries like "anna мария" use the fast path for the ASCII term only.
09339ae to
cd0fdcc
Compare
Problem
The original implementation used the
LIKEoperator for case-insensitive search which only supports case-insensitive matching for ASCII characters (A-Z), not Unicode. SQLite'sLOWER()function also lacks Unicode support, so it wouldn't solve the problem. This caused searches to be case-sensitive for non-English content.Example:
Solution
Use custom SQLite function
lower_unicode()that uses Go'sstrings.ToLower()for proper Unicode case folding. All keyword search queries now use this function to ensure truly case-insensitive matching across all character sets.