Skip to content

Fix Unicode case-insensitive search for keyword queries - #6224

Open
sadorlovsky wants to merge 2 commits into
stashapp:developfrom
sadorlovsky:fix-unicode-case-insensitive-search
Open

Fix Unicode case-insensitive search for keyword queries#6224
sadorlovsky wants to merge 2 commits into
stashapp:developfrom
sadorlovsky:fix-unicode-case-insensitive-search

Conversation

@sadorlovsky

Copy link
Copy Markdown

Problem

The original implementation used the LIKE operator for case-insensitive search which only supports case-insensitive matching for ASCII characters (A-Z), not Unicode. SQLite's LOWER() function also lacks Unicode support, so it wouldn't solve the problem. This caused searches to be case-sensitive for non-English content.

Example:

  • Searching for "анна" would NOT find "Анна"
  • Searching for "chloé" would NOT find "CHLOÉ"
  • Searching for "έλενα" would NOT find "Έλενα"

Solution

Use custom SQLite function lower_unicode() that uses Go's strings.ToLower() for proper Unicode case folding. All keyword search queries now use this function to ensure truly case-insensitive matching across all character sets.

@sadorlovsky
sadorlovsky force-pushed the fix-unicode-case-insensitive-search branch from f16e026 to 46ae680 Compare November 2, 2025 15:36
@WithoutPants

Copy link
Copy Markdown
Collaborator

I think we might want to consider adding/changing name indexes to accommodate this.

@sadorlovsky
sadorlovsky force-pushed the fix-unicode-case-insensitive-search branch from 37997aa to 09339ae Compare August 1, 2026 18:06
@sadorlovsky

Copy link
Copy Markdown
Author

I checked this — there is nothing a name index can accommodate here. Keyword search wraps every term as %term%, and with a leading wildcard SQLite does not use a B-tree index for the lookup. EXPLAIN QUERY PLAN for the performer query is identical before and after this change:

SCAN performers
SEARCH performer_aliases USING COVERING INDEX sqlite_autoindex_performer_aliases_1
(performer_id=?) LEFT-JOIN

The existing name indexes were not used for keyword search in the first place — they only serve uniqueness and exact-name lookups, which this PR does not touch.

I also tested expression indexes on lower_unicode(COALESCE(name,'')). SQLite uses them for equality and prefix queries as a control, but for the actual keyword query (LIKE '%term%') the plan remains a scan and the timings are unchanged. In addition, an index on an application-defined function introduces an operational dependency: connections that open the database without registering that function can fail on writes affecting the indexed table and on maintenance operations such as integrity_check, REINDEX, or VACUUM. I don't think expression indexes are a good fit here.

Benchmarks (SQLite 3.45.1 as bundled by mattn/go-sqlite3 v1.14.22, real performer query shape — LEFT JOIN + name OR alias, warm cache, median of 5 runs):

100k performers 1M performers
plain LIKE '%term%' 33 ms 0.34 s
lower_unicode() scan 0.36 s 3.6 s
+ matching expression indexes ≈ same, plan unchanged ≈ same, plan unchanged
FTS5 trigram MATCH (name only) 0.5 ms 10 ms

The extra cost comes from evaluating the application-defined lower_unicode() function for every scanned row. The latest revision limits this to the cases where it is actually needed: the predicate is chosen per parsed term — ASCII-only terms keep the existing LIKE path (which already provides case-insensitive matching for ASCII), while terms containing non-ASCII characters go through lower_unicode().

This keeps the common search path at the current performance level while fixing the non-Latin case-insensitive search issue.

If we want keyword search to become indexed in the future, that would require a different search approach rather than changing name indexes. SQLite FTS5 with tokenize='trigram' is one possible direction (similar to PostgreSQL pg_trgm), but it would require separate schema/build work and is orthogonal to this fix.

I also merged the latest changes with develop to resolve the conflict.

Use custom lower_unicode() function for proper Unicode case folding
instead of SQLite's LIKE operator which only supports ASCII.
The built-in LIKE is already case-insensitive for ASCII, so pure-ASCII
search terms keep the plain LIKE predicate and pay no per-row function
call cost. Only terms containing non-ASCII characters go through
lower_unicode(). The decision is made per parsed term, so mixed queries
like "anna мария" use the fast path for the ASCII term only.
@sadorlovsky
sadorlovsky force-pushed the fix-unicode-case-insensitive-search branch from 09339ae to cd0fdcc Compare August 15, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants