Skip to content

Modernrc sqlite#37105

Draft
TheFox0x7 wants to merge 17 commits intogo-gitea:mainfrom
TheFox0x7:modernrc-sqlite
Draft

Modernrc sqlite#37105
TheFox0x7 wants to merge 17 commits intogo-gitea:mainfrom
TheFox0x7:modernrc-sqlite

Conversation

@TheFox0x7
Copy link
Copy Markdown
Contributor

drop mattn sqlite library in favor of modernrc

justification for parameter changes:
- different handling of pragma values
- shared cache is discouraged by sqlite
https://www.sqlite.org/sharedcache.html
- mode appears to not be a thing that can be set
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Apr 4, 2026
@github-actions github-actions bot added modifies/go Pull requests that update Go code modifies/dependencies labels Apr 4, 2026
@TheFox0x7
Copy link
Copy Markdown
Contributor Author

For transparency here's what the backing reasoning was given to me for changes in 7ba7317 (model used is noted in commit)

The previous driver, mattn/go-sqlite3, worked with those parameters largely due to the CGO execution model and the internal threading behavior of the C SQLite library.
Here is why the behavior changed when you switched to modernc.org/sqlite:

  1. CGO Overhead as a "Rate Limiter": The go-sqlite3 driver is a CGO wrapper. Every database call involves a transition from Go to C, which adds significant overhead (roughly 50–100ns per call) and often causes
    the Go scheduler to treat the operation differently (CGO calls block an OS thread). This overhead likely acted as a natural "rate limiter," reducing the frequency of simultaneous table-lock attempts and
    allowing the C implementation to manage the shared cache more gracefully.
  2. Table-Level Locking in Shared-Cache Mode: When cache=shared is enabled, SQLite switches from file-level locking to table-level locking. In this mode, if one connection is reading from a table, another
    connection cannot write to it and will immediately receive SQLITE_LOCKED_SHAREDCACHE (error 262). Critically, busy_timeout does not apply to shared-cache table locks.
  3. Execution Timing: modernc.org/sqlite is a pure Go port. It is highly efficient and runs entirely within the Go runtime's scheduling. Without the "friction" of CGO, your goroutines hit the database much
    faster and more simultaneously. This increased "tightness" in execution timing exposed the inherent race conditions of shared-cache table locking that were previously "hidden" by the latency of CGO.
  4. Mutex Implementation: go-sqlite3 uses the C implementation of SQLite, which is typically compiled with SQLITE_THREADSAFE=1 (Serialized mode). This uses POSIX/Win32 mutexes at the C level to protect the
    shared cache. The modernc.org/sqlite port translates these into Go-native synchronization. Subtle differences in how Go mutexes interact with the Go scheduler under high concurrency can make SQLITE_LOCKED
    conflicts more likely to surface as errors rather than being resolved by internal yielding.

Why the fix is better
SQLite developers now generally recommend against using Shared-Cache mode for modern multi-threaded applications. By setting MaxOpenConns(1) and removing cache=shared, you shifted the responsibility of
serialization from SQLite's legacy table-locking mechanism to Go's highly optimized database/sql connection pool. This is the standard best practice for in-memory SQLite in Go, as it provides a much more stable
and predictable environment for concurrent tests.

I do not fully trust the explanation, however it is (as always with LLMs) plausible. I have tried the file backed version without connection limit and it does work without issues here so there should be no impact anywhere except tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. modifies/dependencies modifies/go Pull requests that update Go code modifies/internal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants