Support reading the AlphaTheta OneLibrary format - #196
Conversation
OneLibrary, also documented as Device Library Plus, is the successor to the DeviceSQL export.pdb library that rekordbox writes to USB media. A device usually carries both, and a player that understands OneLibrary prefers it. The library is a single SQLite database encrypted with SQLCipher 4. Its page format is implemented in util/crypto, so neither SQLCipher nor OpenSSL is needed to read one: AES-256-CBC, SHA-512, HMAC and PBKDF2 are written from their specifications, and the write-ahead log is folded in before SQLite sees the file, as rekordbox leaves most of a fresh export in the log. Support is read-only. Track metadata, playlists and crates can be read; everything that would change a database throws unsupported_operation. Beat grids, waveforms, hot cues and loops are not in the database at all, as rekordbox leaves them in the ANLZ files beside the music, so onelibrary::library gives the path recorded for a track alongside the key notation and track colour that the format-agnostic interface has nowhere to put. Reading a database needs SQLite 3.36 or newer, built without SQLITE_OMIT_DESERIALIZE.
|
Hi @v1vendi , thank you very much for this PR. I'll take a look through, but I want to set expectations: as this is the first code for a new format in the library, it may take some time and back-and-forth to get this ready for merging. I don't have any problem with the use of AI to assist with coding - in fact, I should probably add some statements along these lines to the contributing guidelines. One thing I will say is that I will treat this PR as though Claude was not there: it'll still be assessed to the same levels of quality as any other. Anyway, this looks like a great start - first-pass review incoming soon... |
|
Hey! I'm interested in this topic so I might take a look at the proposed changes. I have a couple of questions:
|
|
Hey @acrilique Regarding ANLZ files, tbh I didn't think too far into the future, and now my goal is to make MIXXX import modern Pioneer USBs, and after that I am ready to start investigating export :) |
Reading an encrypted database was dominated by software AES and by copying, so several layers of that are removed: * AES uses the processor's own AES instructions where it has them (x86 AES-NI, the ARMv8 cryptographic extension), decided at run time and confined to `aes_hardware.cpp`, which the build compiles with the intrinsics enabled only where they compile at all. Decryption runs eight independent blocks at a time to keep the pipeline full. * The software fallback moves to fused round tables and to the equivalent inverse cipher, so both directions have the same shape. * SHA-512 keeps a sixteen-word wrapping schedule and unrolls its rounds by rotating variable names instead of shifting values. * HMAC keys absorb both padded blocks once, which halves key derivation over its 256,000 iterations and saves two compressions per page. * Page tags are computed over the ciphertext, IV and page number in place, rather than gathering each page into a scratch buffer. * The write-ahead log is read before the database, so a page the log replaces or truncates away is never decrypted at all. * Page decryption is spread over the available processors, pages being independent of one another. Tests gain the NIST SP 800-38A chaining vectors and run every cipher test over both implementations, so the tables are still exercised on a machine that would otherwise only ever use its AES instructions.
|
Hi @v1vendi I'd be cool if you managed to make mixxx import modern pioneer usbs as a first step. I'm mainly focusing on export since day one, and I'm trying to think of the exported USB as a whole (i.e. onelibrary + pdb + settings files + anlz). In my mind, a library that allowed (but not necessarily enforced) you to generate it all with ease would be the ideal as it would reach the highest point of compatibility with all hardware, old and new. However when it comes to reading, I've done less investigation and brainstorming for a design. I suppose it could be symmetrical: if on the write-side you're able to either add a track/playlist to pdb, to onelibrary or to both at once, then on the read side you could also have the option to read tracks from pdb, from onelibrary, or from both (with something that deduplicated them by e.g. the file path). Sorry if this was a bit too much thinking out loud. I'm looking (not very actively tho) for people to help on ideas for the library design, testing exports with real hardware, reviewing code, etc in case you're ever interested. |
GCC 12.3 on x86-64 rejects `resolve`, where the path is trimmed a level at a time by assigning a piece of a string back to itself. Inlining turns that into a memcpy whose bounds it cannot establish, and it warns of an overlap of nine quintillion bytes, which -Werror then makes fatal. Track the end of the prefix as an index instead and build the string once, at the end. The construct the warning fires on is gone, and no path resolves any differently: the two agree on every string up to length six over `/`, `\` and a letter, and on the absolute, relative, Windows and UNC cases besides.
|
I pushed a significantly more fast implementation of crypto algorithms, which is almost on par with OpenSSL - but still does not require ~4mb of OpenSSL as a dependency. |
Two things went wrong with `-DSYSTEM_SQLITE=OFF`, which is how the macOS builds are configured. A test that talks to SQLite directly failed to link. The bundled amalgamation is compiled into the library, whose symbols are hidden, so there was nothing for the test to link against; the tests that use SQLite now compile it themselves, as they already do for the internal sources they reach into. That then exposed the second: the bundled amalgamation is 3.33, and reading a decrypted database needs `sqlite3_deserialize` from 3.36. The library correctly reports the format unsupported there, so the database test is registered only where SQLite can do the work, and CMake says so when it is left out.
|
So, as I found out, Onelibrary needs |
Reading a decrypted database goes through `sqlite3_deserialize`, which arrived in SQLite 3.36, so the test of it is registered only where that much is available. It asked about the system installation alone, which left the bundled copy out of the reckoning altogether. Read the bundled version out of the amalgamation instead, and ask the same question of both. Nothing changes while the bundled copy is 3.33, which cannot do the work; when it is newer the test follows.
Each of them talks to SQLite directly, and a test cannot borrow it from the library, whose symbols are hidden. With `-DSYSTEM_SQLITE=OFF` that means every one of them compiles the amalgamation into itself, some fifteen seconds apiece, on top of the copy the library already builds. They are commented out rather than deleted, and their sources are left where they are, so uncommenting the block runs them again.
mr-smidge
left a comment
mr-smidge
left a comment
There was a problem hiding this comment.
This PR is a great start, and I've tried to be comprehensive in my review.
I think there are three main themes that we should think about:
- The choice of SQLCipher vs. an explicit crypto implementation in libdjinterop. Although the minimal code has some advantages, I keep thinking that libdjinterop is not a crypto library and shouldn't try to be. Yet at the same time, nobody has tried to get SQLCipher in yet so if there are any challenges we don't yet know them. This can be discussed more on #191.
- I think there needs to be cleaner separation between the high-level and low-level API for OneLibrary.
- The low-level API may presumably undergo version changes like any other format (I don't have enough experience with AlphaTheta equipment/formats to know how much this has already happened since the format was first released), and I think it would be sensible to organise low-level API code such that any breaking changes to the underlying format can be easily handled.
Many thanks for your efforts here - let me know what you think.
| /// memory, so a caller that wants both the unified interface and the extras | ||
| /// below should load a `library` and take `db()` from it rather than also | ||
| /// calling `load_database`. | ||
| class DJINTEROP_PUBLIC library |
There was a problem hiding this comment.
The functions above (database_exists(), load_database()) would be considered part of libdjinterop's high-level API, whereas this library class seems to contain things that are specific to the OneLibrary format and hence part of a low-level API. See GUIDE.md for more info.
I think these should be split into separate public headers.
Does OneLibrary have any kind of versioning scheme to go with it? Has it undergone any significant version changes since it was first released that might give us a hint as to how new versions of the schema are released? (This is the reason that the engine implementations have engine/v2, engine/v3, etc.)
| | `engine_library_v2_low_level` | Uses the low-level Engine v2 API to work with tables directly. | | ||
| | `onelibrary` | Prints the tracks and playlists of an AlphaTheta OneLibrary device. | | ||
|
|
||
| Each application can be minimally compiled in isolation with an invocation |
There was a problem hiding this comment.
Is this a realistic use case, compiling the examples in isolation?
If the user has gone to the trouble of cloning this repo, is there some advantage offered by compiling in isolation rather than the CMake targets that are already there?
| /// \return Returns the path, or no value if the track is not there or | ||
| /// carries no analysis data. | ||
| [[nodiscard]] std::optional<std::string> analysis_path( | ||
| int64_t track_id) const; |
There was a problem hiding this comment.
For this and the below member functions, why are these part of the library class, as opposed to exposing the low-level concepts more directly?
Being more specific, the SQL to implement these appears to be fetching data from the content table... so perhaps a low-level API for OneLibrary should expose the content table as a first-class citizen instead? This would match how the Engine low-level API works in terms of exposing rows and columns from the key tables, and make a clean separation between the low-level and high-level API logic.
| @@ -0,0 +1,124 @@ | |||
| /* | |||
There was a problem hiding this comment.
How come this isn't a public header?
| /// may point at a row that is not there. | ||
| struct content_row | ||
| { | ||
| int64_t id = 0; |
There was a problem hiding this comment.
Does zero have a special meaning?
If so, consider using a constant to make the meaning obvious, e.g. https://github.com/xsco/libdjinterop/blob/main/include/djinterop/engine/v2/track_table.hpp#L57
| #include <string> | ||
| #include <vector> | ||
|
|
||
| #include <sqlite3.h> |
There was a problem hiding this comment.
There is currently no test code that needs to talk to SQLite directly, let alone using the C API. So I think this #include being here is a code smell.
Engine unashamedly offers a function djinterop::engine::create_database_from_scripts(), which is used by test code. Perhaps OneLibrary tests should do something similar?
| /// `album.image_id`, and gives `playlist_content` and `property` primary keys | ||
| /// that an export does not carry. What a device holds is what is written | ||
| /// here. | ||
| inline const std::vector<std::string>& onelibrary_schema_statements() |
There was a problem hiding this comment.
Does an 'empty' database have any entries in the property table, e.g. to indicate the version?
| /// here. | ||
| inline const std::vector<std::string>& onelibrary_schema_statements() | ||
| { | ||
| static const std::vector<std::string> statements{ |
There was a problem hiding this comment.
I think it might be better to follow the existing pattern and put database schema in <libdjinterop>/testdata/ref/onelibrary rather than embed in C++ sources. Combine with an equivalent to Engine's create_database_from_scripts() to get their data.
| /cmake_build* | ||
| /cmake-build* | ||
|
|
||
| /build |
| # The OneLibrary tests are commented out below. Each of them talks to | ||
| # SQLite directly, and a test cannot borrow SQLite from the library: its |
There was a problem hiding this comment.
See comment in database_test.cpp - I think it might be best if the tests only use the public API of libdjinterop and don't talk to SQLite directly.
3 participants
Partially solves #177
The PR is huge and made with help of AI, but what is not nowadays...
At least I did multiple rounds of guided review and cleanup.
Tested on MacOS with a real USB exported from Rekordbox.
I already prepared and tested integration to MIXXX too.
Writing also shouldn't be hard, but seems reasonable to start with readonly mode.
No need for SQLCipher dependency mentioned in #191
The library is one SQLite database encrypted with SQLCipher 4. Its page format is implemented in
util/crypto, so neither SQLCipher nor OpenSSL becomes a dependency: AES-256-CBC, SHA-512, HMAC and PBKDF2 are written from their specifications and covered by test vectors from FIPS 197, FIPS 180-4, RFC 4231 and RFC 6070.Rekordbox leaves most of a fresh export in the write-ahead log rather than the database file, so a reader that ignores the log reports a nearly empty library with no error at all. The log is folded in before SQLite ever sees the bytes.
The passphrase is the same on every installation and depends on neither licence nor machine, so any player can read any device. It can be overridden, should a future release of rekordbox change it.
Scope
Support is read-only: track metadata, playlists and crates can be read, and everything that would change a database throws
unsupported_operation. The format keeps a single tree serving as both playlists and crates, soplaylists_and_crates_are_distinctis false.Beat grids, waveforms, hot cues and loops are not in the database at all — rekordbox leaves them in the ANLZ files beside the music, and exports an empty
cuetable — so those accessors return nothing rather than throwing. For a caller that reads ANLZ files itself,onelibrary::librarygives the analysis path recorded for a track, along with the key notation as written (which may be Camelot, and so cannot be represented bytrack::key()) and the track colour.Tests
Unit tests cover the crypto primitives against published test vectors, the content and playlist tables, and the database as a whole, reading fixtures that are built as plain SQLite and encrypted independently by the test suite, so a round trip checks the implementation against the format rather than against itself.