Vector Search Support

[yabsp]

Summary

This PR integrates vector similarity search into Polypheny by extending the PostgreSQL adapter with support for the pgvector extension.

Vectors become a typed value rather than an opaque array, adapters advertise their vector capabilities through a new feature-detection mechanism, and the planner pushes distance computation down to a capable store. Where a store cannot evaluate a distance natively, Polypheny falls back to its internal distance functions, so functionality never depends on a particular adapter.

Six metrics — L1, L2, cosine, inner product (float vectors) and Hamming, Jaccard (bit vectors) — are exposed from SQL, MQL, and Cypher. HNSW and IVFFlat indexes can be created from SQL DDL and from the web interface.

Changes

• The type system and catalog now track elementsNullable for array and collection types, parsed from <type> NOT NULL ARRAY(1, n) and cascaded through LogicalColumn, PhysicalColumn, RelationalCatalog, and DdlManager. Because it is a serialized field on LogicalColumn, a catalog persisted by an earlier build cannot be deserialized and is rebuilt from the data.
• New SqlDbFeature interface with a per-adapter enum (PostgresqlFeature: PGVECTOR, POSTGIS); SqlDialect answers through supportsFeature() / supportsVector(). Features are known for Docker-managed stores via PostgresqlImageVariant and detected by querying pg_extension for remote stores and sources.
• Two new visitors (CheckingKnnFunctionSupportVisitor, CheckingKnnFunctionVisitor) guard the JdbcRules predicates, so a distance is delegated only when the target dialect can evaluate it.
• Per-metric operators (L1_DISTANCE, L2_DISTANCE, COS_DISTANCE, INNER_PRODUCT_DISTANCE, HAMMING_DISTANCE, JACCARD_DISTANCE) encode the metric in their own Kind, letting PostgresqlSqlDialect.unparseCall emit native pgvector operators.
• JdbcJoinRule now pushes down a TRUE-condition join when both inputs carry a vector column and the dialect supports vectors. Every other Cartesian product keeps the old behavior.
• ALTER TABLE ... ADD INDEX gains a WITH (key=value) clause. Indexes are grouped by a new IndexCategory (VECTOR), and IndexMethodModel carries a category plus IndexParameterModels that the web interface renders its dialog from.
• AbstractJdbcSource gains a fetchColumnMetadata hook; for PostgreSQL it reads attndims and atttypmod to recover array and pgvector dimensions.
• Custom types are registered per connection through a new SqlDialect.initializeConnection hook (PGvector, PGbit, PGgeometry). ResultSetEnumerable.setDynamicParam binds VectorType parameters as native objects, and JdbcToEnumerableConverter reads them back via getCustomArrayRetrievalExpression.
• Added pgvector 0.1.6; bumped bouncycastle 1.80 → 1.80.2 and replaced the four hardcoded allowDependency versions with the bouncycastle_version property.

Features

• VectorType for fixed-dimension float and bit vectors, carried through parsing, validation, planning, and prepared statements.
• Six metrics as per-metric named functions, and native pgvector operators (<+>, <->, <=>, <#>, <~>, <%>), four supported by the generic, pre-existing DISTANCE function.
• Distance pushdown to pgvector for the SQL path, including filtered and cross-join shapes, with an internal fallback for every other store.
• MQL $vectorSearch stage with path, queryVector, metric, limit, and an optional filter.
• Cypher vector_distance(property, queryVector, metric).
• HNSW and IVFFlat index creation from SQL DDL and the web interface, with build parameters (m, ef_construction, lists) and metric selection.
• Feature detection for PostgreSQL stores and sources, plus Docker image variants (plain, pgvector, PostGIS, pgvector + PostGIS), including pgvector column discovery on sources.

Bug Fixes

• Fixed PostgreSQL index deletion.

Tests

• SqlPgvectorOperatorTest — all six metrics through both invocation forms, including filtered and cross-join shapes.
• ArrayNotNullConstraintTest — element-level NOT NULL enforcement for literal arrays and parameterized query vectors.
• JdbcBooleanArrayTest — bit vector insertion, binding, and retrieval round trip.
• MqlVectorDistanceTest / CypherVectorDistanceTest — $vectorSearch and vector_distance syntax, lowering, and validation.
• PostgresqlSqlDialectTest / PostgresqlVectorHelperTest — unparsing, cast handling, and dynamic-parameter emission.
• PostgresqlSourceTest / PostgresqlSourceDiscoveryTest — feature detection and dimension discovery on a source.
• PreparedIndexedStatementTest / PreparedNamedStatementTest — ARRAY/VectorType preserved through the Prism interface.
• Existing suites adjusted: ComplexViewTest, CatalogTransactionTest, SqlParserTest, and the Neo4j tests.

ToDo

• Verify design and implementation
• Verify test coverage and CI build status
• Confirm the pgvector Docker image variants are published to the Polypheny registry
• Support for Hamming and Jaccard metrics as parameters in DISTANCE function additionally to HAMMING_DISTANCE and JACCARD_DISTANCE (as for now the non-parameterized versions must be used)

[gartens]

Related UI PR: polypheny/Polypheny-UI#111