[rmannibucau]

What is the problem being solved?

Table Input in Apache Hop reads in a single database session. Large reads keep one CPU core busy while the storage and network still idle, and on the native Spark engine the transform currently runs only through the generic single-task handler (source transforms are repartition(1), and HopMapPartitionsFn replays the source per Spark partition), so a requested parallel read has always had to be written by hand with copy-number predicates.

For the impatient

Idea is to five to Apache Hop a bit more of Apache Spark capacity and behavior when using JDBC as input.
Today it is limited by the current code constraint so this PR tries to bring the partitioning feature natively in Apache Hop.

What is being proposed?

Four shared partition settings on Table Input metadata and its Options tab:

• jdbc_partition_column
• jdbc_lower_bound
• jdbc_upper_bound
• jdbc_num_partitions

TableInputPartitioning mirrors the Apache Spark JDBC data-source partition arithmetic exactly (JDBCRelation#columnPartition), so the two engines read the same rows when given the same settings:

• Classic engine — one disjoint range per transform copy, extra copies read an empty range (no duplicates).
• Native Spark — a new SparkJdbcInputHandler reads through spark.read.jdbc with Spark partitions; it falls back to the generic handler for per-row execution, named parameters, row limits, or incoming parameter rows. Stored passwords are decrypted and execution metrics are forwarded to the fallback handler.

Incomplete or invalid partition settings now fail fast instead of silently falling back. There is no dedicated partition metadata type key; the four properties join the existing transform metadata, and the existing <exclude>.yca/**</exclude> was not part of this change (see note below).

What has been done

• Added the four @HopMetadataProperty fields and the new test coverage.
• Added TableInputPartitioning and wired classic slicing into TableInput.
• Added SparkJdbcInputHandler, the SparkConst.TABLE_INPUT_PLUGIN_ID registration, explicit-handler support in HopPipelineMetaToSparkConverter, and the needed dependencies.xml paths so the Spark plugin can load the transform and JDBC drivers.
• Added the four fields to the Table Input GUI Options tab (inside a JDBC partitioning group), locale keys for the non-empty bundles, and user-manual documentation.
• Split Spark tests into guard unit tests and H2 integration tests (SparkJdbcInputHandlerIT), and asserted the new handler registration in the converter test.
• Classic-engine and Spark partitioned-read integration tests.

Verification

• mvn -pl plugins/transforms/tableinput,plugins/engines/spark test (unit tests) and the H2 integration tests pass on JDK 21.
• The classic partitioned-pipeline IT and the Spark H2 IT both pass.
• End-to-end: built the Hop 2.21.0-SNAPSHOT client, ran a partitioned H2 pipeline with hop-run.sh -r spark-smoke on the native Spark engine (4 rows read), and confirmed the log shows the native JDBC partitioned read.
• apache-rat:check passes (.yca/** was added to the RAT exclusion list; this was an environment concern unrelated to the feature but is included in the diff but until RAT-578: Support core.excludesFile from global git config creadur-rat#746 is merged there is no real end user workaround for it).

Note: this branch was cut from the current main; the fork's main is behind, so the PR head is this new branch.

[bamaer]

Thanks @rmannibucau, a native partitioned JDBC read is a real win for the Spark engine. The way extra copies read an empty range on the classic engine is neat too.

We built tableinput and engines/spark from this branch on JDK 21 and ran their unit tests and ITs: all green. We also ran a few extra pipelines through HopPipelineMetaToSparkConverter on a local SparkSession against H2.

Blockers

1. Existing Table Inputs on Spark change behaviour, even without partitioning. Registering TableInput as a native handler (HopPipelineMetaToSparkConverter.java:90, :193) takes every Table Input away from the converter's generic handler:

   • With more than one main hop into a Table Input, the converter now passes only the first input (:239). Two parameter sources (SELECT 5, SELECT 200) into SELECT VAL FROM T WHERE ID = ? give 1 row on this branch and 2 rows on main.
   • The internal fallback handler never gets the sample-data accumulator or the sampling context. setSampleDataAccumulator and setExecutionSamplingContext only reach the converter's own generic handler, so Table Input no longer produces execution data samples.

   Suggested fix: send a Table Input to SparkJdbcInputHandler only when partitioning is set, and keep every other Table Input on the converter's generic handler. With that change the two-hop case returns 2 rows again.

2. On Spark, a default Table Input never gets the native read, and the fallback then fails. useNamedParameters is true by default, so isNativePartitionedRead (SparkJdbcInputHandler.java:90) falls back unless someone unticks that option. The fallback runs a single copy, and the new check at TableInput.java:507 stops it: Table Input requires at least 4 copies for the configured JDBC partitions; got 1. The dialog's Preview also runs a single copy, so it hits the same error.

   Suggested fix: fall back only when the SQL actually contains named parameters (TableInputSql.parse(sql).hasNamedParameters()), and read the whole result (no slicing) when the transform runs as a single copy. With both changes, a default Table Input reads natively with 4 partitions.

Should fix

3. Insert data from transform is not detected. Info hops are not part of previousTransforms, so the input != null check (SparkJdbcInputHandler.java:122) misses them and the native read runs the SQL without its parameters: Parameter "#1" is not set. Adding Utils.isEmpty(meta.getLookup()) to isNativePartitionedRead would cover it.
4. Unrelated .yca exclusion. The lines in pom.xml:567 and .gitignore:41 are for a local tool, not this feature. Could you drop them from this PR?

[bamaer]

Thanks @rmannibucau for the quick turnaround, the four review items are resolved. We rebuilt d1caaab, ran the module tests and ITs (all green) and re-ran our extra pipelines. Two parameter hops into a Table Input give 2 rows again, and a default Table Input now takes the native read with 4 partitions.

We should have asked for integration tests in our first review and didn't, sorry about that. We're adding them now as a blocker, because this change touches the most-used transform and the Spark engine for every pipeline.

Blocker

• Integration tests. The parts most likely to break are outside what the unit tests and in-JVM ITs can see: the Spark plugin classpath (lib/jdbc and tableinput in dependencies.xml), running from an assembled client, multi-copy .hpl pipelines, and a database other than H2. Concretely, we'd need:
  • integration-tests/database: a 0008-table-input-partitioned.hpl with 4 copies and the four partition options set, validated against the full row set (no missing or duplicate rows, nulls included). Wire it into main-0008-table-input.hwf or a new main workflow. The existing 0008-table-input-* pipelines show the pattern.
  • integration-tests/spark-native: the same read run through hop-run on the native Spark engine, validated the same way. That project has no database service yet, so it needs an H2 file database or the Postgres service from the database project.

Should fix

• Pipeline Preview and Debug show every row once per copy. TableInput.java:511-512 reads the whole result when copies == 1 || preview. Hop GUI's Preview and Debug on the pipeline graph run the real pipeline in preview mode, with its real copy count. A partitioned Table Input with 4 copies then shows each row 4 times, and anything downstream (counts, lookups) is off in the preview. multiCopyPreviewReadsWholeTablePerCopyLikeUnpartitioned asserts exactly that. The dialog Preview already runs a single copy, so copies == 1 alone covers it. Dropping || preview keeps the slices in a multi-copy preview.