Hive 4.0.1 support

[lfrancke]

Description

We want to upgrade our Apache Hive we ship from version 4.0.0 to 4.0.1 as Hive 4.0.x is our LTS line.
Currently we mark 4.0.1 as experimental due to issues we encountered when we tried it.

Value

Hive 4.0.1 contains bug fixes and security patches that our customers need. Since we're committed to supporting the 4.0.x line for LTS reasons for at least until the 26.3 release, we need to provide the most stable version available in that line.

Dependencies

• Resolution of demo compatibility issue: fix: downgrade Hive version for the anomaly demo demos#173
• Resolution or workaround for Trino integration issue: Repeated ANALYZE fails on Hive metastore 4.0.x trinodb/trino#26214

Tasks

• Research: Analyze the two blocking issues in detail (2 day timeboxed)
  • Investigate the demo issue in fix: downgrade Hive version for the anomaly demo demos#173
  • Understand the Trino integration problem (Repeated ANALYZE fails on Hive metastore 4.0.x trinodb/trino#26214)
  • Determine if these are blockers or can be worked around
  • Present a plan on how to go forward for each of these before continuing any work
• Testing: Run tests
  • All existing demos must continue to work
  • All integration tests must work
• Documentation: Update operator documentation marking 4.0.1 as non-experimental

Acceptance Criteria

• Trino integration works properly or documented limitations/workarounds are provided
• All integration tests pass
• All demos using Hive work

[razvan]

There seems to be no easy workaround for any of these problems. A proper fix upstream requires the Iceberg catalog model to change as described below.

Spark/Iceberg/Hive

Investigate the demo issue in stackabletech/demos#173

Problem

When using Hive 4.0.1 for the hive-iceberg-metastore the Spark application spark-ad fails with the error:

$ k logs spark-ad-driver
...
2025-08-29T07:49:01,678 INFO [Thread-4] hive.metastore - Connected to metastore.
Traceback (most recent call last):
  File "/spark-scripts/spark-ad.py", line 13, in <module>
    spark.sql("CREATE TABLE IF NOT EXISTS prediction.ad.iforest (pickup_ts TIMESTAMP, pickup_minute_group VARCHAR(4), pickup_hour INT, pickup_year INT, pickup_month INT, pickup_dayofmonth INT, pickup_dayofweek INT, norides INT, total_bill DOUBLE, avg_bill DOUBLE, norides_lag INT, pred INT) USING iceberg")
  File "/stackable/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1631, in sql
  File "/stackable/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/stackable/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
  File "/stackable/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o77.sql.
: java.lang.RuntimeException: Failed to get table info from metastore ad.iforest
        at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:160)
...
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table'
...

This is problem with Iceberg and there is an open issue for it: apache/iceberg#12878

Workaround

Patch and build Hive 2 as described in this comment . Then replace the Hive jars shipped by Spark.

Spark 3.5 ships Hive 2.3.9, while Spark 4.0 ships Hive 2.3.10.

See this PR for an attempt to build Hive 2.

Trino/Hive

Understand the Trino integration problem (trinodb/trino#26214)

This comment explains the problem. Hive 4 added a new type for timestamp fields that is not supported by Trino (as of v476).

Workaround

???

[sbernauer]

Just throwing in spark.sql.hive.metastore.jars.path

Location of the jars that should be used to instantiate the HiveMetastoreClient

It maybe won't help, but I wanted to mention that property

[lfrancke]

Okay. My current plan would be: Continue shipping Hive 3 and 4 at the same time.

• Spark/Iceberg/Hive is not something we can untangle anytime soon even if we wanted to, and we'll need to jump to later Hive versions than 4 at some point
• Trino issue is something we might be able to fix upstream but that's also not going to be trivial

So, my suggestion: Follow the relevant upstream discussions and jump in and help if we can and in the meantime clearly document where Hive 3 is still needed and what works/doesn't work with Hive 4. If that is fine then I'd make two issues out of this:

• Update Hive to 4.0.1
• Document limitations

Opinions?

[sbernauer]

I think what would be great is to have a story on downgrading. E.g. users bump 3.6.3 -> 4.0.0 or 4.0.0 -> 4.0.1 and notice that things broke.
Can they downgrade again? I recall for some tools the database migration script would panic, not sure about HMS.
I think we should add docs on downgrading as part of this issue, WDYT?