Skip to content

feat: support integer dtypes in ListMax - #74

Open
dummy1cx wants to merge 3 commits into
ExpediaGroup:mainfrom
dummy1cx:kamae-listmax-int-dtypes
Open

feat: support integer dtypes in ListMax#74
dummy1cx wants to merge 3 commits into
ExpediaGroup:mainfrom
dummy1cx:kamae-listmax-int-dtypes

Conversation

@dummy1cx

@dummy1cx dummy1cx commented Aug 12, 2026

Copy link
Copy Markdown

Description

Widens ListMax to accept integer inputs, and fixes a latent bug in the
min_filter_value code path that this exposed.

1. Integer dtypes

ListMaxTransformer / ListMaxLayer previously accepted only float and string
dtypes. Taking a listwise max over an integer column is a natural operation, but
it required an upstream cast to double purely to satisfy the dtype check. This
adds ByteType/ShortType/IntegerType/LongType on the Spark side and
int8/int16/int32/int64 on the Keras side.

The motivating use case is count features (e.g. number of properties, number of
searches) stored as bigint, where we want the max within a query/segment while
keeping the column integral.

2. Bug fix in the min_filter_value path

ListMaxLayer masks filtered-out values with dtype.min and then substitutes
nan_fill_value for any segment that ends up empty:

fill_val = tf.constant(self.nan_fill_value, dtype=listwise_max.dtype)

nan_fill_value is a Python float, and tf.constant(0.0, dtype=tf.int64) raises
TypeError: Cannot convert 0.0 to EagerTensor of dtype int64. So simply widening
the dtypes would have made the layer crash whenever min_filter_value was set on
an integer column. This was unreachable before this PR, since integer inputs were
rejected by the dtype check.

The value is now narrowed through numpy before constructing the tensor:

fill_val = tf.constant(
    listwise_max.dtype.as_numpy_dtype(self.nan_fill_value),
    dtype=listwise_max.dtype,
)

Note that tf.cast(self.nan_fill_value, listwise_max.dtype) is the more obvious
fix, but it silently loses precision on float64: a Python float is converted to
a float32 tensor first and then widened, so a fill value of 123.456 comes back
as 123.45600128173828.

I compared the numpy narrowing against the previous tf.constant behaviour over
~4000 values per dtype, comparing raw bytes rather than numeric equality.
float64 is identical throughout. The only differences anywhere are:

  • -0.0, which the old path normalised to +0.0 and the new path preserves.
    The two compare equal, so there is no numeric impact.
  • Subnormal values in bfloat16 (e.g. 1e-38), which the old path flushed to
    zero and the new path represents.

Both are cases where the new behaviour is at least as faithful as the old, and
neither is reachable with a realistic fill value, so existing users are
unaffected.

Testing

Every integer dtype added here is covered on both backends:

  • Keras layer cases for int64 with segmentation, and for each of int8,
    int16, int32 and int64 with segmentation plus a min_filter_value that
    empties a segment entirely. That last scenario is the one that exercises the
    nan_fill_value fix, so it is covered for every width rather than just one.
  • Spark/TensorFlow parity cases mirroring the same scenarios across tinyint,
    smallint, int and bigint, confirming the two backends agree on integers
    including when the fill value is applied.
  • A dedicated test asserting the fill value is applied exactly, covering
    float64 and float32 with values that are not exactly representable in
    float32, plus int64 and int32. This fails against the tf.cast approach
    described above and guards the float precision behaviour.
  • Full suite passes locally: 1684 passed, 1 skipped.

Checklists

ListMax is an existing transformer, so most items in the new-transformer
checklists do not apply. The two that are relevant:

  • The compatible_dtypes property has been implemented to specify the
    input/output data types that my transformer/estimator supports.
  • There are unit tests of the new transform. In particular, there are parity
    tests between the Spark and Keras implementations.

No README entry, as this is not a new transformer.

Note for reviewers

Was the original float-and-string restriction deliberate? I could not find a
reason for it in the code or history, and the underlying ops
(tf.math.unsorted_segment_max, F.max) both handle integers, but happy to
adjust if there was intent behind it.

@dummy1cx
dummy1cx requested a review from a team as a code owner August 12, 2026 11:39

@georyetti georyetti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we have int support (especially int8) the neg_inf pattern is not robust anymore. Hoping we can refactor quickly to a mask pattern

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

neg_inf for int8 is actually -128. Which could reasonably be a valid value.

Wondering if we can refactor this more generally and just keep the boolean mask and reuse it later.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're doing this kind of refactor to use the mask rather than setting a neg_inf value, shouldn't that be something we do more generally as a consistent pattern across all transformers @georyetti?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe for now we could stick to higher precision int types for this one

jamesdshinner
jamesdshinner previously approved these changes Aug 27, 2026
The layer decided whether a filter had emptied a segment by testing the
reduction result against dtype.min, the neutral element it substitutes for
filtered entries. That reads as a sentinel, but on the narrow integer dtypes
it is ordinary data: a segment whose genuine maximum is -128 on int8 was
indistinguishable from one the filter had emptied, and the real value was
overwritten with nanFillValue.

Track which entries survive the filter in a mask and reduce that alongside the
values, so emptiness is answered directly instead of inferred from the result.
This mirrors the approach taken in ListSum.

Comparing an integer tensor against the float min_filter_value also raised a
TypeError, so integer thresholds are now narrowed explicitly. Rounding up
leaves >= unchanged on integers, and a threshold beyond the bounds of the
dtype is resolved without a cast, which would otherwise wrap around and
silently invert the comparison. The shared min_filter_mask helper holds this
logic so ListSum can adopt it.

Co-authored-by: Cursor <cursoragent@cursor.com>
@georyetti

Copy link
Copy Markdown
Contributor

@dummy1cx Can you condense this to a summary please? Ideally without the use of AI

@dummy1cx

Copy link
Copy Markdown
Author

Hey George! I have swapped the neg_inf check with a mask in the layer, so it keeps track of which values passed the filter and checks that at the end. Now a real -128 value comes back as the max instead of being replaced with nan_fill_value.

But while testing I got some bugs in the code, as filtering an int column was throwing a TypeError because min_filter_value is a float, so the threshold gets converted now. My first go at that wrapped around when the threshold was outside the dtype range (200 becomes -56 on int8), so those are handled separately without a cast.

I have run the test files covering the Spark/TF match, and the full suite passed locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants