[Data] ray.data.from_tf() fails on TensorFlow datasets with ragged tensors#62704
[Data] ray.data.from_tf() fails on TensorFlow datasets with ragged tensors#62704weimingdiit wants to merge 1 commit into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates ray.data.from_tf to handle tf.RaggedTensor by replacing the as_numpy_iterator call with a manual conversion loop. While this enables support for ragged tensors, the feedback identifies a performance regression for standard datasets and a potential memory issue caused by converting tf.SparseTensor to dense arrays. It is suggested to maintain the optimized iterator path for non-ragged datasets and preserve sparse representations where possible.
weimingdiit
force-pushed
the
data/from-tf-ragged-tensor-support
branch
from
98972b9 to
81c6ab6
Compare
…nsors Replace the as_numpy_iterator-based conversion path in from_tf() with direct TensorFlow dataset iteration so ragged tensor inputs are handled correctly. Add a regression test covering tf.RaggedTensor datasets. Signed-off-by: weimingdiit <[email protected]>
weimingdiit
force-pushed
the
data/from-tf-ragged-tensor-support
branch
from
81c6ab6 to
82e6c28
Compare
ray-gardener
Bot
added
data
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
github-actions
Bot
added
the
stale
|
hey @weimingdiit - sorry for the late reply. Taking a quick look now. |
richardliaw
added
go
2 participants
Description
ray.data.from_tf()currently relies ondataset.as_numpy_iterator(), which raises when the TensorFlow dataset containstf.RaggedTensorvalues.This PR fixes that by iterating over the TensorFlow dataset directly and recursively converting TensorFlow values into Ray-compatible Python / NumPy values.
The change includes:
tf.RaggedTensorvia Ray's existing ragged tensor utilitiestf.SparseTensorand densetf.Tensorray.data.from_tf()with ragged tensor inputsRelated issues
Closes #62703
Additional information
This change does not modify the public API of
ray.data.from_tf(). It only fixes the conversion path so TensorFlow datasets with ragged tensors can be ingested correctly.