fix(snowflake): convert Decimal columns to float64 instead of object dtype#11994
Open
fadil4u wants to merge 2 commits intoibis-project:mainfrom
Open
fix(snowflake): convert Decimal columns to float64 instead of object dtype#11994fadil4u wants to merge 2 commits intoibis-project:mainfrom
fadil4u wants to merge 2 commits intoibis-project:mainfrom
Conversation
Contributor
|
see the linked issue, I'm not yet convinced this is a bug |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
When querying a Snowflake table with
NUMBER(p, s)columns where scale > 0 (e.g.NUMBER(38, 8)), the resulting pandas DataFrame hasobjectdtype instead offloat64: see #11990Snowflake returns
NUMBER(38, 8)asdoublein Arrow format, so the data is alreadyfloat64after_fetch_from_cursor. However__pandas_result__was defaulting to plainPandasDataas thedata_mapper, which runsnormalize_decimalon every value — wrapping eachfloat64in a Pythondecimal.Decimalobject and producingobjectdtype.Fix
execute()in the Snowflake backend to passdata_mapper=SnowflakePandasDatato__pandas_result__convert_DecimaltoSnowflakePandasDatathat casts tofloat64instead of runningnormalize_decimalIssues closed