Skip to content

NaNs Are Returned by non NaN Downsamplers #73

Open

Description

@mike-iqmo

When using the non NaN version of the samplers, I am getting NaNs in the downsampled data.

My understanding from the documentation was that MinMaxLTTBDownsampler would omit all NaN values.

Some code demonstrating this below

n=10_000
y = np.arange(n, dtype=np.float64)
for i in range(1,100):
    y[i+100] = np.nan

sampled=MinMaxLTTBDownsampler().downsample(y,n_out=1000)
print(f"MinMaxLTTBDownsampler:{[i for i in sampled if np.isnan(y[i])]}")

sampled_nan=NaNMinMaxLTTBDownsampler().downsample(y,n_out=1000)
print(f"NaNMinMaxLTTBDownsampler:{[i for i in sampled_nan if np.isnan(y[i])]}")

That will print

MinMaxLTTBDownsampler:[101, 111, 121, 131, 141, 151, 161, 171, 181, 191]
NaNMinMaxLTTBDownsampler:[101, 111, 121, 131, 141, 151, 161, 171, 181, 191]

Activity

  1. changed the title [-]NaNs Are Always Returned[/-] [+]NaNs Are Returned by non NaN Downsamplers[/+] on May 15, 2024
  2. jvdd commented on Jan 31, 2025

    @jvdd
    Collaborator

    Good catch! The code didn't handle the case where the bin is full of NaNs (as is the case in your example).

    #83 now prunes NaN indices after downsampling (in Rust) for the omit NaN downsamplers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions