Skip to content

XRT DEM Iterative Solver Module - #367

Open
joyvelasquez wants to merge 259 commits into
HinodeXRT:mainfrom
joyvelasquez:xray-plasma-decoder-dem-solver
Open

XRT DEM Iterative Solver Module#367
joyvelasquez wants to merge 259 commits into
HinodeXRT:mainfrom
joyvelasquez:xray-plasma-decoder-dem-solver

Conversation

@joyvelasquez

@joyvelasquez joyvelasquez commented Jul 16, 2025

Copy link
Copy Markdown
Contributor

Summary

This PR introduces XRTDEMIterative, a Python implementation of the IDL
routine xrt_dem_iterative2.pro for estimating Differential Emission
Measures (DEMs) from Hinode/XRT multi-filter observations.

What's Included

Core Solver (dem_solver.py)

  • XRTDEMIterative class with full DEM solving pipeline
  • Spline-parameterized DEM representation with iterative least-squares
    fitting via lmfit
  • Temperature response interpolation onto a regular log10(T) grid
  • User-supplied or default intensity uncertainty model:
    max(0.03 * intensity, 2 DN/s/pix), matching IDL default behavior
  • Monte Carlo uncertainty estimation via monte_carlo_runs parameter
  • Input validation with informative warnings (saturation, negative
    intensities, missing errors)
  • summary() method for diagnostic output

Plotting (dem_plotting.py)

  • plot_dem() — base DEM solution visualization
  • plot_dem_mc() — base DEM + Monte Carlo ensemble overlay

Tests

  • 70+ unit tests covering input validation, grid construction,
    response interpolation, spline system, residuals, solver, and
    Monte Carlo behavior
  • Currently - heavy work in progress (UPDATED JULY 2026)

Documentation

  • dem_overview.rst — introduction to DEM analysis with XRTpy
  • API reference for xrtpy.xrt_dem_iterative
  • Changelog entry in docs/changelog/0.6.0.rst

Validation -

  • Validated against IDL xrt_dem_iterative2.pro across multiple
    solar observation cases using Monte Carlo comparison
  • Currently - heavy work in progress (UPDATED JULY 2026)

@joyvelasquez joyvelasquez self-assigned this Jul 16, 2025
@joyvelasquez joyvelasquez added this to the 0.6.0 milestone Jul 16, 2025
@joyvelasquez

Copy link
Copy Markdown
Contributor Author

Ensure the function checks for minimum required inputs (filters, observed intensities, etc.) and validates that all observation values are non-zero before proceeding.

@codecov

codecov Bot commented Mar 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 2560 lines in your changes missing coverage. Please review.
✅ Project coverage is 36.96%. Comparing base (0ef8e36) to head (7d9e2a5).

Files with missing lines Patch % Lines
xrtpy/xrt_dem_iterative/test/test_dem_solver.py 0.00% 502 Missing ⚠️
xrtpy/xrt_dem_iterative/dem_solver.py 0.00% 386 Missing ⚠️
...rative/test/older_testing_methods/plots_results.py 0.00% 294 Missing ⚠️
...ive/test/older_testing_methods/idl_vs_xrtpy_dem.py 0.00% 265 Missing ⚠️
xrtpy/xrt_dem_iterative/test/plot_mc_comparison.py 0.00% 211 Missing ⚠️
...t_dem_iterative/test/test_dem_idl_comparison_mc.py 0.00% 168 Missing ⚠️
xrtpy/xrt_dem_iterative/test/utils_sav_io.py 0.00% 164 Missing ⚠️
...py/xrt_dem_iterative/test/plot_idl_vs_xrtpy_dem.py 0.00% 149 Missing ⚠️
...rative/test/older_testing_methods/utils_case_io.py 0.00% 117 Missing ⚠️
.../xrt_dem_iterative/test/test_dem_idl_comparison.py 0.00% 114 Missing ⚠️
... and 4 more

❗ There is a different number of reports uploaded between BASE (0ef8e36) and HEAD (7d9e2a5). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (0ef8e36) HEAD (7d9e2a5)
3 1
Additional details and impacted files
@@             Coverage Diff             @@
##             main     #367       +/-   ##
===========================================
- Coverage   87.97%   36.96%   -51.02%     
===========================================
  Files          26       40       +14     
  Lines        1855     4415     +2560     
  Branches      110      338      +228     
===========================================
  Hits         1632     1632               
- Misses        180     2740     +2560     
  Partials       43       43               

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread docs/gallery/data_processing/deconvolving.py
Comment thread docs/gallery/data_processing/remove_lightleak.py
Comment thread docs/gallery/data_processing/temperature_from_filter_ratios.py
Comment thread docs/gallery/instrument_response/effective_area.py
Comment thread docs/about_xrt.rst
Comment thread docs/dem_overview.rst
Comment thread docs/dem_overview.rst Outdated
Comment thread docs/dem_overview.rst Outdated
Comment thread docs/dem_overview.rst Outdated
monte_carlo_runs=0,
)

x.validate_inputs()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should validate_inputs() be changed to _validate_inputs()? Is it something that needs to be user-facing?

Comment on lines +72 to +74
minimum_bound_temperature=5.5,
maximum_bound_temperature=7.5,
logarithmic_temperature_step_size=0.1,

@namurphy namurphy Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
minimum_bound_temperature=5.5,
maximum_bound_temperature=7.5,
logarithmic_temperature_step_size=0.1,
**temperature_kwargs,

And define temperature_kwargs = {"minimum_bound_temperature": 5.5, ...} outside of the function, so that we can use keyword argument unpacking here?


x.create_logT_grid()

assert (x.logT[0], x.logT[-1], x.dlogT) == pytest.approx((5.5, 7.5, 0.1))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
assert (x.logT[0], x.logT[-1], x.dlogT) == pytest.approx((5.5, 7.5, 0.1))
np.testing.assert_allclose(
[x.logT[0], x.logT[-1], x.dlogT],
[5.5, 7.5, 0.1],
)

dem.validate_inputs()


def test_create_logT_grid():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a good use case for pytest fixtures to avoid having repeated assertions, but it's probably not worth the effort changing here.

x.create_logT_grid()
x._interpolate_responses_to_grid()

dem, modeled, chi2, result = x._solve_single_dem(

@namurphy namurphy Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎗️ I'm wondering if this would also be useful to think about adding upstream to the interface in sunkit-dem, especially if many DEM solvers are iterative. 🤔

temperature_responses=responses,
)
x.solve()
x.plot_dem()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎗️ Is there a way to do something like a regression test for figures?

Comment on lines +630 to +651
def test_init_rejects_negative_max_iterations():
"""max_iterations=-100 must raise ValueError.
Expected: ValueError because max_iterations must be a positive integer.
"""
filters = ["Al-poly", "Ti-poly"]
intensities = np.array([500.0, 800.0])
responses = generate_temperature_responses(filters, "2012-10-27T00:00:00")

with pytest.raises(ValueError, match="max_iterations must be a positive integer"):
XRTDEMIterative(filters, intensities, responses, max_iterations=-100)


def test_init_rejects_zero_normalization_factor():
"""normalization_factor=0 must raise ValueError.
Expected: ValueError because normalization_factor must be positive.
"""
filters = ["Al-poly", "Ti-poly"]
intensities = np.array([500.0, 800.0])
responses = generate_temperature_responses(filters, "2012-10-27T00:00:00")

with pytest.raises(ValueError, match="normalization_factor must be a positive"):
XRTDEMIterative(filters, intensities, responses, normalization_factor=0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might be able to use @pytest.mark.parametrize for some of these. 🤔

Comment on lines +794 to +795
filters = ["Al-poly", "Ti-poly", "Be-thin"]
intensities = np.array([100.0, 200.0, 300.0], dtype=float)

@namurphy namurphy Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these be defined outside of the test functions to reduce duplication.

joyvelasquez and others added 3 commits July 23, 2026 20:17
Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Removing space.

Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
joyvelasquez and others added 12 commits July 27, 2026 13:54
Removing space pt2.

Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Nice touch.

Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Co-authored-by: Nick Murphy <namurphy@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Code Improvement For general improvements to the codebase without adding new features. enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants