Large update for Speed, Robustness and Pre-reduced focussed updates. - #1373
Large update for Speed, Robustness and Pre-reduced focussed updates. #1373mfitzasp wants to merge 119 commits into
Conversation
|
And of course, as soon as I make a pull request I find a couple of other things... but minor things. |
Also have some changable values to reject a handful of bad wcs frames if the majority of the dataset already has wcs.
|
This looks great. Thank you for the detailed work. Questions:
^ Questions above are largely geared toward integration and compatibility. They are not blockers, if there are indeed breaking changes, well, we can figure out some way to work around them. Probably an important aspect is checking the AAVSO specs to see if the updates are matching. If not, then there are ways around that, too. |
|
Howdy John! When I started the PR, there was one kinda minor one from Tamim I think and that was the only difference between main and dev. Since I posted mine, theres been a couple of other pull requests appear. I would say pause on the pull request for the moment. There is probably a bunch of reasons... ,,, not cause there is anything wrong with this pull request persay or the interaction with AAVSO and CITISENS and EXOTIC.... ... more that I started to type out all the things... and it took a little while!! I'd cleared up a bunch of things and thought "right! now lets just test it and be able to robustly confirm the new stuff is good", so I got a lot of transits from Kalee (160 in total and I have a bunch fo my own). So I constructed ... "THE EXO-GAUNTLET"... but the first transit was so plagued by bad pixels so I had to incorporate an auto-bad-pixel detector... which you can actualy do pretty reliably across a set of hundreds of images... then the second transit was horrendously poorly flatfielded... so then... similarly another approach. So currently I am at 2 out of 160 datasets through the gauntlet! I am intending to set up a test set of data (160 transits) where you can set up a run of exotic through all of them and they should reliably get through all of them.... but as stated, I am up to 3 out of 160! Yes, it is big. But I think worthwhile. Once I make it through the 160 successfully, I can make a good solid boringly technical one hour presentation of whats happened. |
|
There are quite a bunch of gremlins in there as well. Happy to jump on zoom if you want to talk through it. |
It actuall looks great. I've been jonesing to optimize ELCA and to upgrade some of the UN stuff, at bare minimum. The filtering needs work, too, so any improvements are so greatly appreciated! Perhaps we can dig into stuff a bit? My time is tied up for a couple weeks, at least, but I can cheer and comment from the sidelines... Question: If we merge the other PRs can you integrate them during testing? |
we moved to using memmaps, but it wanted an image in memory for the plots.
…efore its end estimate_ephemeris_tmid_and_bounds used floor(phases).max(), which selects the transit at or before the last valid frame. For ingress-only partial transits, where the true mid falls after the last surviving frame, that centers the Tmid search bounds one full period early; the periodic transit model fits the data perfectly at the wrong epoch and the reported absolute Tmid is exactly one period early, while every phase-folded plot looks correct. Fixed-window observations that cut egress hit exactly this geometry, and whether the bug fires depends on which edge frames survive quality rejection. round(median(phases)) selects the epoch nearest the bulk of the data: identical for full transits and one-sided baselines near the mid, correct for the ingress-only case, and insensitive to edge-frame rejection. Regression test uses the geometry of the 2026-07-27 TOI-1516 b MicroObservatory night where two independent WBoM reductions reported the same Tmid one period early (issue #1387). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
WBoM: select the transit epoch nearest the data, fixing the one-period-early Tmid on ingress-only nights
The one where we re-use old valid points in a re-weighted fashion.
3 participants
Ok, this is a big one. Kalee has tested tens of transits both original and this version. I've used it for tens and theres been a few other users who have used it.... only found a couple of bugs over time. I will be around to catch any more that may arise... I will be using EXOTIC a lot from now... and by "I", I means hordes of students.
Unfortunately I was working on it locally and my commits are quite large! But I am pretty dang sure they are fine at this stage. My major motivation was the make it more robust (fallback options for brittle online services) and faster and actually fixes up a few issues in the pre-reduced area in particular that was less well serviced over time. I found a few clangers in there and there are a bunch of feature-unchanging speedups (e.g. using bottleneck rather than numpy for nan-aware array operations and stopping analysis when the target star is outside the field of view... previously it still analysed the whole frame even though it was futile which is one reason why some runs blew out to 2 hours).
Heres the broad list of changes. I found a couple of small bugs later on as well.
Major new external-service architecture:
Added NextAstro plate-solving implementation (new source extraction, brightest-source capping, async polling, WCS writeout).
Added robust fallback routing both directions: NextAstro -> nova and nova -> NextAstro.
Added RA/Dec and pixel-scale astrometry hints to uploads.
Added NextAstro variability API integration and JSON request/response logging.
Added fallback from VSX query failures to NextAstro variability checks.
Added NASA Exoplanet Archive fallback via NextAstro cached exoplanet parameters.
Runtime/CLI feature additions:
New flags for NextAstro astrometry, NextAstro variability server, non-interactive mismatch handling, multiprocessing transformations, multiprocessing candidate fits.
Better non-interactive coordinate mismatch behavior (auto-proceeds with WCS-derived coords).
FITS/WCS and transformation pipeline improvements:
Added optional header-WCS ignore override (force legacy manual alignment path).
Added WCS coverage precheck to skip unnecessary transformation precompute when all files have celestial WCS.
Added transformation fast path and staged fallbacks (FFT-first plus optimized fallback behavior).
Added multiprocess precompute with worker cap (8) and clearer progress diagnostics.
Reduced warning spam and improved plate-status handling.
Photometry and model-fitting quality/performance:
Added fast centroiding path with periodic full-fit cadence.
Improved out-of-frame handling and fail-fast behavior for projected stars outside frame.
Added coarse+refined aperture-grid auto-tuning.
Added comparison-star calibration scoring (pairwise/ensemble stability metrics), with new calibration plots and CSV summary.
Added target-driven vs comp-driven comparison selection toggle.
Added optional strict comparison-star requirement (require_comp_star).
Added prescoring and multiprocessing for candidate lightcurve fit search.
Added relative-flux high-value rejection and phase-binned sigma clipping + refit.
Added robust timing summaries for transformation/photometry/reduction stages.
Input/config behavior upgrades:
Added new inits.json controls: observatory full title, fast aperture mask toggle, ignore header WCS toggle, target-driven comp-selection toggle, require comp-star toggle, pixel scale hint field, LD uncertainty option.
Improved camera type handling (accept alternate keys, map CMOS to CCD behavior, DSLR special-casing).
Pre-reduced route became less brittle:
No longer relies on manual comparison-star entry.
Auto-parses AAVSO metadata for comp star/location/obsdate when available.
Derives observation date from data timestamps when needed.
Allows optional lat/long/elev in pre-reduced mode.
Output/schema changes:
AAVSO header output now writes OBSDATE, optional OBSNAME, and observatory location fields (OBSLAT/OBSLON/OBSELEV).
Filter serialization switched from simple fwhm list to structured filter_width object.
Added comp-star calibration summary export.
Filter/LD updates:
Expanded filter alias set substantially (short forms and clearer canonicalization).
Added/clarified “ClearV” handling and clearer filter naming/width specs.
Limb darkening lookup behavior refined (better alias precedence and uppercase shorthand handling).
Ultranest UX:
Introduced ultranest_utils.py with explicit progress modes (simple heartbeat default, rich opt-in, silent mode).
Integrated wrapper across fitters (elca, ephemeris, joint_fitter, nbody, nested_linear_fitter, rv_fitter).
Reduced ultranest logging noise.
Testing/documentation/versioning:
Added/expanded test coverage across NextAstro astrometry/variability, centroid/WCS, proper-motion handling, input parsing, output headers, plotting masks, NEA fallback, ultranest progress behavior.