2023.3

[javierggt]

ska3-flight 2023.3

This PR includes:

• backstop_history:
  • Incorporate new RTS files for new Long Term ECS SI modes
• chandra_aca:
  • Read grid acq probability model data from chandra_models repo
  • Allow for glob specification of acq prob model
• agasc supplement processing: use kadi commands v2
• cheta:
  • Enable use of MAUDE with computed quaternion pseudo-MSID
  • Improve pitch/roll_comp, logical_intervals, state_intervals, get_ofp_states, get_telem_table
• remove use of deprecated timelines package in several packages
• starcheck
  • Update for expanded high-IR zone
  • Use the dynamic background tccd bonus
  • Stop using maneuver summary (now calculates maneuver duration and uses AOMANVR Backstop cmd as start)
• ska_helpers:
  • Add more support for chandra_models handling.

Interface Impacts:

Should be transparent to the user - the ska_helpers changes that read chandra_models from the git repository also include a change to add /proj/sot/ska/data/chandra_models to the user's .gitconfig file as a trusted/safe repository if the user is running on a system where /proj/sot/ska/data/chandra_models is owned by another user (this is true of HEAD and GRETA systems).

Data that needs to be updated after promotion:

• agasc rc/*.fits files for weekly processing

Testing:

• HEAD.
• GRETA.
• OSX.

skare3 dashboard and test result password at https://icxc.cfa.harvard.edu/aspect/skare3_dash_cred.txt

The latest release candidates will be installed in /proj/sot/ska3/test on HEAD, and all release candidates will be available for testing from the usual channels:

conda create -n ska3-flight-2023.3rc9 --override-channels \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/test \
  ska3-flight==2023.3rc9

Review

All operations critical or impacting PR's are independently and carefully reviewed. For other PR's the level of detail for review is calibrated to operations criticality. Some PR's that are confined to aspect-team-specific processing may have little to no independent review.

Deployment

ska3-flight 2023.3 will be promoted to flight conda channel and installed on HEAD and GRETA Linux upon approval of FSDS Jira ticket.

Code changes

ska3-core changes (2023.1 -> 2023.3rc9)

Updated Packages

• astropy: 5.2.1 -> 5.3.1

ska3-core-latest changes (2023.1 -> 2023.3rc9)

Removed Package

• prompt-toolkit

ska3-flight changes (2023.1 -> 2023.3rc9)

Updated Packages

• aca_weekly_report: 0.1.7 -> 0.2.0 (0.1.7 -> 0.2.0)
  • PR 12 (Jean Connelly): Fix warning for obsid 0 at 2023:044
  • PR 11 (Tom Aldcroft): Remove timelines dependency
• acdc: 4.9.1 -> 4.10.1 (4.9.1 -> 4.10.0 -> 4.10.1)
  • PR 69 (Jean Connelly): Add a dark current caldb process
  • PR 73 (Tom Aldcroft): Improve error message for not enough quads
• agasc: 4.14.1 -> 4.15.0 (4.14.1 -> 4.15.0)
  • PR 135 (Javier Gonzalez): Use commands v2 for AGASC supplement update
  • PR 151 (Javier Gonzalez): Fix process failures
• aimpoint_mon: 1.1.1 -> 1.1.2 (1.1.1 -> 1.1.2)
  • PR 25 (Tom Aldcroft): Fix some residual issues in the Py3 implementation
• backstop_history: 3.1.1 -> 3.2.1 (3.1.1 -> 3.2.1)
  • PR 25 (None): Incorporation of new RTS files for new Long Term ECS SI modes
• chandra_aca: 4.38.3 -> 4.41.0 (4.38.3 -> 4.39.0 -> 4.40.0 -> 4.41.0)
  • PR 140 (Tom Aldcroft): Read grid acq probability model data from in chandra models repo
  • PR 144 (Javier Gonzalez): Fix maude_decom.get_raw_aca_packets
  • PR 145 (Javier Gonzalez): Update pre-commit config and flake workflow
  • PR 147 (Javier Gonzalez): renamed BGDTYP/PIXTLM to AABGDTYP/AAPIXTLM
  • PR 149 (Tom Aldcroft): Allow for glob specification of acq prob model + documentation overhaul
  • PR 148 (Javier Gonzalez): Prevent crash when IMGTYPE==3 when using blobs
  • PR 151 (Tom Aldcroft): Add function to get default acq prob model info
  • PR 150 (Tom Aldcroft): Use chandra_models_cache for caching not lru_cache
• chandra_time: 4.1.1 -> 4.1.2 (4.1.1 -> 4.1.2)
  • PR 58 (Tom Aldcroft): Use chandra_time modules for Chandra.Time
• cheta: 4.58.0 -> 4.59.0 (4.58.0 -> 4.59.0)
  • PR 248 (Tom Aldcroft): Add overwrite=True to cheta sync index table write statement
  • PR 246 (Tom Aldcroft): Enable use of MAUDE with computed quaternion pseudo-MSID
  • PR 249 (Tom Aldcroft): Improve pitch/roll_comp, logical_intervals, state_intervals, get_ofp_states, get_telem_table
  • PR 251 (Tom Aldcroft): Fix misleading doc string
• cxotime: 3.4.0 -> 3.6.0 (3.4.0 -> 3.5.0 -> 3.6.0)
  • PR 37 (Tom Aldcroft): Update pre-commit yaml to make it work again
  • PR 34 (Tom Aldcroft): Remove CxoTime custom fast parser, use astropy built-in
  • PR 36 (Tom Aldcroft): Improve time conversion helpers
  • PR 38 (Javier Gonzalez): Fix test on Windows
  • PR 33 (Tom Aldcroft): Add generic fast format converters for better performance
• fot-matlab: 2.1.1 -> 2.2.0 (2.1.1 -> 2.1.2 -> 2.2.0)
  • PR 19 (James Kristoff): MATLAB-11938
  • PR 20 (James Kristoff): Matlab 11984
• kadi: 7.3.0 -> 7.4.1 (7.3.0 -> 7.4.0 -> 7.4.1)
  • PR 269 (Tom Aldcroft): Add task_schedule_validate.cfg
  • PR 282 (Tom Aldcroft): pre-commit autoupdate
  • PR 284 (Tom Aldcroft): Ensure that timezone TZ env var is unmodified by django.setup()
  • PR 285 (Tom Aldcroft): Factor out get_ofp_states and get_telem_values for cheta PR 249 improvements
  • PR 286 (Javier Gonzalez): fix test on windows
  • PR 288 (Jean Connelly): Compare regression test quaternions with delta-quat tolerance check
  • PR 287 (Tom Aldcroft): Improve error message for stale local commands archive
• mica: 4.31.3 -> 4.33.0 (4.31.3 -> 4.32.0 -> 4.33.0)
  • PR 279 (Tom Aldcroft): Remove timelines / cmd_states.db3 from mica
  • PR 283 (Jean Connelly): Update CDA code and tests for DS10.12
• parse_cm: 3.9.1 -> 3.10.0 (3.9.1 -> 3.10.0)
  • PR 39 (John ZuHone): Routine to read the ACIS tables
• proseco: 5.8.1 -> 5.9.0 (5.8.1 -> 5.9.0)
  • PR 381 (Jean Connelly): Pin chandra_models version to 3.48 for all tests
  • PR 382 (Tom Aldcroft): Add acq prob model info dict as a new pickled attribute
• ska3-core: 2023.1 -> 2023.3
• ska_helpers: 0.8.0 -> 0.10.4 (0.8.0 -> 0.9.0 -> 0.9.1 -> 0.10.0 -> 0.10.2 -> 0.10.3 -> -> 0.10.4)
  • PR 29 (Tom Aldcroft): Add environment module with new function configure_ska_environment()
  • PR 33 (Tom Aldcroft): Add support for two new env vars for chandra_models defaults + get_data info
  • PR 34 (Tom Aldcroft): Fixes to get chandra_models and paths working on Windows
  • PR 35 (Tom Aldcroft): Add git_helpers.py w/ function for safe directory handling
  • PR 36 (Tom Aldcroft): Chandra models caching and a generic LRUDict class (evict least recently used entries)
  • PR 37 (Jean Connelly): Add the repo dir to safe config if used directly
  • PR 38 (Tom Aldcroft): Remove the check on repo tip being at the latest tag
  • PR 39 (Javier Gonzalez): clear git_helpers.make_git_repo_safe cache in test
  • PR 41 (Jean Connelly): Remove auto git repo safe handling
  • PR 42 (Jean Connelly): Replace git_helpers subprocess with gitpython and revert PR 41
• ska_matplotlib: 4.0.0 -> 4.0.1 (4.0.0 -> 4.0.1)
  • PR 28 (Jean Connelly): Add a note to cxctime2plotdate about CxoTime
• ska_shell: 4.0.0 -> 4.0.1 (4.0.0 -> 4.0.1)
  • PR 28 (Jean Connelly): Rename test setup() setup_method()
• ska_sun: 3.9.1 -> 3.10.1 (3.9.1 -> 3.10.0 -> 3.10.1)
  • PR 28 (Tom Aldcroft): Fix a test fail from numerical issues
  • PR 27 (Tom Aldcroft): Change allowed_rolldev to use chandra_models
  • PR 31 (Tom Aldcroft): Update for new ska_helpers.chandra_models
  • PR 32 (Tom Aldcroft): Allow repeated pitch vals and make unit tests work after chandra_models updates
• skare3_tools: 1.0.9 -> 1.0.10 (1.0.9 -> 1.0.10)
  • PR 95 (Javier Gonzalez): Remove use of --tags option to git
  • PR 94 (Javier Gonzalez): Fixes and improvements in Ska dashboard
• sparkles: 4.21.0 -> 4.22.2 (4.21.0 -> 4.22.0 -> 4.22.1 -> 4.22.2)
  • PR 182 (Jean Connelly): Use creep-away to downgrade guide count and cluster critical warns
  • PR 183 (Jean Connelly): Check for 4x4 dither for 3.5 to 4.0 guide_count
  • PR 185 (Jean Connelly): Handle updated pitch roll table in roll optimizations
  • PR 186 (Tom Aldcroft): Fix pickling
  • PR 187 (Tom Aldcroft): Apply black formatting and add config files
  • PR 189 (Jean Connelly): Add dyn-bgd-n-faint option for review
  • PR 188 (Tom Aldcroft): Add linting with ruff and add workflow for black + ruff
  • PR 192 (Tom Aldcroft): Fix issue computing t_ccds_bonus and refactor into function
  • PR 194 (Jean Connelly): Pin chandra_models version to 3.48 for tests
• starcheck: 14.0.2 -> 14.2.1 (14.0.2 -> 14.0.3 -> 14.1.0 -> 14.2.1)
  • PR 405 (Tom Aldcroft): Apply perltidy to starcheck
  • PR 408 (Jean Connelly): Calculate maneuver times and angle from cmds, replace maneuver summary file
  • PR 416 (Jean Connelly): Ignore mag clipping chandra_aca warnings
  • PR 413 (Jean Connelly): Update for expanded high-IR zone
  • PR 418 (Jean Connelly): Add a unit test for dyn bgd hotpix imposter calc
  • PR 412 (Jean Connelly): Use dyn_bgd_n_faint = 2
  • PR 419 (Jean Connelly): Use ccd_temp instead of t_ccd from dark cal data
  • PR 421 (Jean Connelly): Handle vehicle mode with custom filtered backstop
• xija: 4.29.3 -> 4.30.0 (4.29.3 -> 4.30.0)
  • PR 133 (Tom Aldcroft): Use ska_helpers.paths for model paths
  • PR 124 (Tom Aldcroft): Apply black and isort formatting and modernize config files

Related Issues

Fixes #1089
Fixes #1092
Fixes #1111
Fixes #1112
Fixes #1114
Fixes #1115
Fixes #1116
Fixes #1118
Fixes #1119
Fixes #1120
Fixes #1121
Fixes #1122
Fixes #1123
Fixes #1124
Fixes #1125
Fixes #1126
Fixes #1127
Fixes #1129
Fixes #1135