We currently track machine downtime via DTM in intervals of time, but program time accounting is tracked in BTM as seconds in a given state per hour. This difference in time recording is problematic when correlating and generating reports. In addition, we allow BTM to provide competing downtime information vs DTM. The very small discrepancies on a given shift are tolerated on cross checks as a user friendly slop factor, but in aggregate across a large time period such as a run these tiny shift slops add up to noticeable differences. The goal of this issue (software change request) is to eliminate these discrepancies.
Tasks
-
Remove Internal Mode DOWN and Internal Mode FSD tracking from BOOM
BOOM is the meter/diagnostic tool that provides program information to BTM via EPICS CA. It currently tracks Internal Mode Downtime and FSDs, and subtracts this from program time. We need BOOM to instead simply report program time, unmodified by downtime and FSDs. DTM already tracks FSDs and Downtime more accurately in intervals.
-
Update BTM to monitor EPICS CA / BOOM for program changes and store the program in intervals (instead of seconds per hour)
This info will be exposed in BTM on a new first tab labeled State Timeline. These intervals will need to be editable to allow mistakes to be corrected (just as DTM allows). The currently selected program should be displayed prominently, followed by a paginated history table of past planned program executions, in chronological order.
-
Update BTM timesheets to calculate Internal Down based on Program from the state timeline combined with downtime from DTM This can be done by correlating intervals of program vs intervals of downtime. This allows us to remove some of the downtime related cross checks and ensures there are no longer discrepancies between BOOM measured downtime and DTM recorded downtime. This automatic correlation also alleviates the need to perform cumbersome manual program classification of downtimes in DTM, which would be especially clumsy when programs change during downtime.
Notes
A. Why do we need to differentiate between internal vs physics?
When we run reports later it is often important to subtract downtime from program time to determine amount of time machine was actually used as intended, on a machine mode basis. In particular, the Joule report metric research hours is actual time used during physics, which excludes internal mode. We could possibly change the way Physics Downtime is calculated going forward: correlate DTM vs BTM intervals to find downtimes that occurred during Physics mode. This would be more direct than figuring out Internal Down and subtracting it from Total Downtime.
B. How do we accommodate State Timeline modifications after a timesheet has already been started?
It will be helpful for Crew Chiefs to start a timesheet by first ensuring the program state timeline is correct. In most cases it will be a simple verification and no other action is needed. In the case when they forgot to click a button in BOOM or otherwise state is incorrect, they'll need to make corrections. Then, filling out their timesheet would proceed as normal. In the case of retroactive program changes after a timesheet has been submitted the measured values would change, but the reported values stay the same and would require manual intervention to fix. This is similar to the current situation.
C. How does the State Timeline compare to other schedules?
The NPES, PD Shift Plan, and Whiteboard are three schedules that inform the Crew Chief schedule, which up until now was found as the "Reported" row of seconds in a given state per hour inside a shift timesheet. The Crew Chief Schedule State Timeline is an alternative (and more useful format) of this schedule. Persisting the Crew Chief schedule in intervals may allow us to simplify timesheets in the future by removing overlapping metrics. In particular, the "Accelerator Program" section of timesheets could possibly be removed (or at least automatically calculated). The Actual Operations Program intervals can be calculated by subtracting unscheduled downtime from the Crew Chief's State Timeline (Scheduled Program) intervals. This automatically calculated internal downtime and actual internal program could also be calculated at a fine grained level: Actual Beam Studies vs Downtime during Beam Studies, Actual Restore vs Downtime during Restore, Actual ACC vs Downtime during ACC.
We currently track machine downtime via DTM in intervals of time, but program time accounting is tracked in BTM as seconds in a given state per hour. This difference in time recording is problematic when correlating and generating reports. In addition, we allow BTM to provide competing downtime information vs DTM. The very small discrepancies on a given shift are tolerated on cross checks as a user friendly slop factor, but in aggregate across a large time period such as a run these tiny shift slops add up to noticeable differences. The goal of this issue (software change request) is to eliminate these discrepancies.
Tasks
Remove Internal Mode DOWN and Internal Mode FSD tracking from BOOM
BOOM is the meter/diagnostic tool that provides program information to BTM via EPICS CA. It currently tracks Internal Mode Downtime and FSDs, and subtracts this from program time. We need BOOM to instead simply report program time, unmodified by downtime and FSDs. DTM already tracks FSDs and Downtime more accurately in intervals.
Update BTM to monitor EPICS CA / BOOM for program changes and store the program in intervals (instead of seconds per hour)
This info will be exposed in BTM on a new first tab labeled
State Timeline. These intervals will need to be editable to allow mistakes to be corrected (just as DTM allows). The currently selected program should be displayed prominently, followed by a paginated history table of past planned program executions, in chronological order.Update BTM timesheets to calculate Internal Down based on Program from the state timeline combined with downtime from DTM This can be done by correlating intervals of program vs intervals of downtime. This allows us to remove some of the downtime related cross checks and ensures there are no longer discrepancies between BOOM measured downtime and DTM recorded downtime. This automatic correlation also alleviates the need to perform cumbersome manual program classification of downtimes in DTM, which would be especially clumsy when programs change during downtime.
Notes
A. Why do we need to differentiate between internal vs physics?
When we run reports later it is often important to subtract downtime from program time to determine amount of time machine was actually used as intended, on a machine mode basis. In particular, the Joule report metric research hours is actual time used during physics, which excludes internal mode. We could possibly change the way Physics Downtime is calculated going forward: correlate DTM vs BTM intervals to find downtimes that occurred during Physics mode. This would be more direct than figuring out Internal Down and subtracting it from Total Downtime.
B. How do we accommodate State Timeline modifications after a timesheet has already been started?
It will be helpful for Crew Chiefs to start a timesheet by first ensuring the program state timeline is correct. In most cases it will be a simple verification and no other action is needed. In the case when they forgot to click a button in BOOM or otherwise state is incorrect, they'll need to make corrections. Then, filling out their timesheet would proceed as normal. In the case of retroactive program changes after a timesheet has been submitted the measured values would change, but the reported values stay the same and would require manual intervention to fix. This is similar to the current situation.
C. How does the State Timeline compare to other schedules?
The NPES, PD Shift Plan, and Whiteboard are three schedules that inform the Crew Chief schedule, which up until now was found as the "Reported" row of seconds in a given state per hour inside a shift timesheet. The Crew Chief Schedule
State Timelineis an alternative (and more useful format) of this schedule. Persisting the Crew Chief schedule in intervals may allow us to simplify timesheets in the future by removing overlapping metrics. In particular, the "Accelerator Program" section of timesheets could possibly be removed (or at least automatically calculated). The Actual Operations Program intervals can be calculated by subtracting unscheduled downtime from the Crew Chief's State Timeline (Scheduled Program) intervals. This automatically calculated internal downtime and actual internal program could also be calculated at a fine grained level: Actual Beam Studies vs Downtime during Beam Studies, Actual Restore vs Downtime during Restore, Actual ACC vs Downtime during ACC.