staging not working, upgrades not possible #10404

Activity

1. mrjones-plip

   added

   Type: Technical issueImprove something that users won't noticeImprove something that users won't notice

   on Oct 23, 2025
2. mrjones-plip

   marked Admin upgrade page: "Error fetching available versions" on EKS hosted CHT 5 #10400 as a duplicate of this issue on Oct 23, 2025
3. andrablaj

   added this to CHT Stewardship Activities Q4 2025on Oct 23, 2025
4. andrablaj

   moved this to 💻 In Progress in CHT Stewardship Activities Q4 2025on Oct 23, 2025
5. andrablaj

   assigned

   Hareet

   on Oct 23, 2025
6. mrjones-plip commented on Oct 23, 2025

   

   mrjones-plip

   on Oct 23, 2025

   Last edited by mrjones-plip

   ContributorAuthor

   More actions

   The timeline was something like this (all times Pacific)

   1. Oct 22 ~2pm: test push of mrjones.dev.medicmobile.org to dev EKS cluster test helm chart upgrades
   2. Oct 22 ~2pm: Ingress for mrjones.dev.medicmobile.org doesn't - it's missing a bunch annotations, finalizers and host values
   3. Oct 22 ~3pm: kubectl edit ingress -n mrjones-dev -o yaml is done to manually fix annotations, finalizers and host on ingress. this works, but due to a miss-configurations, causes staging.dev.medicmobile.org to be routed to mrjones.dev.medicmobile.org deployment. At this point all upgrades are impossible because staging is effectively offline
   4. Oct 22 11.24pm: at least one an end user notices an issue with their *.dev.medicmobile.org on dev EKS cluster also being miss-routed
   5. Oct 23 ~5.50am: @dianabarsan notices that staging server has been replaced with a CHT instance
   6. Oct 23 6.52am: ingress is fixed, upgrades now work

   The crux of the issue seems to have been on stop 3 above:

   spec:
     ingressClassName: alb
     rules:
     - host: mrjones.dev.medicmobile.org
     - http:
         paths:
   

   This passed 2 rules to the ingress, one for host and one separate one for http (which went and created the empty rule with no host). http shouldn't have the "dash" (-), which then would nest those parameters under the host rule.
7. mrjones-plip

   closed this as completedon Oct 23, 2025
8. github-project-automation

   moved this from 💻 In Progress to 🚀 Done in CHT Stewardship Activities Q4 2025on Oct 23, 2025
9. mrjones-plip commented on Oct 23, 2025

   

   mrjones-plip

   on Oct 23, 2025

   ContributorAuthor

   More actions

   See private slack thread as well