[lorerod]

Description

closes #1943

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

[lorerod]

Hi @dianabarsan
I’ve been working on the CHT 4.x to 5.x Helm charts migration documentation and would love your help testing it on AWS.
I’m running into some AWS account permission issues, so while I sort that out, it would be amazing if you could be my QA for this ticket.
I’ve already successfully tested the migration locally on k3s-k3d, building 5.0.0 images from the master branch.

Proposed scenario:

1. Deploy an instance with the old Helm charts (4.x)
2. Upload data and two users (supervisor and chw so they can share data)
3. Log in as the offline users and create more data. Go offline.
4. Upgrade to 5.0.0 using this guide
5. Create more data offline. Go online.
6. Verify the upgrade and user access and data

Your feedback would be super helpful while I fix my permissions 🙏

[mrjones-plip]

Looking good! So great to see this guide being published to get closer to a 5.0 release.

I made some changes tone down our warnings!! and such - a breaking change I think speaks for itself.

I'll leave the technical testing to Diana - but am available to help if she's busy!

[lorerod]

Hi @mrjones, thank you for the feedback! We could definitely use a couple more pairs of eyes on this migration, as there are so many scenarios to cover.
If you have time and would like to, testing an upgrade from a 4.x multi-Couch instance to a 5.x multi-Couch instance on AWS would be amazing.

[mrjones-plip]

I finally got a chance to run through this! I may have done it wrong!

But, let's work through what I found.

[lorerod]

@mrjones-plip Thank you so much for the feedback! I’ve addressed all the points. Could you please try again when you have a chance? I’ll also test this thoroughly once I have the EKS permissions. You’re helping me keep this moving, thanks!

[mrjones-plip]

@lorerod - yo! This PR is coming together really nicely. It's getting easier and easier to follow with your improvements - thank you! I suspect we'll have at least a couple more rounds to finalize it.

I made some suggestions to tighten up the text and be more explicit.

Finally, it's still failing for me on the helm upgrade call (see below). I'm attaching my 2 values files with password replaced: yaml.for.lorena.zip

---

helm upgrade mrjones-dev ../build/helm \ 
  -f migration-5x-values.yaml \ 
  --namespace mrjones-dev     
Error: UPGRADE FAILED: cannot patch "couchdb-claim0" with kind PersistentVolumeClaim: PersistentVolumeClaim "couchdb-claim0" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
  core.PersistentVolumeClaimSpec{
        AccessModes:      {"ReadWriteOnce"},
        Selector:         nil,
        Resources:        {Requests: {s"storage": {i: {...}, s: "100Mi", Format: "BinarySI"}}},
-       VolumeName:       "pvc-881c6798-98a5-4164-9507-5694f9d989a7",
+       VolumeName:       "couchdb-pv-mrjones-dev",
-       StorageClassName: &"ebs-gp2",
+       StorageClassName: &"",
        VolumeMode:       &"Filesystem",
        DataSource:       nil,
        DataSourceRef:    nil,
  }

[mrjones-plip]

minor formatting I forgot to add on the first review

[lorerod]

@lorerod - yo! This PR is coming together really nicely. It's getting easier and easier to follow with your improvements - thank you! I suspect we'll have at least a couple more rounds to finalize it.

I made some suggestions to tighten up the text and be more explicit.

Finally, it's still failing for me on the helm upgrade call (see below). I'm attaching my 2 values files with password replaced: yaml.for.lorena.zip

Hi @mrjones-plip, I'm resuming this work. Sorry for the delay. I can't access the .zip file you are sharing here. Could you please share it again?

Quick update, @mrjones-plip: I've finally accessed the .zip. You don't need to share it again.

[mrjones-plip]

Thanks for the info no the zip - sorry if it gave you troubles! I'll keep an eye out here for any other questions or updates!

[lorerod]

@lorerod - yo! This PR is coming together really nicely. It's getting easier and easier to follow with your improvements - thank you! I suspect we'll have at least a couple more rounds to finalize it.

I made some suggestions to tighten up the text and be more explicit.

Finally, it's still failing for me on the helm upgrade call (see below). I'm attaching my 2 values files with password replaced: yaml.for.lorena.zip

helm upgrade mrjones-dev ../build/helm \ 
  -f migration-5x-values.yaml \ 
  --namespace mrjones-dev     
Error: UPGRADE FAILED: cannot patch "couchdb-claim0" with kind PersistentVolumeClaim: PersistentVolumeClaim "couchdb-claim0" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
  core.PersistentVolumeClaimSpec{
        AccessModes:      {"ReadWriteOnce"},
        Selector:         nil,
        Resources:        {Requests: {s"storage": {i: {...}, s: "100Mi", Format: "BinarySI"}}},
-       VolumeName:       "pvc-881c6798-98a5-4164-9507-5694f9d989a7",
+       VolumeName:       "couchdb-pv-mrjones-dev",
-       StorageClassName: &"ebs-gp2",
+       StorageClassName: &"",
        VolumeMode:       &"Filesystem",
        DataSource:       nil,
        DataSourceRef:    nil,
  }

Hi @mrjones, thanks for the great feedback! I have implemented all the suggestions.

I dug into the upgrade error, and it looks like two issues in migration-5x-values.yaml are causing it:

1. String vs boolean
   preExistingDataAvailable: "false" is a string, which is truthy in Helm, so 5.x charts treat it as enabled and try static provisioning, with leads to storage class conflict. To fix this please make it a boolean.

couchdb_data:
  preExistingDataAvailable: false

2. Clustering config mismatch: You have clusteredCouchEnabled: false (single-node) but noOfCouchDBNodes: 3 (multi-node). If single-node, please keep clusteredCouchEnabled: false and remove the entire clusteredCouch: section.

couchdb:
  clusteredCouchEnabled: false
# remove clusteredCouch: …

If multi-node, please set clusteredCouchEnabled: true and keep clusteredCouch: with your node count.

couchdb:
  clusteredCouchEnabled: true
clusteredCouch:
  noOfCouchDBNodes: 3

Also, please remove the USER-SUPPLIED VALUES: header on line 1.

I’ve updated the migration guide to use boolean false and clarified the single- vs multi-node settings.
I’ll also review the cht-core Helm charts for any remaining cases where booleans are quoted to prevent this kind of issue in future upgrades.

This is great feedback, thank you so much for testing and catching it!

[mrjones-plip]

Yay! thanks for doing another round of work on this.

I think this is maybe good to go? I updated my original values file to have true booleans (instead of truthy "false" values) and followed this PR and it worked! By the way, I ONLY changed the booleans and left everything else as is. Helm was ignoring all the multi-node stuffs before, so it makes sense it'd still ignore it.

helm upgrade mrjones-dev ../build/helm \ 
  -f migration-5x-values.yaml \
  --namespace mrjones-dev

Release "mrjones-dev" has been upgraded. Happy Helming!
NAME: mrjones-dev
LAST DEPLOYED: Tue Oct 21 15:26:31 2025
NAMESPACE: mrjones-dev
STATUS: deployed
REVISION: 2
TEST SUITE: None

However, the instance wasn't responding to HTTP requests. Checking kubectl -n mrjones-dev get pods I saw all ImagePullBackOff for the new services:

kubectl -n mrjones-dev get pods
NAME                                       READY   STATUS             RESTARTS   AGE                                                                                                                                                          
cht-api-6659b6f8d5-8cwlm                   1/1     Running            0          64m
cht-api-777c4887d-wkn4n                    0/1     ImagePullBackOff   0          90s
cht-couchdb-7cf9b88df7-tpwjs               0/1     ImagePullBackOff   0          90s
cht-couchdb-9f9bc9d47-sl7qm                1/1     Running            0          64m
cht-couchdb-nouveau-6db44d877b-s5j4j       0/1     ImagePullBackOff   0          87s
cht-haproxy-6f48f8c99b-q8srb               1/1     Running            0          64m
cht-haproxy-79f75b78bd-t5q7z               0/1     ImagePullBackOff   0          89s
cht-haproxy-healthcheck-5b5bf94dcf-q2q8b   0/1     ImagePullBackOff   0          89s
cht-haproxy-healthcheck-659585859b-r4wdc   1/1     Running            0          64m
cht-sentinel-656db545f5-2kcvx              0/1     ImagePullBackOff   0          88s
cht-sentinel-6cb97c9b98-rmv6n              1/1     Running            0          64m

Checking what might be the issue, of course 5.0.0 doesn't exist yet! Failed to pull image "public.ecr.aws/medic/cht-couchdb:5.0.0": rpc error: code = NotFound desc = failed to pull and unpack image "public.ecr.aws/medic/cht-couchdb:5.0.0": failed

I changed my chtversion and cht_image_tag to 5.0.0-test-replication-nouveau and re-ran the helm upgrade. Now it all looks good!

kubectl -n mrjones-dev get pods                           
NAME                                       READY   STATUS    RESTARTS   AGE
cht-api-645f48d486-dc6pt                   1/1     Running   0          15m
cht-couchdb-57cb9f5657-brpzw               1/1     Running   0          15m
cht-couchdb-nouveau-75989565c6-pp6pr       1/1     Running   0          15m
cht-haproxy-5499f7d8fd-tzwg8               1/1     Running   0          15m
cht-haproxy-healthcheck-7978fc74dc-9pznb   1/1     Running   0          15m
cht-sentinel-6b8b96c9f9-wchfc              1/1     Running   0          15m

However, the ALB isn't tying requests to deployment - only 404s for https://mrjones.dev.medicmobile.org. This is what I see when I delete a deployment vs when the ALB is sending traffic in but I get a an error 503.

I think it's worth debugging this deployment so we can be extra confident that it's nothing to do with these new helm charts. I suspect it's my bungling something. If you have admin perms, feel free to look at mrjones-dev namespace with mrjones-dev deployment.

Otherwise, maybe we can pair tomorrow (Wed) to debug?

[lorerod]

@mrjones-plip I’ve been reviewing the Helm charts, focusing on the Ingress and API configurations, and everything looks identical. Both versions route traffic to the same api service on port 5988, use the same service selector cht.service: api, and the pods also share the same label. The service type defaults to ClusterIP in both cases. I could be missing something, though. Maybe the load balancer needs a bit more time to catch up? Let’s take another look together tomorrow.

Thanks for all the feedback, it’s been super helpful!

[mrjones-plip]

sounds good! We have a call set up for tomorrow (Thur) - thanks for sending the invite.

i don't think load balancer needs to catch up - it's been 12+ hours and no change. We'll look into it tomorrow!

[mrjones-plip]

I decided to have another pass at this. On the hunch that this had to do with failed ingress, I spun up an alternate namespace (deleteme-mrjones-dev) that I kept on the old helm charts and then upgraded mine (mrjones-dev) to the new one.

By exporting the ingress to yaml and comparing the two name spaces I could quickly see that a bunch of annotations were missing, along with a finalizers and host values. with out these, there's no way that EKS would know to batch up the requests to the ALB and route traffic to the instance.

You can see the yaml files here:

broken ingress on `mrjones-dev`

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      meta.helm.sh/release-name: mrjones-dev
      meta.helm.sh/release-namespace: mrjones-dev
    creationTimestamp: "2025-10-22T21:36:42Z"
    generation: 2
    labels:
      app.kubernetes.io/managed-by: Helm
    name: api-ingress
    namespace: mrjones-dev
    resourceVersion: "510211979"
    uid: 588627f1-9ca0-4fe7-86bb-ebdeecd6f891
  spec:
    rules:
    - http:
        paths:
        - backend:
            service:
              name: api
              port:
                number: 5988
          path: /
          pathType: Prefix
  status:
    loadBalancer:
      ingress:
      - hostname: k8s-devchtalb-3eb0781cbb-694321496.eu-west-2.elb.amazonaws.com
kind: List
metadata:
  resourceVersion: ""

working ingress on `deleteme-mrjones-dev`

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:iam::720541322708:server-certificate/2025-q4-wildcard-dev-medicmobile-org-letsencrypt
      alb.ingress.kubernetes.io/group.name: dev-cht-alb
      alb.ingress.kubernetes.io/healthcheck-port: traffic-port
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/tags: Environment=dev,Team=QA
      alb.ingress.kubernetes.io/target-type: ip
      meta.helm.sh/release-name: deleteme-mrjones-dev
      meta.helm.sh/release-namespace: deleteme-mrjones-dev
    creationTimestamp: "2025-10-22T21:48:37Z"
    finalizers:
    - group.ingress.k8s.aws/dev-cht-alb
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    name: api-ingress
    namespace: deleteme-mrjones-dev
    resourceVersion: "510210384"
    uid: 17630888-630c-4a5a-b06f-d13b82c38612
  spec:
    ingressClassName: alb
    rules:
    - host: deleteme-mrjones.dev.medicmobile.org
      http:
        paths:
        - backend:
            service:
              name: api
              port:
                number: 5988
          path: /
          pathType: Prefix
  status:
    loadBalancer:
      ingress:
      - hostname: k8s-devchtalb-3eb0781cbb-694321496.eu-west-2.elb.amazonaws.com
kind: List
metadata:
  resourceVersion: ""

By calling kubectl edit ingress -n mrjones-dev -o yaml and then manually adding the missing values, the instance came up. w00t w00t! Hat tip to @Hareet for skilling me up on debugging k8s ingress trickery!

Let's keep our meeting tomorrow, if only to review how this failed and if we need to improve the docs in this PR. If you're able to fix it before then with these notes, we can always cancel!

[lorerod]

Great debugging! The broken ingress looks very similar to the k3s-k3d template. So I may have a problem with cluster_type == "eks".
I will check on this tomorrow. Thanks!

[lorerod]

@mrjones-plip I created this branch, adding a missing condition to the k3s-k3d ingress template so it doesn't overwrite the EKS/GCP ingress. It would be amazing if you can try this. Thanks.

[mrjones-plip]

Let's do this!!

With the changes that you'll submit in a separate PR from your k3s branch, this PR is good to go I think. yay!!!

Added a minor sed command which should address some of the earlier errors I was getting. Other changes are to line up the numbering.