Add RH Cost Management OnPrem to nerc-ocp-test cluster

[computate]

Motivation

• Open Sovereign AI Cloud and Data Center Management goals include Cost Management On Prem features as sovereign cost data.

Completion Criteria

• Ensure Cost Management On Prem Operator is working.
• Deploy Cost Management On Prem to nerc-ocp-test cluster and configure nerc-rates settings for AI workloads.
• Test MIG GPU configurations in Cost Management OnPrem.

Description

• Cost Management Koku operator is not available in OpenShift 4.21 yet. The nerc-ocp-test cluster is on 4.21.5. Hopefully the week of Apr 13 we can try again.

[msdisme]

@hakasapl @computate any updates on this? Still in progress?

[computate]

@msdisme Cost Management Operator is now up and running on nerc-ocp-test:
https://console-openshift-console.apps.ocp-test.nerc.mghpcc.org/k8s/ns/costmanagement-metrics-operator/core~v1~Pod?page=1&perPage=50
I will continue to work with the Cost Managment team to understand what is next to observe the cost management metrics being collected.

[joachimweyl]

nerc-ocp-test is going away in a month or so. Do we plan to have this work done by then, or should we wait for 2.0?

[computate]

The MOC 1.0 testing of RH Cost Management On Prem operator has been completed. The UI for cost management is not ready, so that testing could happen in the future in MOC 2.0.

[hpdempsey]

Where are the results? Can it be deployed without the UI to simply prepare the cost data?

[computate]

@hpdempsey the UI for cost management on prem is not available. They told me:
The observability UI work (COST-7470) is still underway, so the downtime while MOC 2.0 is being built could give us the runway to get wrap things up. On our end, as long as we can install the operators on the new platform once it's up, we should be good.

[computate]

@hpdempsey however, there are CSV reports that are stored on the costmanagement pod on the test cluster:

oc -n costmanagement-metrics-operator rsh $(oc -n costmanagement-metrics-operator get pod -l app=costmanagement-metrics-operator -o name)

$ ls /tmp/costmanagement-metrics-operator-reports/data/
cm-openshift-namespace-usage-202607.csv   cm-openshift-pod-usage-202607.csv
cm-openshift-node-usage-202607.csv	  cm-openshift-storage-usage-202607.csv
cm-openshift-nvidia-gpu-usage-202607.csv  cm-openshift-vm-usage-202607.csv

[computate]

@hpdempsey here are the column headers of each of those files:

$ head -n 1 *
==> cm-openshift-namespace-usage-202607.csv <==
report_period_start,report_period_end,interval_start,interval_end,namespace,namespace_labels

==> cm-openshift-node-usage-202607.csv <==
report_period_start,report_period_end,interval_start,interval_end,node,node_labels

==> cm-openshift-nvidia-gpu-usage-202607.csv <==
report_period_start,report_period_end,interval_start,interval_end,node,namespace,pod,gpu_uuid,gpu_model_name,gpu_vendor_name,gpu_memory_capacity_mib,gpu_pod_uptime,gpu_pod_utilization,gpu_max_slices,mig_instance_id,mig_profile,mig_strategy

==> cm-openshift-pod-usage-202607.csv <==
report_period_start,report_period_end,interval_start,interval_end,node,namespace,pod,pod_usage_cpu_core_seconds,pod_request_cpu_core_seconds,pod_limit_cpu_core_seconds,pod_usage_memory_byte_seconds,pod_request_memory_byte_seconds,pod_limit_memory_byte_seconds,node_capacity_cpu_cores,node_capacity_cpu_core_seconds,node_capacity_memory_bytes,node_capacity_memory_byte_seconds,node_role,resource_id,pod_labels

==> cm-openshift-storage-usage-202607.csv <==
report_period_start,report_period_end,interval_start,interval_end,namespace,pod,node,persistentvolumeclaim,persistentvolume,storageclass,csi_driver,csi_volume_handle,persistentvolumeclaim_capacity_bytes,persistentvolumeclaim_capacity_byte_seconds,volume_request_storage_byte_seconds,persistentvolumeclaim_usage_byte_seconds,persistentvolume_labels,persistentvolumeclaim_labels

==> cm-openshift-vm-usage-202607.csv <==
report_period_start,report_period_end,interval_start,interval_end,node,resource_id,namespace,vm_name,vm_instance_type,vm_os,vm_guest_os_arch,vm_guest_os_name,vm_guest_os_version,vm_uptime_total_seconds,vm_cpu_limit_cores,vm_cpu_limit_core_seconds,vm_cpu_request_cores,vm_cpu_request_core_seconds,vm_cpu_request_sockets,vm_cpu_request_socket_seconds,vm_cpu_request_threads,vm_cpu_request_thread_seconds,vm_cpu_usage_total_seconds,vm_memory_limit_bytes,vm_memory_limit_byte_seconds,vm_memory_request_bytes,vm_memory_request_byte_seconds,vm_memory_usage_byte_seconds,vm_device,vm_volume_mode,vm_persistentvolumeclaim_name,vm_disk_allocated_size_byte_seconds,vm_labels