Fix broken datasource causing panels not to load

[Sleuth56]

Broken panels show the following error.
Datasource ${DS_PROMETHEUS} was not found

This is caused from the datasource variable being renamed but not being properly set in the panels or some variables.

[cubic-dev-ai]

1 issue found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="dashboards/vlagent.json">

<violation number="1" location="dashboards/vlagent.json:203">
P1: Datasource migration is incomplete: added references still use `${DS_PROMETHEUS}` even though the variable is now `ds`, so panels/queries can still fail with datasource-not-found errors.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}

[cubic-dev-ai]

P1: Datasource migration is incomplete: added references still use ${DS_PROMETHEUS} even though the variable is now ds, so panels/queries can still fail with datasource-not-found errors.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At dashboards/vlagent.json, line 203:

<comment>Datasource migration is incomplete: added references still use `${DS_PROMETHEUS}` even though the variable is now `ds`, so panels/queries can still fail with datasource-not-found errors.</comment>

<file context>
@@ -244,12 +195,12 @@
           "datasource": {
             "type": "prometheus",
-            "uid": "$DS_PROMETHEUS"
+            "uid": "${DS_PROMETHEUS}"
           },
           "expr": "sum(increase(vm_log_messages_total{job=~\"$job\", instance=~\"$instance\", level!=\"info\"}[30m]))",
</file context>

[Sleuth56]

I think it got confused. The variable name didn't change.

[cubic-dev-ai]

Thanks for the clarification—looks like I misread the rename. I’ll revisit the comment accordingly.

[Sleuth56]

A variable can have both a name and a uid. The UID didn't change.

[cubic-dev-ai]

Thanks for the feedback! I've saved this as a new learning to improve future reviews.

[Sleuth56]

Some additional context that I realized I left out. When added through the Grafana UI the dashboard works as expected but when provisioned it does not. Below is the helm values I used to test this.

dashboards:
  default:
    vl-agent:
      url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaLogs/refs/heads/dashboards/vlagent-datasource-broken/dashboards/vlagent.json
      # gnetId: 24513
      # revision: 2
      # datasource: prometheus

[vadimalekseev]

A long time ago, we used the $ds variable, but we migrated to DS_PROMETHEUS for the following reasons:

1. When exporting a dashboard for "external use" Grafana throw an error for some reason. However, using a variable named DS_PROMETHEUS does not trigger this error for reasons I couldn't determine. I was able to reproduce this error using the dashboard from this PR:

2. We use the "external use" approach because it is impossible to create a new dashboard on the Grafana website due to a validation error (see PR below). However, once a dashboard is already created, uploading a new version works without issues. See this pull request, which exports dashboards for "external use" for consistency: Resave all of VictoriaLogs dashboards due to upload errors to grafana.com #861

Taking into account that Grafana allows uploading new versions of dashboards without validation errors (unlike creating new ones), I am generally not against returning to the $ds approach. However, if we do so, it should be applied to all dashboards for consistency. Also, could you please describe in a comment on this PR the correct way to export the dashboard so that other contributors know how to reproduce the result?

[Sleuth56]

So I edited that one by hand but I also tested doing a export of dashboard using Grafana's export feature and that method also works. I can use that in the commit if you would rather.

The steps for the working export: Grafana v13

1. Provisioned import using the json in this repo.
2. Export option in Grafana
   a. Check the box Share dashboard with another instance
3. Export JSON
   (I hosted the JSON from the export in my git repo than used helm to import it as a provisioned dashboard)
   That is my exported dashboard. https://gitlab.com/homelab2.0/k8s/-/raw/main/pr-talos/grafana/test-dashboard.yml?ref_type=heads

The reasoning for manually editing was to keep the commit changes to a minimum.