From 84f8d095a20a44d8900809852ad53feea84df6e5 Mon Sep 17 00:00:00 2001 From: moizbigdata Date: Wed, 3 Sep 2025 09:06:01 +0530 Subject: [PATCH 1/3] added readme file by moiz --- ideas/moiz.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 ideas/moiz.md diff --git a/ideas/moiz.md b/ideas/moiz.md new file mode 100644 index 0000000..db5db98 --- /dev/null +++ b/ideas/moiz.md @@ -0,0 +1,26 @@ +# Project Title + +A brief summary of what this project is about. + +## Table of Contents + +- [Introduction](#introduction) +- [Features](#features) +- [Installation](#installation) +- [Usage](#usage) +- [Contributing](#contributing) +- [License](#license) +- [Contact](#contact) + +## Introduction + +Describe the purpose and background of the project. + +## Features + +- List key features here. + +## Installation + +Include step-by-step instructions. + From e9694c416d0e4be582a1359d434e4453683639bd Mon Sep 17 00:00:00 2001 From: moizbigdata Date: Wed, 17 Jun 2026 14:51:34 +0530 Subject: [PATCH 2/3] Add files via upload --- azure-resource-tags-ingestion.md | 177 +++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 azure-resource-tags-ingestion.md diff --git a/azure-resource-tags-ingestion.md b/azure-resource-tags-ingestion.md new file mode 100644 index 0000000..36a7302 --- /dev/null +++ b/azure-resource-tags-ingestion.md @@ -0,0 +1,177 @@ +# Resource tags in Log Analytics (Grafana variables) + +Tags on Arc machines (`Company`, `Site`, `ClusterId`, etc.) are **Azure Resource Manager metadata**. They are **not** collected by the Azure Monitor Agent or your Default/Custom DCR. + +| Data | Source | Collected by DCR? | +|------|--------|-------------------| +| Perf, Event, Heartbeat | AMA on each node | Yes | +| Resource tags | Azure Resource Graph (ARM) | **No** | + +There is **no built-in `AzureResource` table** in Log Analytics. Use one of the options below. + +--- + +## Option A — Recommended: `arg("").Resources` (no DCR) + +Query Azure Resource Graph **directly from Log Analytics** using cross-service KQL. No extra ingestion, no custom table. + +**Prerequisites** + +- Grafana SP: **Log Analytics Reader** on central LAW +- Grafana SP: **Reader** on all site subscriptions (ARG read access) +- Run queries from the central LAW in the Azure portal first to validate + +**Example — list companies** + +```kusto +arg("").Resources +| where type =~ "microsoft.hybridcompute/machines" +| extend Company = tostring(tags.Company) +| where isnotempty(Company) +| distinct Company +| order by Company asc +``` + +**Example — cluster RG from tags** + +```kusto +arg("").Resources +| where type =~ "microsoft.hybridcompute/machines" +| extend Company = tostring(tags.Company) +| extend Site = tostring(tags.Site) +| extend ClusterId = tostring(tags.ClusterId) +| extend Env = tostring(tags.Env) +| extend ClusterRg = tostring(tags.Cluster) +| where Company == "CompanyA" and Site == "SiteA" and ClusterId == "C1" and Env == "Prod" +| distinct ClusterRg +``` + +**Grafana variable queries** in [`grafana/queries/`](../grafana/queries/) use this pattern. + +**Notes** + +- Preview feature; editor may show false syntax errors — run the query anyway +- `arg()` returns max **1,000 rows** per query (enough for variable dropdowns) +- Tag property names are case-sensitive (`tags.Company`, not `tags.company`) + +Reference: [Correlate ARG with Log Analytics](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/azure-monitor-data-explorer-proxy) + +--- + +## Option B — Custom table + scheduled sync (optional DCR) + +Use this only if you need a **physical table** in the LAW (offline queries, no ARG dependency in Grafana, or arg() unavailable). + +### Architecture + +``` +Azure Resource Graph + │ + ▼ +Azure Automation runbook (daily/hourly) + │ + ▼ +Logs Ingestion API ──► DCR (Direct) ──► HybridResourceTags_CL +``` + +### Step 1 — Create custom table + +```bash +az monitor log-analytics workspace table create \ + --resource-group azr-mon-rg-monitoring \ + --workspace-name azr-mon-law-central \ + --name HybridResourceTags_CL \ + --columns '[{"name":"TimeGenerated","type":"datetime"},{"name":"ResourceId","type":"string"},{"name":"Company","type":"string"},{"name":"Site","type":"string"},{"name":"ClusterId","type":"string"},{"name":"Env","type":"string"},{"name":"ClusterRg","type":"string"},{"name":"SubscriptionId","type":"string"},{"name":"Name","type":"string"}]' +``` + +### Step 2 — Create ingestion DCR (Direct kind) + +Deploy [`infra/modules/dcr-resource-tags.bicep`](../infra/modules/dcr-resource-tags.bicep) or create in Portal: + +- **Stream**: `Custom-HybridResourceTags` +- **Destination**: central LAW +- **Transform**: `source` +- **Output**: `HybridResourceTags_CL` + +This DCR does **not** attach to Arc machines — it receives data from the **Logs Ingestion API** only. + +### Step 3 — Grant Automation managed identity + +- Role: **Monitoring Metrics Publisher** on the DCR +- Role: **Reader** on subscriptions (to query ARG) + +### Step 4 — Runbook posts tag inventory + +Runbook queries ARG and POSTs JSON to the DCR immutable ID endpoint (see Microsoft sample for Logs Ingestion API). + +### Step 5 — Grafana variables query custom table + +```kusto +HybridResourceTags_CL +| where TimeGenerated > ago(1d) +| summarize arg_max(TimeGenerated, *) by ResourceId +| extend Company = Company +| distinct Company +``` + +--- + +## Option C — Legacy: HTTP Data Collector (no DCR) + +Older pattern: Automation → HTTP Data Collector API → `VMResourceTags_CL`. Microsoft recommends **Logs Ingestion API + DCR** (Option B) for new deployments. + +--- + +## What your Default/Custom DCR already collect + +Your AMA DCRs ([`dcr-default.bicep`](../infra/modules/dcr-default.bicep)) collect telemetry only: + +- `Microsoft-Perf` → `Perf` +- `Microsoft-Event` → `Event` (includes `_ResourceId` on each row) +- `Microsoft-InsightsMetrics` → `InsightsMetrics` +- Heartbeat (automatic with LAW association) + +Each row includes `_ResourceId`, which you can **join** to tag data: + +```kusto +Heartbeat +| where TimeGenerated > ago(1h) +| lookup ( + arg("").Resources + | where type =~ "microsoft.hybridcompute/machines" + | project _ResourceId=tolower(id), tags +) on _ResourceId +| where tostring(tags.Company) == "CompanyA" +``` + +--- + +## Verify tag queries work + +Run in central LAW: + +```kusto +// Option A — ARG cross-query +arg("").Resources +| where type =~ "microsoft.hybridcompute/machines" +| where isnotempty(tags.Company) +| project name, resourceGroup, subscriptionId, tags +| take 10 + +// Option B — custom table (if deployed) +HybridResourceTags_CL +| summarize arg_max(TimeGenerated, *) by ResourceId +| take 10 +``` + +--- + +## Summary + +| Goal | Approach | +|------|----------| +| Grafana tag variables (`company`, `site`, …) | **Option A**: `arg("").Resources` — update queries in `grafana/queries/` | +| Physical tag table in LAW | **Option B**: Custom table + ingestion DCR + Automation | +| Agent metrics/logs | Existing **Default/Custom DCR** (unchanged) | + +**Do not add tag collection to the Default DCR** — it will not work; AMA cannot read ARM tags. From 04ae0b83552dfc11b16e87258c8c09b310564584 Mon Sep 17 00:00:00 2001 From: moizbigdata Date: Wed, 17 Jun 2026 14:53:09 +0530 Subject: [PATCH 3/3] Add files via upload --- grafana-rbac.md | 97 ++++++++++++++++ grafana-variables.md | 216 +++++++++++++++++++++++++++++++++++ onboarding-runbook.md | 191 +++++++++++++++++++++++++++++++ tagging-standard.md | 260 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 764 insertions(+) create mode 100644 grafana-rbac.md create mode 100644 grafana-variables.md create mode 100644 onboarding-runbook.md create mode 100644 tagging-standard.md diff --git a/grafana-rbac.md b/grafana-rbac.md new file mode 100644 index 0000000..0814969 --- /dev/null +++ b/grafana-rbac.md @@ -0,0 +1,97 @@ +# Grafana RBAC for Hybrid Monitoring + +Self-hosted Grafana uses the Azure Monitor data source plugin to query the central Log Analytics workspace and list ARM resources for dashboard variables. + +## Service principal requirements + +Grant the Grafana service principal these roles: + +| Role | Scope | Purpose | +|------|-------|---------| +| **Log Analytics Reader** | Central LAW | Run KQL queries (Perf, Event, Heartbeat, InsightsMetrics) | +| **Reader** | Site subscriptions and/or cluster RGs | ARM-based variables (`microsoft.hybridcompute/machines`) | +| **Monitoring Reader** | Optional — cluster RGs | Azure Stack HCI platform metrics (`microsoft.azurestackhci/clusters`) | + +### POC service principal + +Client ID (from existing POC): `1dbeedd6-ac59-4d5c-ae41-4850755c7670` + +## Assign roles (Cloud Ops) + +```bash +GRAFANA_SP_OBJECT_ID="" +LAW_ID="/subscriptions//resourceGroups/azr-mon-rg-monitoring/providers/Microsoft.OperationalInsights/workspaces/azr-mon-law-central" +SITE_SUB_ID="" +CLUSTER_RG="azr-131-itsusra1-delldev01" + +# Log Analytics Reader on central workspace +az role assignment create \ + --assignee-object-id "${GRAFANA_SP_OBJECT_ID}" \ + --assignee-principal-type ServicePrincipal \ + --role "Log Analytics Reader" \ + --scope "${LAW_ID}" + +# Reader on site subscription (ARM variables) +az role assignment create \ + --assignee-object-id "${GRAFANA_SP_OBJECT_ID}" \ + --assignee-principal-type ServicePrincipal \ + --role "Reader" \ + --scope "/subscriptions/${SITE_SUB_ID}" + +# Optional: Reader scoped to cluster RG only (least privilege) +az role assignment create \ + --assignee-object-id "${GRAFANA_SP_OBJECT_ID}" \ + --assignee-principal-type ServicePrincipal \ + --role "Reader" \ + --scope "/subscriptions/${SITE_SUB_ID}/resourceGroups/${CLUSTER_RG}" +``` + +## Grafana data source configuration + +| Setting | Value | +|---------|-------| +| Authentication | App Registration / Service Principal | +| Tenant ID | Your Azure AD tenant | +| Client ID | Grafana SP client ID | +| Default subscription | Site subscription for multi-cluster dashboards | +| Log Analytics default workspace | Central LAW (`azr-mon-law-central`) | + +## Dashboard variables (AZR-208) + +| Variable | Source | RBAC needed | +|----------|--------|-------------| +| `az_monitor` | Azure Monitor datasource | LAW access | +| `subscription` | Azure Subscriptions API | Reader on subscription | +| `cluster` | Resource Groups | Reader on subscription | +| `computer` | **KQL** ([`list_computer_query`](../list_computer_query)) | Log Analytics Reader only | +| `disk` / `pool` | SDDC EventID 3002 KQL | Log Analytics Reader only | + +### Recommended: SDDC KQL for `computer` variable + +For Azure Local HCI, prefer the SDDC EventID 3000 + Heartbeat KQL variable over ARM `microsoft.hybridcompute/machines`: + +- No extra ARM Reader permission required +- Lists only HCI cluster nodes (not unrelated Arc machines) +- Matches panel query hostname logic (`HostKey` join) + +See [`list_computer_variable_arm.md`](../list_computer_variable_arm.md) for ARM troubleshooting if HybridCompute namespace is missing from the dropdown. + +## Multi-site dashboard patterns + +**Per-cluster dashboard (current AZR-208):** + +- Resource scope: `/subscriptions/${subscription}/resourcegroups/${cluster}` +- KQL filter: `| where _ResourceId has tolower("/resourcegroups/${cluster}/")` + +**Cross-site overview (future):** + +- Query central LAW without cluster filter +- Add `Site` tag column if propagated to custom logs +- Or use subscription variable to switch sites + +## Verification + +1. Open Grafana → Explore → Azure Monitor → Logs +2. Run: `Heartbeat | where TimeGenerated > ago(1h) | take 10` +3. Open AZR-208 dashboard; confirm `computer` variable populates +4. If ARM variable needed: confirm `microsoft.hybridcompute` appears after Reader is granted diff --git a/grafana-variables.md b/grafana-variables.md new file mode 100644 index 0000000..7ad3327 --- /dev/null +++ b/grafana-variables.md @@ -0,0 +1,216 @@ +# Grafana Dashboard Variables (Tag-Based) + +Dashboard variables follow the tagging hierarchy defined in [tagging-standard.md](tagging-standard.md): + +``` +Company → Site → ClusterId → Env → Cluster (RG) → Subscription → Physical Node (computer) → Disk / Pool +``` + +All tag-driven variables query the **central Log Analytics workspace** via the Azure Monitor data source. Node and storage variables scope to the selected cluster resource group. + +## Variable summary + +| Order | Variable | Label | Type | Depends on | Tag / source | +|-------|----------|-------|------|------------|--------------| +| 1 | `az_monitor` | Azure Monitor | Data source | — | Grafana Azure Monitor plugin | +| 2 | `law_subscription` | LAW Subscription | Azure Subscriptions (hidden) | — | Monitoring sub hosting central LAW | +| 3 | `company` | Company | Log Analytics KQL | — | `tags.Company` | +| 4 | `site` | Site | Log Analytics KQL | `company` | `tags.Site` | +| 5 | `cluster_id` | Cluster ID | Log Analytics KQL | `company`, `site` | `tags.ClusterId` | +| 6 | `env` | Environment | Log Analytics KQL | `company`, `site`, `cluster_id` | `tags.Env` | +| 7 | `cluster` | Cluster | Log Analytics KQL | above | `tags.Cluster` (RG name) | +| 8 | `subscription` | Subscription | Log Analytics KQL | `cluster` | Site `subscriptionId` from Arc machine | +| 9 | `computer` | Physical Node | Log Analytics KQL | `subscription`, `cluster` | SDDC 3000 + Heartbeat | +| 10 | `disk` | Disk | Log Analytics KQL | `subscription`, `cluster` | SDDC 3002 volumes | +| 11 | `pool` | Storage Pool | Log Analytics KQL (hidden) | `subscription`, `cluster`, `computer` | SDDC 3002 pools | + +## Architecture flow + +```mermaid +flowchart LR + law_sub[law_subscription] --> company[company] + company --> site[site] + site --> cluster_id[cluster_id] + cluster_id --> env[env] + env --> cluster[cluster] + cluster --> subscription[subscription] + cluster --> computer[computer] + cluster --> disk[disk] + computer --> pool[pool] +``` + +`law_subscription` is hidden — set to the monitoring subscription that hosts the central LAW. Tag queries (`company`–`cluster`) run against that workspace context. `subscription` is the **site** subscription resolved from the selected cluster. + +## Query files + +| Variable | KQL file | +|----------|----------| +| `company` | [`grafana/queries/list_company_query.kql`](../grafana/queries/list_company_query.kql) | +| `site` | [`grafana/queries/list_site_query.kql`](../grafana/queries/list_site_query.kql) | +| `cluster_id` | [`grafana/queries/list_cluster_id_query.kql`](../grafana/queries/list_cluster_id_query.kql) | +| `env` | [`grafana/queries/list_env_query.kql`](../grafana/queries/list_env_query.kql) | +| `cluster` | [`grafana/queries/list_cluster_query.kql`](../grafana/queries/list_cluster_query.kql) | +| `subscription` | [`grafana/queries/list_subscription_query.kql`](../grafana/queries/list_subscription_query.kql) | +| `computer` | [`grafana/queries/list_computer_query.kql`](../grafana/queries/list_computer_query.kql) | +| `disk` | [`grafana/queries/list_disk_query.kql`](../grafana/queries/list_disk_query.kql) | +| `pool` | [`grafana/queries/list_pool_query.kql`](../grafana/queries/list_pool_query.kql) | + +Import-ready JSON snippet: [`grafana/dashboard-variables.json`](../grafana/dashboard-variables.json) + +## Grafana setup (per variable) + +### 1. `az_monitor` — Data source + +| Setting | Value | +|---------|-------| +| Type | Data source | +| Name | `az_monitor` | +| Query | `grafana-azure-monitor-datasource` | + +Point at the central LAW in the monitoring subscription. + +### 2. `law_subscription` — LAW hosting subscription (hidden) + +| Setting | Value | +|---------|-------| +| Type | Azure Subscriptions | +| Hide | Variable | +| Default | Monitoring subscription ID (e.g. POC: `5dcf3298-c3a2-4aa6-90d7-8cdd9f6f3563`) | + +Used as API context for all `AzureResource` tag queries. Do not confuse with `subscription` (site sub). + +### 3–8. Tag hierarchy variables — Azure Log Analytics + +Common settings for `company`, `site`, `cluster_id`, `env`, `cluster`, `subscription`: + +| Setting | Value | +|---------|-------| +| Data source | `${az_monitor}` | +| Query type | Azure Log Analytics | +| Mode | Raw KQL | +| Subscription | `${law_subscription}` | +| Refresh | On dashboard load | +| Resource scope | `/subscriptions/${law_subscription}` | + +**Include All** — enable for `company`, `site`, `cluster_id`, `env` with custom all value `All`. + +**Include All** — disable for `cluster` and `subscription` (a cluster must be selected for panels). + +Copy KQL from the query files into each variable's query editor. + +### 8. `computer` — Physical node + +| Setting | Value | +|---------|-------| +| Multi-value | ON | +| Include All | ON | +| Custom all value | `All` | +| Resource scope | `/subscriptions/${subscription}/resourcegroups/${cluster}` | + +Uses SDDC EventID 3000 node inventory joined to Heartbeat — no ARM Reader required. + +### 9. `disk` — Volume + +| Setting | Value | +|---------|-------| +| Multi-value | ON | +| Include All | ON | +| Custom all value | `all` | +| Resource scope | `/subscriptions/${subscription}/resourcegroups/${cluster}` | + +### 10. `pool` — Storage pool (hidden) + +| Setting | Value | +|---------|-------| +| Hide | Variable | +| Resource scope | `/subscriptions/${subscription}/resourcegroups/${cluster}` | + +## Panel resource scope + +After selecting the tag hierarchy, panels use: + +``` +/subscriptions/${subscription}/resourcegroups/${cluster} +``` + +KQL filter (used in all AZR-208 panels): + +```kusto +| where _ResourceId has tolower("/resourcegroups/${cluster}/") +``` + +Computer filter: + +```kusto +| where "${computer}" == "All" or Computer in (${computer:doublequote}) +``` + +## Prerequisites + +### Resource tags (Grafana hierarchy variables) + +Tag variables use **`arg("").Resources`** (Azure Resource Graph cross-query), **not** a DCR table. Tags are ARM metadata — AMA/DCR cannot collect them. + +See [azure-resource-tags-ingestion.md](azure-resource-tags-ingestion.md) for: +- Option A: `arg("").Resources` (recommended, no DCR) +- Option B: Custom `HybridResourceTags_CL` table + ingestion DCR + Automation + +**Prerequisites for tag variables** + +| Role | Scope | +|------|-------| +| Log Analytics Reader | Central LAW | +| Reader | All site subscriptions (ARG read for `arg()`) | + +Verify in LAW: + +```kusto +arg("").Resources +| where type =~ "microsoft.hybridcompute/machines" +| where isnotempty(tags.Company) +| take 5 +``` + +### RBAC + +| Role | Scope | Variables | +|------|-------|-----------| +| Log Analytics Reader | Central LAW | All KQL variables | +| Reader | Site subscriptions | Optional — ARM fallback only | + +See [grafana-rbac.md](grafana-rbac.md). + +## POC fallback (no AzureResource) + +If `AzureResource` is not yet populated, use static custom variables from [`grafana/poc-tag-mapping.json`](../grafana/poc-tag-mapping.json): + +1. Create custom variable `cluster_mapping` (constant JSON or query) +2. Or keep legacy flow: `subscription` (ARM) → `cluster` (ARM Resource Groups) → `computer` (SDDC KQL) + +Legacy AZR-208 order (pre-tags): + +``` +az_monitor → subscription → cluster → computer → disk → pool +``` + +Migrate to tag hierarchy once Arc machines are tagged and `AzureResource` is available. + +## Apply to AZR-208 dashboard + +1. Open **AZR-208-Metrics and Logs** → **Settings** → **Variables** +2. Add `company`, `site`, `cluster_id`, `env` above existing `subscription` +3. Replace `cluster` query with [`list_cluster_query.kql`](../grafana/queries/list_cluster_query.kql) +4. Replace `subscription` query with [`list_subscription_query.kql`](../grafana/queries/list_subscription_query.kql) (depends on `cluster`) +5. Update `computer` / `disk` / `pool` from [`grafana/queries/`](../grafana/queries/) +6. Re-order variables per table above +7. Save and test: **CompanyA → SiteA → C1 → Prod → azr-131-itsusra1-delldev01** + +Or import [`grafana/dashboard-variables.json`](../grafana/dashboard-variables.json) into a new dashboard and copy the templating block. + +## Example selection (architecture diagram) + +| Diagram group | Variable values | +|---------------|-----------------| +| Site/Company A C1 | `company=CompanyA`, `site=SiteA`, `cluster_id=C1`, `cluster=azr-131-itsusra1-delldev01` | +| Site/Company A C2 | `company=CompanyA`, `site=SiteA`, `cluster_id=C2`, `cluster=azr-100-companya-c2-rg` | +| Site/Company B C1 | `company=CompanyB`, `site=SiteB`, `cluster_id=C1`, `cluster=azr-200-companyb-c1-rg` | diff --git a/onboarding-runbook.md b/onboarding-runbook.md new file mode 100644 index 0000000..56365a7 --- /dev/null +++ b/onboarding-runbook.md @@ -0,0 +1,191 @@ +# Hybrid Monitoring Onboarding Runbook + +Step-by-step guide to onboard Azure Arc servers, Azure VMs, and Azure Local HCI clusters from multiple sites/subscriptions into the central Log Analytics workspace. + +## Prerequisites + +- Monitoring subscription with Contributor access +- Cloud Ops: network/private endpoint access to LAW (if applicable) +- Arc agents installed on on-prem/HCI nodes +- Azure CLI 2.50+ with `az bicep` available + +## Phase 1 — Platform foundation (monitoring subscription) + +### 1.1 Deploy central LAW and shared DCRs + +```bash +# Production monitoring subscription +SUBSCRIPTION_ID= \ +RESOURCE_GROUP=azr-mon-rg-monitoring \ +PARAM_FILE=infra/parameters/monitoring.prod.bicepparam \ +./scripts/deploy.sh +``` + +Record outputs: + +- `logAnalyticsWorkspaceId` +- `defaultDcrId` +- `customDcrId` + +### 1.2 Verify deployment + +```bash +az monitor data-collection rule show \ + --name dcr-hybrid-default-shared \ + --resource-group azr-mon-rg-monitoring + +az monitor data-collection rule show \ + --name dcr-hybrid-custom-shared \ + --resource-group azr-mon-rg-monitoring +``` + +### 1.3 Policy assignments + +Policy assignments are deployed automatically when `deployPolicies=true`. Verify in Azure Portal: + +- **Hybrid monitoring — associate Default DCR by ObservabilityScope tag** +- **Hybrid monitoring — associate Custom DCR by Role tag** +- **Hybrid monitoring — deploy AMA on Arc Windows machines** + +Assign policies at management group level for all site subscriptions, or per subscription during phased rollout. + +## Phase 2 — Per-site / per-cluster onboarding + +### 2.1 Arc onboarding + +Ensure each HCI node and standalone server is registered: + +```bash +az connectedmachine list \ + --resource-group \ + --output table +``` + +### 2.2 Apply tags + +See [tagging-standard.md](tagging-standard.md). Minimum for baseline monitoring: + +``` +Site= +Env=Prod|Dev|Test +ObservabilityScope=hybrid-baseline +Cluster= +Role=HCI # optional; adds Custom DCR for storage/IIS extras +``` + +```bash +SUBSCRIPTION_ID= \ +RESOURCE_GROUP=azr-131-itsusra1-delldev01 \ +SITE=POC ENV=Prod ROLE=HCI \ +CLUSTER=azr-131-itsusra1-delldev01 \ +./scripts/apply-tags.sh --bulk-arc +``` + +### 2.3 Install Azure Monitor Agent + +If not deployed by policy: + +```bash +az connectedmachine extension create \ + --name AzureMonitorWindowsAgent \ + --publisher Microsoft.Azure.Monitor \ + --type AzureMonitorWindowsAgent \ + --resource-group \ + --machine-name +``` + +### 2.4 Verify DCR associations + +```bash +az monitor data-collection rule association list \ + --resource /subscriptions//resourceGroups//providers/Microsoft.HybridCompute/machines/ \ + --output table +``` + +Expected: associations to both `dcr-hybrid-default-shared` and `dcr-hybrid-custom-shared` (if `Role` tag set). + +### 2.5 Register Azure Local cluster (optional platform metrics) + +```bash +az resource show \ + --resource-group \ + --resource-type Microsoft.AzureStackHCI/clusters \ + --name +``` + +## Phase 3 — Validation + +### 3.1 Run automated checks + +```bash +SUBSCRIPTION_ID= \ +RESOURCE_GROUP=azr-mon-rg-monitoring \ +WORKSPACE_ID=azr-mon-law-central \ +CLUSTER_RG=azr-131-itsusra1-delldev01 \ +./scripts/pilot-validation.sh +``` + +### 3.2 Run KQL validation pack + +Open [`validation/onboarding-checks.kql`](../validation/onboarding-checks.kql) in Log Analytics. Replace `ClusterRg` with your cluster resource group. + +**Pass criteria:** + +| Check | Expected | +|-------|----------| +| Heartbeat | All HCI nodes appear with `Category=Azure Monitor Agent` | +| SDDC EventID 3000 | Recent inventory events per cluster | +| SDDC EventID 3002 | Volume/pool events for storage panels | +| Perf counters | `Processor Information`, `Memory`, `LogicalDisk`, `Network Interface`, `System` | +| InsightsMetrics | Present as fallback on Arc hosts | + +### 3.3 Validate Grafana AZR-208 dashboard + +1. Set datasource `${az_monitor}` to central LAW +2. Set `${subscription}` to site subscription +3. Set `${cluster}` to cluster RG name +4. Confirm panels: Server Stats, CPU, Memory, Disk, Network, Volume Usage, Node Up/Down + +Use [`list_computer_query`](../list_computer_query) for the `computer` variable (SDDC-based, no ARM Reader required). + +## Phase 4 — Migrate from legacy per-cluster DCR + +For POC cluster using `azr-131-dcr-delldev01`: + +1. Deploy shared `dcr-hybrid-default-shared` (Phase 1) +2. Tag all Arc machines with `ObservabilityScope=hybrid-baseline` +3. Wait for policy to create new association +4. Remove old association to `azr-131-dcr-delldev01` +5. Run validation; confirm AZR-208 panels unchanged +6. Decommission legacy DCR after 7-day parallel run + +## Phase 5 — Multi-site expansion + +Repeat Phase 2–3 for each site subscription: + +| Site | Subscription | Cluster RG example | +|------|-------------|-------------------| +| NYC | Sub-A | `azr-100-nyc-cluster01` | +| Sydney | Sub-B | `azr-200-syd-cluster01` | +| Dev/Test | Sub-A/B | `azr-dev-cluster01` | + +Use [`scripts/rollout-site.sh`](../scripts/rollout-site.sh) for a repeatable site checklist. + +## Troubleshooting + +| Symptom | Action | +|---------|--------| +| No Heartbeat | Verify AMA extension; check DCR association; confirm network egress | +| SDDC events missing | Confirm `Microsoft-Windows-SDDC-Management/Operational` in Default DCR | +| Perf counters partial on Arc | Expected — use explicit counters + InsightsMetrics; see `server_stat_test` | +| Policy not associating DCR | Verify `ObservabilityScope=hybrid-baseline` tag; trigger remediation | +| Grafana computer list empty | Use SDDC KQL variable; or grant ARM Reader (see grafana-rbac.md) | +| Hostname mismatch in panels | Queries use `HostKey = toupper(split(Computer,".")[0])` — no change needed | + +## Related files + +- Infrastructure: [`infra/main.bicep`](../infra/main.bicep) +- Default DCR spec: [`infra/modules/dcr-default.bicep`](../infra/modules/dcr-default.bicep) +- Custom DCR spec: [`infra/modules/dcr-custom.bicep`](../infra/modules/dcr-custom.bicep) +- Tagging: [tagging-standard.md](tagging-standard.md) +- Grafana RBAC: [grafana-rbac.md](grafana-rbac.md) diff --git a/tagging-standard.md b/tagging-standard.md new file mode 100644 index 0000000..8fbe337 --- /dev/null +++ b/tagging-standard.md @@ -0,0 +1,260 @@ +# Hybrid Monitoring Tagging Standard + +All Arc-enabled servers, Azure VMs, and cluster resource groups that report to the central Log Analytics workspace must carry the tags below. Tags mirror the architecture grouping **Site / Company → Cluster → Physical Node** and drive Azure Policy DCR associations and Grafana dashboard scoping. + +## Architecture alignment + +The monitoring design centralizes telemetry from multiple sites, companies, and clusters into one Log Analytics workspace via shared Data Collection Rules: + +``` +Site/Company A ── Cluster C1 ── Physical Node 1 (HCI OS + AMA) + │ └── Physical Node 2 (HCI OS + AMA) + └── Cluster C2 ── Physical Node 1 (HCI OS + AMA) + └── Physical Node 2 (HCI OS + AMA) + +Site/Company B ── Cluster C1 ── Physical Node 1 (HCI OS + AMA) + └── Physical Node 2 (HCI OS + AMA) + │ + ▼ + Data Collection Rules (Default + Custom) + │ + ▼ + Azure Log Analytics Workspace (Azure Region 1) +``` + +Each **physical node** is an Arc-enabled machine (`Microsoft.HybridCompute/machines`) with the Azure Monitor Windows Agent extension. Node identity in queries comes from `Computer` / `HostKey` in Heartbeat and Perf; tags identify **where** the node belongs in the hierarchy. + +## Tag hierarchy + +| Level | Diagram label | Tag | Example | +|-------|---------------|-----|---------| +| Organization | Site/Company A, Site/Company B | `Company` | `CompanyA`, `CompanyB` | +| Location / tenant scope | Site (within or across companies) | `Site` | `NYC`, `Sydney`, `SiteA` | +| Cluster instance | C1, C2 | `ClusterId` | `C1`, `C2` | +| Azure resource group | (maps to Grafana `${cluster}`) | `Cluster` | `azr-131-itsusra1-delldev01` | +| Physical node | Physical Node 1, Physical Node 2 | *(resource name)* | `itsusra1dazl001` | +| Platform | HCI OS | `Platform` | `HCI` | + +**Display grouping** in dashboards and reports should follow: `Company` → `Site` → `ClusterId` → node (`Computer`). + +## Required tags + +Apply to **Arc machines**, **Azure VMs**, and **cluster resource groups**. + +| Tag | Example values | Purpose | +|-----|----------------|---------| +| `Company` | `CompanyA`, `CompanyB` | Tenant / business-unit boundary (matches diagram Site/Company boxes) | +| `Site` | `NYC`, `Sydney`, `SiteA` | Geographic or logical site for grouping and cost/showback | +| `ClusterId` | `C1`, `C2` | Cluster instance within a company/site (matches diagram C1, C2) | +| `Cluster` | `azr-131-itsusra1-delldev01` | Azure resource group name — maps to Grafana `${cluster}` variable | +| `Env` | `Prod`, `Dev`, `Test` | Environment scoping | +| `Platform` | `HCI`, `Windows`, `Linux` | OS / platform type (HCI nodes use `HCI`) | +| `ObservabilityScope` | `hybrid-baseline` | Triggers **Default DCR** policy association | + +## Conditional tags + +| Tag | When required | Example values | Purpose | +|-----|---------------|----------------|---------| +| `Role` | When Custom DCR collection is needed | `HCI`, `Web`, `SQL`, `App`, `Infra` | Triggers **Custom DCR** policy association | + +Machines without a `Role` tag receive only the Default DCR. Machines with `Role=Web|SQL|App|HCI` receive **both** Default and Custom DCR associations. + +## Tag examples (per architecture diagram) + +### Site/Company A — Cluster C1 (Azure Local HCI, production) + +Represents the first cluster box under Company A in the architecture diagram. + +``` +Company=CompanyA +Site=SiteA +ClusterId=C1 +Cluster=azr-131-itsusra1-delldev01 +Env=Prod +Platform=HCI +ObservabilityScope=hybrid-baseline +Role=HCI +``` + +### Site/Company A — Cluster C2 + +Second cluster under the same company, different resource group. + +``` +Company=CompanyA +Site=SiteA +ClusterId=C2 +Cluster=azr-100-companya-c2-rg +Env=Prod +Platform=HCI +ObservabilityScope=hybrid-baseline +Role=HCI +``` + +### Site/Company B — Cluster C1 + +Separate company/tenant, own subscription or region. + +``` +Company=CompanyB +Site=SiteB +ClusterId=C1 +Cluster=azr-200-companyb-c1-rg +Env=Prod +Platform=HCI +ObservabilityScope=hybrid-baseline +Role=HCI +``` + +### Physical node (Arc machine) + +Tags are set on the Arc machine resource; node name is the machine name (e.g. `itsusra1dazl001`), not a tag. + +``` +Company=CompanyA +Site=SiteA +ClusterId=C1 +Cluster=azr-131-itsusra1-delldev01 +Env=Prod +Platform=HCI +ObservabilityScope=hybrid-baseline +Role=HCI +``` + +### IIS web server (Custom DCR) + +``` +Company=CompanyA +Site=NYC +ClusterId=C1 +Cluster=azr-200-web-rg +Env=Prod +Platform=Windows +ObservabilityScope=hybrid-baseline +Role=Web +``` + +### Dev/test cluster + +``` +Company=CompanyA +Site=SiteA +ClusterId=C1 +Cluster=azr-dev-cluster-rg +Env=Test +Platform=HCI +ObservabilityScope=hybrid-baseline +``` + +## Site-to-tag mapping (shared DCR model) + +All sites and companies use the same shared DCRs in the monitoring subscription. Differences are expressed through tags, not separate DCRs. + +| Diagram group | Company | Site | ClusterId | Cluster RG (example) | +|---------------|---------|------|-----------|----------------------| +| Site/Company A C1 | `CompanyA` | `SiteA` | `C1` | `azr-131-itsusra1-delldev01` | +| Site/Company A C2 | `CompanyA` | `SiteA` | `C2` | `azr-100-companya-c2-rg` | +| Site/Company B C1 | `CompanyB` | `SiteB` | `C1` | `azr-200-companyb-c1-rg` | + +## Apply tags + +Use the tagging script for single resources or bulk Arc machines in a cluster RG: + +```bash +# Bulk tag all Arc machines + RG in a cluster (Company A / C1 POC) +SUBSCRIPTION_ID= \ +RESOURCE_GROUP=azr-131-itsusra1-delldev01 \ +COMPANY=CompanyA \ +SITE=SiteA \ +CLUSTER_ID=C1 \ +ENV=Prod \ +PLATFORM=HCI \ +ROLE=HCI \ +CLUSTER=azr-131-itsusra1-delldev01 \ +./scripts/apply-tags.sh --bulk-arc + +# Single Arc machine (physical node) +SUBSCRIPTION_ID= \ +RESOURCE_GROUP=azr-131-itsusra1-delldev01 \ +RESOURCE_NAME=itsusra1dazl001 \ +COMPANY=CompanyA \ +SITE=SiteA \ +CLUSTER_ID=C1 \ +ENV=Prod \ +PLATFORM=HCI \ +ROLE=HCI \ +CLUSTER=azr-131-itsusra1-delldev01 \ +./scripts/apply-tags.sh +``` + +Or via Azure CLI: + +```bash +az tag update \ + --resource-id /subscriptions//resourceGroups//providers/Microsoft.HybridCompute/machines/ \ + --tags \ + Company=CompanyA \ + Site=SiteA \ + ClusterId=C1 \ + Cluster=azr-131-itsusra1-delldev01 \ + Env=Prod \ + Platform=HCI \ + ObservabilityScope=hybrid-baseline \ + Role=HCI +``` + +Tag the **cluster resource group** with the same `Company`, `Site`, `ClusterId`, `Cluster`, `Env`, and `Platform` values so RG-level resources inherit consistent scope. + +## Grafana and KQL scoping + +Full variable definitions: [grafana-variables.md](grafana-variables.md) + +| Order | Grafana variable | Tag / source | Notes | +|-------|------------------|--------------|-------| +| 1 | `az_monitor` | Data source | Central LAW | +| 2 | `law_subscription` | Monitoring sub (hidden) | LAW hosting subscription for `AzureResource` queries | +| 3 | `company` | `tags.Company` | Include All | +| 4 | `site` | `tags.Site` | Filtered by `${company}` | +| 5 | `cluster_id` | `tags.ClusterId` | Filtered by `${company}`, `${site}` | +| 6 | `env` | `tags.Env` | Filtered by hierarchy above | +| 7 | `cluster` | `tags.Cluster` (RG name) | Drives panel resource scope | +| 8 | `subscription` | `subscriptionId` from Arc machine | Site subscription for `${cluster}` | +| 9 | `computer` | SDDC 3000 + Heartbeat | Physical node — `grafana/queries/list_computer_query.kql` | +| 10 | `disk` / `pool` | SDDC 3002 | Volume/pool panels | + +Example KQL filter by hierarchy (when tags are available on `AzureResource` or custom columns): + +```kusto +| where tostring(Tags.Company) == "CompanyA" +| where tostring(Tags.Site) == "SiteA" +| where tostring(Tags.ClusterId) == "C1" +``` + +Cluster-scoped panels (current AZR-208 pattern): + +```kusto +| where _ResourceId has tolower("/resourcegroups/${cluster}/") +``` + +## Policy behavior + +| Tag state | Default DCR | Custom DCR | +|-----------|-------------|------------| +| `ObservabilityScope=hybrid-baseline` | Associated | — | +| `Role=Web` (or SQL/App/HCI) | Associated (if ObservabilityScope set) | Associated | +| Missing `ObservabilityScope` | Not associated by tag policy | — | + +After tagging, allow up to 30 minutes for Azure Policy remediation. Trigger on-demand remediation from the Policy compliance blade if needed. + +## Naming alignment + +| Concept | Convention | Example | +|---------|------------|---------| +| Diagram group | `Site/Company {Company} {ClusterId}` | Site/Company A C1 | +| Company tag | `Company{A\|B\|...}` | `CompanyA` | +| ClusterId tag | `C1`, `C2`, ... | `C1` | +| Cluster RG | `azr-{sub}-{site}{cluster}-{name}` | `azr-131-itsusra1-delldev01` | +| Arc machine (node) | hostname | `itsusra1dazl001` | +| Default DCR | `dcr-hybrid-default-shared` | Shared across all sites/companies | +| Custom DCR | `dcr-hybrid-custom-shared` | Shared across all sites/companies | +| Central LAW | `azr-mon-law-central` | Monitoring subscription, Azure Region 1 |