feat: return metadata in the SL API

[betodealmeida]

Summary

This PR adds richer metadata to the semantic layer API responses, allowing clients to provide a better UX. For example, this DJ metric:

---
type: metric
name: finance.total_revenue
display_name: Total Revenue
description: Total recognized revenue
query: SELECT SUM(revenue) FROM finance.orders
unit:
  kind: currency
  code: USD

Is returned as:

 {
   "id": "finance.total_revenue",
   "name": "total_revenue",
   "type": "floating",
   "definition": "finance.total_revenue",
   "description": "Total recognized revenue",
   "aggregation": "OTHER",
   "metadata": {
     "display_name": "Total Revenue",
     "semantic_type": "currency",
     "unit": {
       "kind": "currency",
       "code": "USD"
     },
     "format": {
       "preset": "currency"
     },
     "filter": {
       "kind": "number",
       "operators": [
         "=",
         "!=",
         ">",
         ">=",
         "<",
         "<=",
         "IS NULL",
         "IS NOT NULL"
       ],
       "default_operator": "="
     },
     "extensions": {
       "superset": {
         "d3format": "$,.2f"
       },
       "google_sheets": {
         "numberFormat": {
           "type": "CURRENCY",
           "pattern": "$#,##0.00"
         }
       }
     }
   }
 }

Fields are generated conservatively based on DJ metadata, while allowing for custom overrides:

 type: metric
 name: finance.total_revenue
 display_name: Total Revenue
 description: Total recognized revenue
 query: SELECT SUM(revenue) FROM finance.orders

 unit:
   kind: currency
   code: USD

 custom_metadata:
   semantic_layer:
     display_name: Net Revenue

     format:
       preset: currency
       precision: 0

     filter:
       kind: range
       operators:
         - "="
         - ">"
         - ">="
         - "<"
         - "<="
       default_operator: ">="

     extensions:
       superset:
         d3format: "$,.0f"

       google_sheets:
         numberFormat:
           type: CURRENCY
           pattern: "$#,##0"

@shangyian I wonder if it makes sense to eventually consolidate the "semantic" APIs that DJ has?

Test Plan

• PR has an associated issue: #
• make check passes
• make test shows 100% unit test coverage

Deployment Plan

[netlify]

✅ Deploy Preview for thriving-cassata-78ae72 canceled.

Name	Link
🔨 Latest commit	435123a
🔍 Latest deploy log	https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/6aa1c6cd13ded40008bd268a

[shangyian]

@betodealmeida can you clarify what you mean by consolidate semantic APIs? Do you mean the various semantic metadata APIs that we provide (e.g., under GET /metrics)?

[shangyian]

Hmm, so looking at this part, when a node's custom_metadata has no semantic_layer key, raw becomes the entirety of custom_metadata right? Is that intentional? I thought this would only return the dict under the semantic_layer key, since all other items on the the custom_metadata could be completely unrelated.

[shangyian]

minor: It looks like "duration" is not a member of UnitKind

[shangyian]

Could this part just reuse _fixed_decimal_pattern rather than inlining a similar expression?

[shangyian]

If I'm reading this setup correctly, this can't actually deploy successfully due to the owner field which isn't part of the semantic layer JSON schema. I wonder if you could add a test that actually asserts that failure?

[shangyian]

Does this mean that if a node has a semantic_layer key under custom_metadata, e.g.:

custom_metadata:
  semantic_layer:
    format: {preset: currency}
    owner: finance          # a random key

Then raw_metadata becomes {format: ..., owner: finance}, which would fail the JSON schema validation.

However, if a node doesn't have the semantic_layer key, e.g.:

custom_metadata:
  tier: 2
  something: "abcd"

Then _raw_column_metadata doesn't find the key, so raw_metadata just becomes everything in custom_metadata. I assume that's unintentional?

[betodealmeida]

@betodealmeida can you clarify what you mean by consolidate semantic APIs? Do you mean the various semantic metadata APIs that we provide (e.g., under GET /metrics)?

Yeah, sorry if I wasn't clear. What I mean is that we have the DJ-native API (GET /metrics/{name}, eg) and we have the semantic layer REST API (POST /semantic/views/{name}), and they also return metadata in different schemas. But I think they might be different enough for us to justify both, at least for now. For example, the POST /semantic/views/{name} supports passing additional configuration, so it allows clients to filter out branches they don't want. It's a more user-focused API, compared to the more engineering-centric native API, I think.

[shangyian]

@betodealmeida Yeah, I think for simplicity for the time being we might just want to keep both for now. If we do remove or try to consolidate, we'll need to look at existing consumers of those APIs and make sure they're migrated before we can remove.