feat: unify and improve identity building across snowflake connectors (#3997)

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

Author: mbani01

.claude/skills/scaffold-snowflake-connector/SKILL.md

@@ -93,7 +93,7 @@ Ask the user:
 If the user provides a path: verify it by checking `{path}/services/apps/snowflake_connectors/src/integrations/index.ts` exists. If confirmed, set `CROWD_DEV_ROOT = {path}`. Proceed to Phase 1.
 
 If the path doesn't exist or they need to clone:
-> "Run: `git clone git@github.com:linuxfoundation/crowd.dev.git`
+> "Run: `git clone https://github.com/linuxfoundation/crowd.dev.git`
 > Then provide the path to the cloned directory."
 
 Wait for a valid path before continuing.
@@ -307,12 +307,20 @@ Example format for an ambiguity:
 
 ### 3a. Identity Mapping
 
-**Rule:** Every member record must produce at least one identity with `type: MemberIdentityType.USERNAME`. The fallback chain is (try in order):
-1. Platform-native username column from schema
-2. LFID column value (used as username on `PlatformType.LFID`)
-3. Email value used as the platform USERNAME (last resort)
+**Rule:** Every member record must produce at least one identity with `type: MemberIdentityType.USERNAME`. The standard approach is to use the unified `buildMemberIdentities()` method on `TransformerBase` (added in the identity deduplication refactor). Only fall back to inline identity construction if the user explicitly requests it and can justify why the unified method cannot be used (e.g., fundamentally different identity shape not covered by the method's logic).
 
-If Pre-Analysis resolved email, username, and LFID columns with HIGH confidence and the user confirmed them, skip to the summary step below.
+The unified method covers the standard fallback chain automatically. Full behavior by case:
+
+| `platformUsername` | `lfUsername` | Identities produced |
+|---|---|---|
+| null | null | EMAIL(platform) + USERNAME(platform, email) |
+| set | null | EMAIL(platform) + USERNAME(platform, platformUsername) |
+| null | set | EMAIL(platform) + USERNAME(LFID, lfUsername) + USERNAME(platform, lfUsername) |
+| set | set | EMAIL(platform) + USERNAME(platform, platformUsername) + USERNAME(LFID, lfUsername) |
+
+**Critical:** Never pass `lfUsername` as `platformUsername`. When a source only has an LFID column (no platform-native username), pass `platformUsername: null` — the lfUsername-only path (row 3 above) already produces the correct USERNAME identity for the platform using the lfUsername value.
+
+If Pre-Analysis resolved email, platformUsername, and LFID columns with HIGH confidence and the user confirmed them, skip to the summary step below.
 
 For any unresolved identity field, use this pattern:
 - **Multiple candidates found**: "I see columns `A` and `B` that could be the email — which one?" (present choices, not open-ended)
@@ -325,8 +333,10 @@ For each confirmed identity column also confirm:
 
 **Critical:** If a JOIN table for users is NOT the same table used by an existing implementation, validate every column explicitly regardless of Pre-Analysis confidence. Column name heuristics alone are not sufficient for unknown tables.
 
-After all identity fields are confirmed, summarize the full identity-building logic and ask:
-> "Here is how identities will be built: [summary]. Does this look correct?"
+After all identity fields are confirmed, summarize how `buildMemberIdentities()` will be called and ask:
+> "Here is how identities will be built:
+> `this.buildMemberIdentities({ email, sourceId: [col or null], platformUsername: [col or null], lfUsername: [col or null] })`
+> Does this look correct?"
 
 ---
 
@@ -578,11 +588,16 @@ File: `services/apps/snowflake_connectors/src/integrations/{platform}/{source}/t
 - All string comparisons must be case-insensitive: use `.toLowerCase()` on both sides of comparison only; preserve the original value in the output
 - No broad `else` statements — every branch must have an explicit condition
 - All column names referenced in code must exactly match the schema registry — never assumed
-- Identity fallback chain (always produces at least one USERNAME identity):
-  1. If platform-native username column present and non-null → push EMAIL + USERNAME identities for platform
-  2. Else if LFID present and non-null → push EMAIL for platform + LFID value as USERNAME for platform
-  3. Else → push EMAIL value as USERNAME for platform (email-as-username)
-- After building platform identities, if LFID column is present and non-null → push separate LFID identity: `{ platform: PlatformType.LFID, value: lfid, type: MemberIdentityType.USERNAME, ... }`
+- **Identity building — always use `this.buildMemberIdentities()` first (preferred):**
+  - Call `this.buildMemberIdentities({ email, sourceId, platformUsername, lfUsername })` from `TransformerBase`
+  - Always pass all 4 arguments explicitly, even when the value is `null` or `undefined` — never omit an argument
+  - `sourceId` = the user ID column from the source table (`undefined` if the table has no user ID column)
+  - `platformUsername` = the platform-native username column (`null` if absent — do NOT substitute `lfUsername` here)
+  - `lfUsername` = the LFID column value (`null` if absent)
+  - **Never pass `lfUsername` as `platformUsername`** — when a source only has an LFID column, pass `platformUsername: null`; the method's lfUsername-only path already produces the correct platform USERNAME from the lfUsername value
+  - The method handles the full fallback chain automatically (see the 4-case table in §3a)
+  - Do NOT import `IMemberData` or `MemberIdentityType` in the transformer — those are only needed if falling back to inline construction
+  - **Only use inline identity construction if the user explicitly requests it and justifies why `buildMemberIdentities()` cannot be used** (e.g., non-standard identity shape not covered by the method). Document the justification in a comment.
 - `isIndividualNoAccount` must call `this.isIndividualNoAccount(displayName)` from `TransformerBase` — never reimplement
 - **Do not set member attributes** (e.g., `MemberAttributeName.JOB_TITLE`, `AVATAR_URL`, `COUNTRY`) unless: (a) the user explicitly requested them, or (b) the same table and column are already used for that attribute in an existing implementation — in which case follow the existing pattern exactly
 - Extends the platform base class if one was confirmed in Phase 3e; otherwise extends `TransformerBase` directly

services/apps/snowflake_connectors/src/core/transformerBase.ts

@@ -5,7 +5,7 @@
  * how raw exported data is transformed into activities.
  */
 import { getServiceChildLogger } from '@crowd/logging'
-import { IActivityData, PlatformType } from '@crowd/types'
+import { IActivityData, IMemberData, MemberIdentityType, PlatformType } from '@crowd/types'
 
 const log = getServiceChildLogger('transformer')
 
@@ -38,6 +38,70 @@ export abstract class TransformerBase {
     return TransformerBase.INDIVIDUAL_NO_ACCOUNT_RE.test(displayName.trim())
   }
 
+  protected buildMemberIdentities(params: {
+    email: string
+    sourceId?: string
+    platformUsername?: string | null
+    lfUsername?: string | null
+  }): IMemberData['identities'] {
+    const { email, sourceId, platformUsername, lfUsername } = params
+    const identities: IMemberData['identities'] = [
+      {
+        platform: this.platform,
+        value: email,
+        type: MemberIdentityType.EMAIL,
+        verified: true,
+        verifiedBy: this.platform,
+        sourceId,
+      },
+    ]
+    if (!lfUsername && !platformUsername) {
+      identities.push({
+        platform: this.platform,
+        value: email,
+        type: MemberIdentityType.USERNAME,
+        verified: true,
+        verifiedBy: this.platform,
+        sourceId,
+      })
+      return identities
+    }
+
+    if (platformUsername) {
+      identities.push({
+        platform: this.platform,
+        value: platformUsername,
+        type: MemberIdentityType.USERNAME,
+        verified: true,
+        verifiedBy: this.platform,
+        sourceId,
+      })
+    }
+
+    if (lfUsername) {
+      identities.push({
+        platform: PlatformType.LFID,
+        value: lfUsername,
+        type: MemberIdentityType.USERNAME,
+        verified: true,
+        verifiedBy: this.platform,
+        sourceId,
+      })
+      if (!platformUsername) {
+        identities.push({
+          platform: this.platform,
+          value: lfUsername,
+          type: MemberIdentityType.USERNAME,
+          verified: true,
+          verifiedBy: this.platform,
+          sourceId,
+        })
+      }
+    }
+
+    return identities
+  }
+
   /**
    * Safe wrapper around transformRow that catches errors and returns null.
    */

services/apps/snowflake_connectors/src/integrations/cvent/event-registrations/transformer.ts

@@ -2,10 +2,8 @@ import { CVENT_GRID, CventActivityType } from '@crowd/integrations'
 import { getServiceChildLogger } from '@crowd/logging'
 import {
   IActivityData,
-  IMemberData,
   IOrganizationIdentity,
   MemberAttributeName,
-  MemberIdentityType,
   OrganizationIdentityType,
   OrganizationSource,
   PlatformType,
@@ -34,55 +32,19 @@ export class CventTransformer extends TransformerBase {
     }
 
     const registrationId = (row.REGISTRATION_ID as string)?.trim()
+    const sourceId = (row.USER_ID as string | null) || undefined
 
     const displayName =
       fullName ||
       (firstName && lastName ? `${firstName} ${lastName}` : firstName || lastName) ||
       userName
 
-    const identities: IMemberData['identities'] = []
-    const sourceId = (row.USER_ID as string | null) || undefined
-
-    if (userName) {
-      identities.push(
-        {
-          platform: PlatformType.CVENT,
-          value: email,
-          type: MemberIdentityType.EMAIL,
-          verified: true,
-          verifiedBy: PlatformType.CVENT,
-          sourceId,
-        },
-        {
-          platform: PlatformType.CVENT,
-          value: userName,
-          type: MemberIdentityType.USERNAME,
-          verified: true,
-          verifiedBy: PlatformType.CVENT,
-          sourceId,
-        },
-      )
-    } else {
-      identities.push({
-        platform: PlatformType.CVENT,
-        value: email,
-        type: MemberIdentityType.USERNAME,
-        verified: true,
-        verifiedBy: PlatformType.CVENT,
-        sourceId,
-      })
-    }
-
-    if (lfUsername) {
-      identities.push({
-        platform: PlatformType.LFID,
-        value: lfUsername,
-        type: MemberIdentityType.USERNAME,
-        verified: true,
-        verifiedBy: PlatformType.CVENT,
-        sourceId,
-      })
-    }
+    const identities = this.buildMemberIdentities({
+      email,
+      sourceId,
+      platformUsername: userName,
+      lfUsername,
+    })
 
     const type =
       row.USER_ATTENDED === true

services/apps/snowflake_connectors/src/integrations/tnc/certificates/transformer.ts

@@ -1,12 +1,6 @@
 import { TNC_GRID, TncActivityType } from '@crowd/integrations'
 import { getServiceChildLogger } from '@crowd/logging'
-import {
-  IActivityData,
-  IMemberData,
-  MemberAttributeName,
-  MemberIdentityType,
-  PlatformType,
-} from '@crowd/types'
+import { IActivityData, MemberAttributeName, PlatformType } from '@crowd/types'
 
 import { TransformedActivity } from '../../../core/transformerBase'
 import { TncTransformerBase } from '../tncTransformerBase'
@@ -24,47 +18,14 @@ export class TncCertificatesTransformer extends TncTransformerBase {
     const certificateId = (row.CERTIFICATE_ID as string)?.trim()
     const learnerName = (row.LEARNER_NAME as string | null)?.trim() || null
     const lfUsername = (row.LFID as string | null)?.trim() || null
-
-    const identities: IMemberData['identities'] = []
     const sourceId = (row.USER_ID as string | null) || undefined
 
-    if (lfUsername) {
-      identities.push(
-        {
-          platform: PlatformType.TNC,
-          value: email,
-          type: MemberIdentityType.EMAIL,
-          verified: true,
-          verifiedBy: PlatformType.TNC,
-          sourceId,
-        },
-        {
-          platform: PlatformType.TNC,
-          value: lfUsername,
-          type: MemberIdentityType.USERNAME,
-          verified: true,
-          verifiedBy: PlatformType.TNC,
-          sourceId,
-        },
-        {
-          platform: PlatformType.LFID,
-          value: lfUsername,
-          type: MemberIdentityType.USERNAME,
-          verified: true,
-          verifiedBy: PlatformType.TNC,
-          sourceId,
-        },
-      )
-    } else {
-      identities.push({
-        platform: PlatformType.TNC,
-        value: email,
-        type: MemberIdentityType.USERNAME,
-        verified: true,
-        verifiedBy: PlatformType.TNC,
-        sourceId,
-      })
-    }
+    const identities = this.buildMemberIdentities({
+      email,
+      sourceId,
+      platformUsername: null,
+      lfUsername,
+    })
 
     const activity: IActivityData = {
       type: TncActivityType.ISSUED_CERTIFICATION,

services/apps/snowflake_connectors/src/integrations/tnc/courses/transformer.ts

@@ -1,12 +1,6 @@
 import { TNC_GRID, TncActivityType } from '@crowd/integrations'
 import { getServiceChildLogger } from '@crowd/logging'
-import {
-  IActivityData,
-  IMemberData,
-  MemberAttributeName,
-  MemberIdentityType,
-  PlatformType,
-} from '@crowd/types'
+import { IActivityData, MemberAttributeName, PlatformType } from '@crowd/types'
 
 import { TransformedActivity } from '../../../core/transformerBase'
 import { TncTransformerBase } from '../tncTransformerBase'
@@ -32,47 +26,14 @@ export class TncCoursesTransformer extends TncTransformerBase {
 
     const learnerName = (row.LEARNER_NAME as string | null)?.trim() || null
     const lfUsername = (row.LFID as string | null)?.trim() || null
+    const sourceId = (row.INTERNAL_TI_USER_ID as string | null) || undefined
 
-    const identities: IMemberData['identities'] = []
-    const sourceId = undefined
-
-    if (lfUsername) {
-      identities.push(
-        {
-          platform: PlatformType.TNC,
-          value: email,
-          type: MemberIdentityType.EMAIL,
-          verified: true,
-          verifiedBy: PlatformType.TNC,
-          sourceId,
-        },
-        {
-          platform: PlatformType.TNC,
-          value: lfUsername,
-          type: MemberIdentityType.USERNAME,
-          verified: true,
-          verifiedBy: PlatformType.TNC,
-          sourceId,
-        },
-        {
-          platform: PlatformType.LFID,
-          value: lfUsername,
-          type: MemberIdentityType.USERNAME,
-          verified: true,
-          verifiedBy: PlatformType.TNC,
-          sourceId,
-        },
-      )
-    } else {
-      identities.push({
-        platform: PlatformType.TNC,
-        value: email,
-        type: MemberIdentityType.USERNAME,
-        verified: true,
-        verifiedBy: PlatformType.TNC,
-        sourceId,
-      })
-    }
+    const identities = this.buildMemberIdentities({
+      email,
+      sourceId,
+      platformUsername: null,
+      lfUsername,
+    })
 
     const productType = (row.PRODUCT_TYPE as string | null)?.trim() || null