-
Notifications
You must be signed in to change notification settings - Fork 151
Add privacy-safe PM user research pipeline #1096
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ross0x01
wants to merge
3
commits into
main
Choose a base branch
from
codex/pm-user-research
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| --- | ||
| name: hackerai-user-research | ||
| description: Run privacy-safe HackerAI customer research from a Linear question and a product-data cohort. Use for requests to understand user types, recurring jobs, workflows, friction, value drivers, reasons to pay, or customer avatars from actual HackerAI messages, including HAC-65-style top-spender research. Also use when a PM asks how to run, repeat, or interpret the `pm-user-research` Trigger task. Do not use for support investigations, decisions about one person's eligibility or risk, or exporting raw customer content. | ||
| --- | ||
|
|
||
| # HackerAI User Research | ||
|
|
||
| Turn a research question and 3-20 internal user IDs into restricted per-user | ||
| profiles and an aggregated cohort report. The deployed task samples messages, | ||
| redacts sensitive data, and uses Grok 4.6 with reasoning disabled. | ||
|
|
||
| Read [references/privacy-policy.md](references/privacy-policy.md) and | ||
| [references/pm-runbook.md](references/pm-runbook.md) before running the task. | ||
|
|
||
| ## Workflow | ||
|
|
||
| 1. Read the owning Linear issue. Extract the research question, cohort rule, | ||
| exclusions, requested output, and privacy constraints. Confirm the responsible | ||
| owner explicitly approved customer-message research. If no approved issue | ||
| exists, create or update one and stop until approval is recorded; creating the | ||
| issue does not itself grant approval. | ||
| 2. Select the cohort in PostHog. Use Stripe-synced revenue in PostHog when its | ||
| freshness and account mapping are sufficient. Check Stripe directly only for | ||
| unmatched customers, refunds/disputes, payer-versus-user ambiguity, or other | ||
| reconciliation gaps. Never use Google Drive. | ||
| 3. Resolve each cohort member to the internal user ID used by Convex. Exclude | ||
| internal/test/fraud accounts and deduplicate payer or organization | ||
| relationships before triggering analysis. Stop unless 3-20 unique internal | ||
| user IDs remain after filtering. | ||
| 4. Discover the Trigger task `pm-user-research` and inspect its current schema. | ||
| Trigger it in the intended environment with the Linear issue ID, exact | ||
| question, descriptive cohort label, 3-20 unique user IDs, PM name/handle, and | ||
| optional chat limit. Never call the worker task directly. | ||
| 5. Wait for the run to complete. Keep the returned `analysisId`; it is the audit | ||
| and lookup key for the restricted Convex records. | ||
| 6. Present only the aggregate answer, evidence coverage, supported user types, | ||
| avatars, primary/secondary target, confidence, unknowns, and experiments. | ||
| Detailed pseudonym-level profiles remain in restricted Convex records and are | ||
| not returned through Trigger. | ||
| 7. Update Linear only when asked. Copy aggregate findings, coverage, confidence, | ||
| unknowns, and experiments. Never copy cohort IDs, pseudonym-level profiles, | ||
| raw evidence, direct identifiers, or per-user findings or targeting decisions. | ||
|
|
||
| ## Trigger payload | ||
|
|
||
| Use the current task schema as the authority. A typical HAC-65 run is: | ||
|
|
||
| ```json | ||
| { | ||
| "linearIssueId": "HAC-65", | ||
| "question": "What kinds of users are our highest-spending customers, what recurring work do they use HackerAI for, and why do they pay?", | ||
| "cohortLabel": "Top 10 users by reconciled lifetime net paid spend", | ||
| "userIds": ["internal-user-id-1", "internal-user-id-2", "internal-user-id-3"], | ||
| "requestedBy": "PM name or handle", | ||
| "maxChatsPerUser": 12 | ||
| } | ||
| ``` | ||
|
|
||
| Do not place email addresses, Stripe customer IDs, or message content in the | ||
| payload. `userIds` must be the internal Convex/WorkOS user IDs. | ||
|
|
||
| ## Quality checks | ||
|
|
||
| - Treat a profile as directional when fewer than three chats were available or | ||
| confidence is low. | ||
| - Verify `usersAnalyzed`, `chatsReviewed`, and `messagesReviewed` before using a | ||
| conclusion. | ||
| - Do not turn one-off requests into an avatar. Prefer patterns supported across | ||
| multiple chats and users. | ||
| - Keep observed product behavior separate from acquisition or messaging | ||
| hypotheses. | ||
| - Say `unknown` when the evidence does not establish context. Never infer a | ||
| company or occupation from an email address. | ||
| - A failed or partial run is not permission to inspect messages manually. Fix | ||
| cohort mapping or deployment/configuration and rerun the bounded task. | ||
|
|
||
| ## Result boundary | ||
|
|
||
| The Trigger result contains only aggregate internal research. Detailed profiles | ||
| remain restricted and deletion-aware in Convex. The aggregate report is the only | ||
| part that may be copied to Linear, under the owning issue's privacy rules. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| interface: | ||
| display_name: "HackerAI User Research" | ||
| short_description: "Analyze user workflows from private product evidence" | ||
| default_prompt: "Use $hackerai-user-research to answer this Linear research question from privacy-safe product evidence." |
38 changes: 38 additions & 0 deletions
38
.agents/skills/hackerai-user-research/references/pm-runbook.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| # PM runbook | ||
|
|
||
| ## 1. Prepare the cohort | ||
|
|
||
| Start only after the responsible owner explicitly approves customer-message | ||
| research in the Linear issue. Creating or editing the issue is not approval. For | ||
| spend-ranked research, use | ||
| PostHog's Stripe-synced lifetime net paid amount when available and current. | ||
| Exclude refunds/disputes, internal and test users, fraud, duplicates, and | ||
| unmatched customers. Open Stripe only to resolve discrepancies or payer/account | ||
| ownership. Produce internal Convex/WorkOS user IDs, not emails or Stripe IDs. | ||
|
|
||
| ## 2. Run through Codex | ||
|
|
||
| Ask Codex: | ||
|
|
||
| > Use $hackerai-user-research for HAC-65. Select the reconciled top-spender | ||
| > cohort, run the analysis, wait for it, and give me the aggregate findings with | ||
| > coverage, confidence, unknowns, and recommended experiments. | ||
|
|
||
| Codex should use Trigger's task discovery/schema tools, trigger | ||
| `pm-user-research` in production, then wait for completion. The task runs one | ||
| parallel worker per user and a final cohort synthesis. Both calls use | ||
| `x-ai/grok-4.6` with OpenRouter reasoning explicitly disabled and zero-data- | ||
| retention routing required. | ||
|
|
||
| ## 3. Interpret the result | ||
|
|
||
| Use the aggregate report to understand supported user types, customer avatars, | ||
| and decisions. Always include coverage and confidence. Detailed profiles remain | ||
| restricted in Convex. Treat acquisition channels and marketing messages as | ||
| hypotheses until a separate experiment validates them. | ||
|
|
||
| ## 4. Share safely | ||
|
|
||
| Keep the complete Trigger result and Convex records restricted. A Linear update | ||
| may include only the aggregate answer, avatars, coverage, confidence, unknowns, | ||
| and experiments. Do not include the cohort IDs or pseudonym-level profiles. |
36 changes: 36 additions & 0 deletions
36
.agents/skills/hackerai-user-research/references/privacy-policy.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # Customer research privacy policy | ||
|
|
||
| Use customer messages only for an approved, specific internal research purpose. | ||
| The owning Linear issue must define the cohort and intended output. | ||
|
|
||
| ## Allowed | ||
|
|
||
| - Product behavior: recurring jobs, Ask/Agent usage, workflow stages, broad tool | ||
| or environment categories, friction, value, and reasons to pay. | ||
| - Analyst-visible per-user profile content uses generated pseudonyms. Restricted | ||
| Convex records retain the internal user ID needed for deletion and lifecycle | ||
| handling; never expose the pseudonym-to-user linkage. | ||
| - Cohort-level avatars, confidence, unknowns, and testable hypotheses. | ||
|
|
||
| ## Prohibited | ||
|
|
||
| - Sensitive-trait, demographic, health, political, religious, sexual, or other | ||
| personal profiling. | ||
| - Inferring identity, employer, company, occupation, geography, or legitimacy | ||
| from an email address or isolated clue. | ||
| - User contact, sales outreach, public marketing claims, eligibility decisions, | ||
| fraud decisions, or adverse actions based on this research. | ||
| - Copying cohort IDs, pseudonym-level profiles, raw prompts, transcripts, direct | ||
| quotes, identifiers, evidence, files, code, commands, payloads, secrets, or | ||
| per-user targets, findings, or targeting decisions into Linear or task output. | ||
| - Sanitized aggregate report fields may be copied to the approved Linear issue. | ||
| - Manual browsing as a fallback when the bounded task fails. | ||
|
|
||
| ## Storage and deletion | ||
|
|
||
| Raw message excerpts exist only in the analysis worker's memory and model | ||
| request. Model calls require an OpenRouter zero-data-retention route and fail | ||
| closed if no such Grok 4.6 endpoint is available. Convex stores the run audit, | ||
| pseudonymized structured profiles, and the aggregate report. Account deletion | ||
| removes that user's stored profile and run-membership linkage; runs and reports | ||
| are retained only as cohort-level outputs from cohorts of at least three. | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.