Skip to content

Third-party app guidance is fragile: hardcoded English button-text matching #10

Description

@NITISH-R-G

Objective

Replace or supplement book_taxi.json's hardcoded text_any English-string matching ("Where to", "Confirm", "Book", "Request", ...) against Uber/Ola/Rapido with something that survives those apps' normal update cadence and regional-language UI.

Rationale

Verified live this session: Uber's actual destination field is only reachable via the content-description "Where to?" (confirmed via a real uiautomator dump against the installed app, see this session's history), which happens to match today. But there is no fallback, no version awareness, and no handling if Uber/Ola/Rapido A/B test their copy, ship a regional-language build, or simply rename a button - all of which are normal occurrences for apps with hundreds of millions of users on independent release cycles this project doesn't control.

Acceptance criteria

  • A step whose text_any candidates match nothing on screen fails visibly (debug panel note, e.g. "target not found in ") rather than silently pointing at the wrong element or nothing
  • At least one fallback matching strategy beyond exact/substring English text - candidates to evaluate: fuzzy/semantic text similarity, matching by element role + position heuristics (e.g. "the first prominent EditText near the top of a ride-hailing app's home screen"), or accepting that this needs a per-app adapter rather than one generic text_any list
  • Whatever strategy is chosen, document why it was chosen over the alternatives (this issue exists partly because that decision needs to be made deliberately, not under demo time pressure)

Implementation steps

  1. Decide the matching strategy (see acceptance criteria) - this is a design decision, not a quick patch, and should probably get its own short design note before code
  2. Implement it behind the existing TaskStep.textAny mechanism if possible, so the task DSL contract doesn't need to change
  3. Add a "not found" telemetry path to the debug panel so a demo-day failure is diagnosable instead of silent
  4. Re-verify live against the actual installed Uber/Ola/Rapido builds - this needs a physical device with those apps installed, not just unit tests

Estimated effort

Medium-high (2-3 days: mostly design/decision time, moderate implementation once the strategy is chosen)

Risks

  • Any fallback strategy trades precision for robustness - a fuzzy match is more likely to point at the wrong element, which is arguably worse than a clear "not found"
  • This cannot be meaningfully unit-tested against the real apps (their UI trees aren't available offline) - verification requires a physical device with each app installed and will need re-verification whenever those apps update

Testing checklist

  • Manual device test against current Uber, Ola, and Rapido builds
  • Manual test: temporarily rename text_any candidates to simulate an app UI change, confirm the failure mode is a visible "not found," not a silent wrong-target point
  • Re-run after the next update to any of the three apps, to establish whether the chosen strategy actually survives real-world drift (this is the entire point of the issue)

Metadata

Metadata

Assignees

No one assigned

    Labels

    audit-followupFlagged by the repo audit; too large/risky for a small PRbugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions