Skip to content

esim: harden AtClient with retry loops and reconnect#37771

Merged
adeebshihadeh merged 11 commits intocommaai:masterfrom
greatgitsby:lpa-atclient-hardening
Apr 7, 2026
Merged

esim: harden AtClient with retry loops and reconnect#37771
adeebshihadeh merged 11 commits intocommaai:masterfrom
greatgitsby:lpa-atclient-hardening

Conversation

@greatgitsby
Copy link
Copy Markdown
Contributor

@greatgitsby greatgitsby commented Apr 6, 2026

split from #37742

Summary

  • AtClient: lazy serial open, _ensure_serial(reconnect) for recovery, retry in query on SerialException
  • open_isdr: retry loop (10 attempts, 250ms backoff, lte.sh modem reset at attempt 5)
  • send_apdu: retry loop (3 attempts, auto-reopen channel on failure)
  • Retry/delay constants extracted to top of file
  • DEBUG gated by env var instead of hardcoded True
  • esim.py: remove redundant modem reboot (handled internally by LPA now)

Test plan

Deployed to mici and tizi, ran 30x list_profiles stress test on both devices.

mici (comma mici, 725dfb9b) - 30/30 OK

[ 1] 4 profiles  active=mobi1     0.11s  OK
...
[30] 4 profiles  active=mobi1     0.09s  OK
Total: 30/30 OK, 0 failures

tizi (comma tizi, bb546e42) - 30/30 OK

[ 1] 3 profiles  active=mm1       0.11s  OK
...
[30] 3 profiles  active=mm1       0.08s  OK
Total: 30/30 OK, 0 failures

greatgitsby and others added 2 commits April 5, 2026 16:32
- AtClient: lazy serial open (deferred to first query), store device/baud
  for reconnect, add _reconnect_serial for recovery from serial exceptions
- open_isdr: extract _open_isdr_once, wrap in 10-attempt retry loop with
  lte.sh modem reset at attempt 3 for stuck SIM (CME ERROR 13)
- send_apdu: add retry loop with max_retries=3, auto-reopen channel on
  failure with backoff sleep
- query: catch SerialException and reconnect before retry
- close: wrap CCHC in try/except to handle already-closed channels
- Update DEFAULT_DEVICE to /dev/modem_at0, enable DEBUG by default
- base.py: add process_notifications() default, is_comma_profile uses
  provider field instead of ICCID prefix
- esim.py: remove post-mutation modem reboot (now handled by LPA internally)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@greatgitsby greatgitsby marked this pull request as draft April 6, 2026 05:41
greatgitsby and others added 3 commits April 5, 2026 22:43
- Replace debug print in open_isdr retry with cloudlog.warning
- Remove debug param from AtClient, use module-level DEBUG for AT output
- AT wire-level prints (SER >>/<<, DBUS >>/<<) stay as print() gated by DEBUG

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 6, 2026

Process replay diff report

Replays driving segments through this PR and compares the behavior to master.
Please review any changes carefully to ensure they are expected.

✅ 0 changed, 66 passed, 0 errors

greatgitsby and others added 4 commits April 6, 2026 07:27
Channel is opened on demand in send_apdu when needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Single method for opening the serial port, used by both query (lazy
open) and _reconnect_serial (close + reopen).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Single method with reconnect flag: _ensure_serial() for lazy open,
_ensure_serial(reconnect=True) to close and reopen.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@greatgitsby greatgitsby force-pushed the lpa-atclient-hardening branch 10 times, most recently from 44a6177 to 438e255 Compare April 7, 2026 04:51
lte.sh already blocks until the modem is online, so no post-reset sleep needed.
open_isdr and send_apdu retry loops recover without delays.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@greatgitsby greatgitsby force-pushed the lpa-atclient-hardening branch from 438e255 to 2f39a0d Compare April 7, 2026 04:57
@greatgitsby greatgitsby marked this pull request as ready for review April 7, 2026 05:04
@adeebshihadeh adeebshihadeh merged commit c7382f8 into commaai:master Apr 7, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants