a better remote body teleoperation experience by stefpi · Pull Request #37732 · commaai/openpilot

stefpi · 2026-03-25T22:31:53Z

changes

layouts

new layouts and widgets specifically when attached to Body V1 or V2
- tici and mici onroad/home screens with faces and connect buttons for easy connection
- tici screen has a new sidebar which appears from the top, mici preserves all other screens
- raylib dot-matrix face animations (normal, asleep, sleepy, inquisitive, wink), smooth transitions, and action-based animations (turning)
- can now connect one-time via QR code and control device through connect app
- tici and mici main layouts detect notCar and swap in body-specific layouts/sidebar

webrtcd

1 video channel
- camera switching via datachannel param + frame stream going into stream encoder
  - New LivestreamCamera param
- Stream encoder bitrate bumped from 1 Mbps to 4 Mbps, GOP reduced from 15 to 5
- Removed road camera mapping from stream encoder (only driver and wideRoad remain)
1 audio channel
latency measurement only when "stats" menu is open on frontend with metadata injection into H264 stream before packets are sent
better error handling for different scenarios so end user can know why connections failed
- other people on stream already
- ignition is not on
HTTPS with auto-generated self-signed certs, /trust endpoint for cert acceptance. This is required for connecting to the device on local network but through a website hosted somewhere else which is using HTTPS.
CORS headers including Access-Control-Allow-Private-Network

soundd

sound bite requests from frontend go to /stream on webrtc and then post a message on cereal, which is then picked up by the sound daemon and then that alert sound is queued up to be played
- this was done because soundd already takes up the audio drivers, so instead of unloading and reloading, I thought it would be easier to pass it to soundd directly.
New soundRequest cereal service for remote sound triggers

athena

New athenad getNotCar() RPC reading CarParamsPersistent
New athenad startJoystickStream(sdp) RPC posting to local webrtcd

debug

New bodyview.py debug script to simulate body UI (--big, --joystick, --monitor)
New body_injection_test.py for body controller testing

process manager

micd now also runs on body (changed from iscar to only_onroad)
dmonitoringmodeld and dmonitoringd disabled on body (also ignored in selfdrived)

other

Body enable delay reduced from 5s to 2s
Pandad disables IR LEDs on body (checks CarParamsPersistent every 100 frames)
Removed old webjoystick process
Entire old body teleop web UI (tools/bodyteleop/) removed
CONNECT_HOST / CONNECT_HOST_DISPLAY env-overridable constants used across all pairing dialogs and settings. Useful for local, preview and deployed QR code generation and link display

Related PRs that this PR depends on:

opendbc (body v1 and v2 support): Body V2 opendbc#2972
panda (body v2 firmware updates + others): Body Hoverboard Motors Implementation panda#2308
teleoprtc (fixes for webrtc connections in firefox): better webrtc support for firefox teleoprtc#10
connect: remote web joystick in connect connect#584

…r screen

…tream for better error finding

…webrtc negotiation for better error handling

… about being starved during cereal bridge + webrtc connection

github-actions · 2026-03-26T00:10:18Z

UI Preview

mici: ⚠️ Videos differ! View Diff Report
big: ⚠️ Videos differ! View Diff Report

sshane · 2026-03-26T02:37:44Z

selfdrive/ui/body/widgets/body_pairing.py

+    ssid = self._wifi_manager.connected_ssid
+    if ssid and len(ssid) > 20:
+      ssid = ssid[:17] + "..."
+    wifi_text = f"WiFi: {ssid}" if ssid else "wifi: not connected"
+    rl.draw_text_ex(self._font, wifi_text, rl.Vector2(label_x, y), 22, 0, MICI_TEXT_COLOR)


you can use a UnifiedLabel for automatic eliding (the ...) and positioning, and VBoxLayout to remove a bunch of code as well

sshane · 2026-03-26T02:39:49Z

selfdrive/ui/body/widgets/body_pairing.py

+    for _ in gui_app.render():
+      pass


Suggested change

for _ in gui_app.render():

pass

gui_app.render()

you can just do this now

sshane · 2026-03-26T02:40:28Z

selfdrive/ui/body/widgets/sidebar_tici.py

why is this all duplicated?

sshane · 2026-03-26T02:41:17Z

selfdrive/ui/body/widgets/sidebar_tici.py

+  # def _draw_flag_button(self, rect: rl.Rectangle):
+  #   if not ui_state.started:
+  #     return
+
+  #   mouse_pos = rl.get_mouse_position()
+  #   mouse_down = self.is_pressed and rl.is_mouse_button_down(rl.MouseButton.MOUSE_BUTTON_LEFT)
+
+  #   btn_x = int(rect.x + rect.width - 100)
+  #   btn_y = int(rect.y + 30)
+  #   flag_rect = rl.Rectangle(btn_x, btn_y, 60, 60)
+  #   flag_pressed = mouse_down and rl.check_collision_point_rec(mouse_pos, flag_rect)
+  #   tint = Colors.BUTTON_PRESSED if flag_pressed else Colors.BUTTON_NORMAL
+  #   rl.draw_texture(self._flag_img, btn_x, btn_y, tint)


unused code

sshane · 2026-03-26T02:42:05Z

selfdrive/ui/layouts/main.py

-    self._layouts = {MainState.HOME: HomeLayout(), MainState.SETTINGS: SettingsLayout(), MainState.ONROAD: AugmentedRoadView()}
+    if self._is_body:
+      self._layouts = {MainState.HOME: BodyLayout(), MainState.SETTINGS: SettingsLayout(), MainState.ONROAD: BodyLayout()}
+      self._sidebar.set_visible(False)


you can just make the body side bar set itself not visible

sshane · 2026-03-26T03:29:40Z

selfdrive/ui/body/widgets/body_pairing.py

+class _PairingBigButton(BigButton):
+  def _get_label_font_size(self):
+    return 64
+
+
+class MiciBodyPairingScreen(NavScroller):
+  """MICI pairing screen: NavScroller with BigButtons for each pairing method."""
+
+  def __init__(self):
+    super().__init__()
+
+    self._wifi_manager = WifiManager()
+    self._wifi_manager.set_active(False)
+
+    self._connect_panel = OneTimeConnectPanel()
+    connect_btn = _PairingBigButton("connect", "",
+                                    gui_app.texture("icons_mici/settings/network/wifi_strength_full.png", 76, 56))
+    connect_btn.set_click_callback(lambda: gui_app.push_widget(self._connect_panel))
+
+    self._connect_btn = connect_btn
+    self._scroller.add_widgets([connect_btn])


wait you have two pairing screens? I don't see this one when I click connet on home screen for mici

sshane · 2026-03-26T03:30:34Z

selfdrive/ui/body/widgets/sidebar_tici.py

+    self.color = color
+
+
+class BodySidebar(Widget):


just subclass Sidebar and override render?

sshane · 2026-03-26T03:33:02Z

selfdrive/ui/layouts/main.py

+    if self._is_body:
+      self._layouts[MainState.HOME].set_click_callback(self._on_onroad_clicked)
+      self._layouts[MainState.ONROAD].set_click_callback(self._on_onroad_clicked)
+


This breaks tapping onroad for cars now! Why do we do this?

sshane · 2026-03-26T03:34:09Z

selfdrive/ui/layouts/main.py

+      # Body sidebar always starts closed; regular sidebar starts open
+      if not self._is_body:
+        self._sidebar.set_visible(True)


another splittable PR, if this was using the nav stack each sidebar could manage its own show event behavior ;)

sshane · 2026-03-26T03:36:15Z

selfdrive/ui/layouts/main.py

+    if self._is_body: # overlay sidebar but recompute boundaries for proper click events
+      parent_rect = rl.Rectangle(self._rect.x, self._rect.y + BODY_SIDEBAR_HEIGHT,
+                                 self._rect.width, self._rect.height - BODY_SIDEBAR_HEIGHT) if self._sidebar.is_visible else None
+      self._layouts[self._current_mode].set_parent_rect(parent_rect)
+      self._layouts[self._current_mode].render(self._rect)
+      if self._sidebar.is_visible:
+        self._sidebar.render(self._sidebar_rect)
+    else:
+      if self._sidebar.is_visible:
+        self._sidebar.render(self._sidebar_rect)
+      content_rect = self._content_rect if self._sidebar.is_visible else self._rect
+      self._layouts[self._current_mode].render(content_rect)


two

if self._sidebar.is_visible: self._sidebar.render(self._sidebar_rect)

sshane · 2026-03-26T03:46:55Z

failing static analysis test too, make sure it's all in a final state, i.e. something you're happy with merging now

sshane · 2026-03-26T22:14:59Z

for the generic nav stack touch valid fix, I think we first need to fix the different use of enabled and touch_valid. can do that one later @stefpi

stefpi added 30 commits March 11, 2026 11:49

feat: forward joystick mode sdp offer through athena

63adbdd

feat: add getNotCar dispatcher

16a2fbf

fix: context manager with statement

b346efb

track fix/datachannel-double-counting

8cb630a

feat: (v1) bring back comma body face

d7883cd

feat: raylib face instead of gif, animation harness and barebones pai…

5299b70

…r screen

feat: 10mbps bitrate for connect webrtc livestream

b591fb6

feat: FaceAnimations, new faces and reactivity, battery indicator

b83e436

fix: asleep animation, body_sidebar

6bb0a22

fix: home click and release, battery status

f3c0b98

fix: cleaner transition into driving mode

d2ed390

fix: add CORS headers to webrtcd

a7e54c1

fix: simple CORS header for local direct connection, logging in get_s…

d0db6a1

…tream for better error finding

fix: add back CORS middleware block

e2df6b4

fix: inquisitive and responsive turning faces

e6bb701

fix: swap right and left move animations

2e41637

feat: add support for all 3 cameras

0694182

feat: optional wait for joystick bodyview, wink animation, status in …

8e15567

…webrtc negotiation for better error handling

clean: remove old bodyteleop

f65e11d

fix: change notCar dispatcher to use persistent car params

6e589eb

fix: proper exceptions when webrtcd is not running

29eba8b

fix: check resp.ok from webrtcd

93de8e1

fix: remove wink all the time

dd13a9b

fix: remove redudant and breaking body active check

02cc1c9

fix: remove sdp codec overwite and only accept H264 requests

4f44844

feat: body layout refactoring and adding mici layout

19e459c

fix: body face change on ignition

8a5df0a

fix: make pairing button on MICI bigger

819da5a

feat: ignore unnecessary processe in body so that they don't complain…

ebb6ea0

… about being starved during cereal bridge + webrtc connection

fix: reverse direction of input on body injection test

0ea0244

stefpi added 3 commits March 25, 2026 16:57

bump teleoprtc commit

bb998d7

update git modules

f3396e5

fix ci CP is none error

51715da

stefpi added 2 commits March 25, 2026 18:26

cleanup

4febc0d

cleanups and small refactors

4413ae6

stefpi force-pushed the good-remote-body branch from 61f7f7c to 4413ae6 Compare March 26, 2026 02:00

fix is_body call

fc9ec7e

stefpi marked this pull request as ready for review March 26, 2026 02:08

stefpi requested a review from sshane March 26, 2026 02:08

sshane requested changes Mar 26, 2026

View reviewed changes

stefpi added 2 commits March 26, 2026 09:47

bump commit opendbc_repo

6baf5bd

implement shane's comments

3eb44a4

stefpi force-pushed the good-remote-body branch from f8a23d3 to 71a2950 Compare March 26, 2026 22:27

fix button touch valid

23e38f7

stefpi force-pushed the good-remote-body branch from 71a2950 to 23e38f7 Compare March 26, 2026 22:30

stefpi mentioned this pull request Mar 27, 2026

remote web joystick in connect commaai/connect#584

Draft

stefpi added 5 commits March 27, 2026 00:08

replace IP with env variable for https URL

3bab051

connect qr points to body url

8eb532d

fix bug and remove unused code body_pairing

42c2855

show connect host name properly for hacakthon

af4936c

fix connect button on tici visibility

a234f75

stefpi marked this pull request as draft March 27, 2026 23:30

stefpi added 5 commits March 27, 2026 16:31

hide tici pairing screen on connection

95f9c60

tici move connect/disconnect event reaction

625dd5b

fix pairing button on tici

4271e90

Merge branch 'master' into good-remote-body

58ae7db

bump commits

7a3f526

Conversation

stefpi commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

changes

layouts

webrtcd

soundd

athena

debug

process manager

other

Related PRs that this PR depends on:

Uh oh!

github-actions bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

UI Preview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sshane commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sshane commented Mar 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stefpi commented Mar 25, 2026 •

edited

Loading

github-actions bot commented Mar 26, 2026 •

edited

Loading

sshane commented Mar 26, 2026 •

edited

Loading