Skip to content

Added newton based teleop scripts in the newton-teleop folder #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RijulChaddha9504 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Added newton based teleop scripts in the newton-teleop folder #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.sh text eol=lf
Binary file added GlitchMove4.mp4
View file
Open in desktop
Binary file not shown.
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Most modules expose `<name>_dev` under `MODE_OF_OPERATION=develop`. **`interfaci
| `behaviour` | `joint_command`, `voxel_grid` |
| `simulation_isaac` | **Isaac Lab 2.3.2** — SO101 IL, HumanoidRL, Quest teleop |
| `simulation_mj` | MuJoCo / mjlab RL (`mjlabs` service) |
| `simulation_mj.cpu` | MuJoCo / mjlab RL CPU-only (`mjlabs` service) |

**Isaac Lab sim (recommended):** see [docker/simulation/isaac_lab/QUICKSTART.md](docker/simulation/isaac_lab/QUICKSTART.md).

Expand Down
177 changes: 177 additions & 0 deletions autonomy/simulation/Humanoid_Wato/wato_bimanual_arm/urdf/sektion_cabinet.urdf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
<!-- Place for location to be decided -->
<robot name="sektion_cabinet">
<material name="cabinet_wood">
<color rgba="0.9 0.85 0.8 1.0"/>
</material>
<material name="cabinet_drawer_white">
<color rgba="0.95 0.95 0.95 1.0"/>
</material>
<material name="cabinet_metal">
<color rgba="0.4 0.4 0.4 1.0"/>
</material>

<!-- Cabinet base link -->
<link name="cabinet_base_link">
<inertial>
<mass value="50.0"/>
<inertia ixx="1.0" ixy="0.0" ixz="0.0" iyy="1.0" iyz="0.0" izz="1.0"/>
</inertial>
</link>

<!-- Cabinet main body structure -->
<link name="cabinet_body">
<visual>
<geometry>
<!-- A cabinet structure: size 0.5m depth (X), 0.6m width (Y), 0.8m height (Z) -->
<box size="0.5 0.6 0.8"/>
</geometry>
<origin xyz="0.0 0.0 0.4" rpy="0 0 0"/>
<material name="cabinet_wood"/>
</visual>
<collision>
<geometry>
<box size="0.5 0.6 0.8"/>
</geometry>
<origin xyz="0.0 0.0 0.4" rpy="0 0 0"/>
</collision>
<inertial>
<mass value="30.0"/>
<origin xyz="0.0 0.0 0.4"/>
<inertia ixx="1.0" ixy="0.0" ixz="0.0" iyy="1.0" iyz="0.0" izz="1.0"/>
</inertial>
</link>

<joint name="cabinet_body_fixed" type="fixed">
<parent link="cabinet_base_link"/>
<child link="cabinet_body"/>
<origin xyz="0 0 0" rpy="0 0 0"/>
</joint>

<!-- Top Drawer -->
<link name="drawer_top">
<visual>
<geometry>
<!-- Top Drawer: depth 0.45m, width 0.54m, height 0.3m -->
<box size="0.45 0.54 0.3"/>
</geometry>
<origin xyz="0.0 0.0 0.0" rpy="0 0 0"/>
<material name="cabinet_drawer_white"/>
</visual>
<collision>
<geometry>
<box size="0.45 0.54 0.3"/>
</geometry>
<origin xyz="0.0 0.0 0.0" rpy="0 0 0"/>
</collision>
<inertial>
<mass value="2.0"/>
<origin xyz="0.0 0.0 0.0"/>
<inertia ixx="0.05" ixy="0.0" ixz="0.0" iyy="0.05" iyz="0.0" izz="0.05"/>
</inertial>
</link>

<!-- Top Drawer Joint: pulling toward the robot opens drawer (pos goes from 0 to -0.35) -->
<joint name="drawer_top_joint" type="prismatic">
<parent link="cabinet_body"/>
<child link="drawer_top"/>
<!-- Positioned in the upper half of the cabinet -->
<origin xyz="-0.025 0.0 0.55" rpy="0 0 0"/>
<axis xyz="1.0 0.0 0.0"/>
<limit lower="-0.35" upper="0.0" effort="100.0" velocity="2.0"/>
<dynamics damping="3.0" friction="0.5"/>
</joint>

<!-- Top Drawer Handle -->
<link name="drawer_top_handle">
<visual>
<geometry>
<!-- Horizontal handle bar: depth 0.02m, width 0.15m, height 0.02m -->
<box size="0.02 0.15 0.02"/>
</geometry>
<origin xyz="0.0 0.0 0.0" rpy="0 0 0"/>
<material name="cabinet_metal"/>
</visual>
<collision>
<geometry>
<box size="0.02 0.15 0.02"/>
</geometry>
<origin xyz="0.0 0.0 0.0" rpy="0 0 0"/>
</collision>
<inertial>
<mass value="0.1"/>
<origin xyz="0.0 0.0 0.0"/>
<inertia ixx="0.0001" ixy="0.0" ixz="0.0" iyy="0.0001" iyz="0.0" izz="0.0001"/>
</inertial>
</link>

<joint name="drawer_top_handle_joint" type="fixed">
<parent link="drawer_top"/>
<child link="drawer_top_handle"/>
<!-- Mounted on the front face of the top drawer -->
<origin xyz="0.235 0.0 0.0" rpy="0 0 0"/>
</joint>


<!-- Bottom Drawer -->
<link name="drawer_bottom">
<visual>
<geometry>
<box size="0.45 0.54 0.3"/>
</geometry>
<origin xyz="0.0 0.0 0.0" rpy="0 0 0"/>
<material name="cabinet_drawer_white"/>
</visual>
<collision>
<geometry>
<box size="0.45 0.54 0.3"/>
</geometry>
<origin xyz="0.0 0.0 0.0" rpy="0 0 0"/>
</collision>
<inertial>
<mass value="2.0"/>
<origin xyz="0.0 0.0 0.0"/>
<inertia ixx="0.05" ixy="0.0" ixz="0.0" iyy="0.05" iyz="0.0" izz="0.05"/>
</inertial>
</link>

<!-- Bottom Drawer Joint: pulling toward the robot opens drawer (pos goes from 0 to -0.35) -->
<joint name="drawer_bottom_joint" type="prismatic">
<parent link="cabinet_body"/>
<child link="drawer_bottom"/>
<!-- Positioned in the lower half of the cabinet -->
<origin xyz="-0.025 0.0 0.2" rpy="0 0 0"/>
<axis xyz="1.0 0.0 0.0"/>
<limit lower="-0.35" upper="0.0" effort="100.0" velocity="2.0"/>
<dynamics damping="3.0" friction="0.5"/>
</joint>

<!-- Bottom Drawer Handle -->
<link name="drawer_bottom_handle">
<visual>
<geometry>
<box size="0.02 0.15 0.02"/>
</geometry>
<origin xyz="0.0 0.0 0.0" rpy="0 0 0"/>
<material name="cabinet_metal"/>
</visual>
<collision>
<geometry>
<box size="0.02 0.15 0.02"/>
</geometry>
<origin xyz="0.0 0.0 0.0" rpy="0 0 0"/>
</collision>
<inertial>
<mass value="0.1"/>
<origin xyz="0.0 0.0 0.0"/>
<inertia ixx="0.0001" ixy="0.0" ixz="0.0" iyy="0.0001" iyz="0.0" izz="0.0001"/>
</inertial>
</link>

<joint name="drawer_bottom_handle_joint" type="fixed">
<parent link="drawer_bottom"/>
<child link="drawer_bottom_handle"/>
<!-- Mounted on the front face of the bottom drawer -->
<origin xyz="0.235 0.0 0.0" rpy="0 0 0"/>
</joint>

</robot>
2 changes: 2 additions & 0 deletions autonomy/simulation/MUJOCO_LOG.TXT
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
WARNING Sun Jul 12 02:26:36 2026: Nan, Inf or huge value in QACC at DOF 15. The simulation is unstable. Time = 119.5160.

72 changes: 72 additions & 0 deletions autonomy/simulation/Teleop/mjlabs_keyboard_based_teleoperation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Keyboard-Based Teleoperation for WATO Humanoid Bimanual Arm (MuJoCo / Warp)

This directory contains scripts for keyboard-based teleoperation of the bimanual WATO humanoid arm using either MuJoCo or NVIDIA Warp/Newton.

## Prerequisites & Docker Environment Setup

1. **Configure Docker environment:**
Create or edit the local config file `watod-config.local.sh` in the root of the repository:
```bash
# Use "simulation_mj" if running with an NVIDIA GPU, or "simulation_mj.cpu" for CPU-only systems
ACTIVE_MODULES="simulation_mj"
MODE_OF_OPERATION="develop"
```

2. **Build and start the services:**
Run the following commands in the root of the humanoid workspace directory:
```bash
./watod down
./watod build
./watod up -d
```

3. **Attach to the `mjlabs` service:**
Start an interactive bash shell in the running service container:
```bash
./watod -t mjlabs
```

---

## Running the Simulation

### Option A: NVIDIA Warp + Newton Physics Simulation (Recommended)
This uses NVIDIA Warp on CPU/GPU to run an XPBD (Extended Position-Based Dynamics) rigid body solver, using a Viser-based standalone browser visualizer.

1. **Install Python package dependencies** (inside the container shell):
```bash
pip install warp-lang newton viser trimesh
```

2. **Launch the teleoperation script:**
```bash
python3 /root/ament_ws/src/simulation/Teleop/mjlabs_keyboard_based_teleoperation/newton_keyboard_teleop.py
```

3. **Access the visualizer:**
Open [http://localhost:8080](http://localhost:8080) (or the port shown in your terminal output) in any web browser.

#### Controls
* **I / K** : Move target marker along X-axis (+/-)
* **J / L** : Move target marker along Y-axis (+/-)
* **U / O** : Move target marker along Z-axis (+/-)
* **C / D** : Rotate right wrist joint roll (+/-) (limit: $[-180^\circ, 180^\circ]$)
* **G** : Toggle gripper fingers (Open/Closed)
* **R** : Reset right arm to defaults

---

### Option B: Pure MuJoCo Simulation (Fallback)
This runs the original, pure MuJoCo simulation using the `mjviser` viewer.

1. **Launch the teleoperation script** (inside the container shell):
```bash
python3 /root/ament_ws/src/simulation/Teleop/mjlabs_keyboard_based_teleoperation/mjlabs_keyboard_teleop.py
```

#### Controls
* **I / K** : Move target marker along X-axis (+/-)
* **J / L** : Move target marker along Y-axis (+/-)
* **U / O** : Move target marker along Z-axis (+/-)
* **G** : Toggle gripper fingers (Open/Closed)
* **R** : Reset left arm to defaults
Loading
Loading