eus_vive

Multi robot teleoperation system with Vive/SpaceNav/Oculus/Tablis Cockpit

Note

This package depends on these branches below:

• For PR2 and Baxter

  • knorth55/rviz_camera_stream@master
  • knorth55/rviz_textured_sphere@add-padding
  • knorth55/vive_ros@use-camera

• For Baxter only

  • knorth55/baxter_interface@baxter-vive-control
  • pazeshun/dynamixel_motor@gripper-v6-devel

• For Dragon only

  • knorth55/jsk_aeriao_robot@drone-euslisp

Tested Environment

Build environment

Ubuntu 20.04 + ROS Noetic

Steam does not work on Melodic or Kinetic now.

• NVidia driver: 525.105
• OpenVR: 1.3.22
• Steam VR: 1.6.10

User interface devices

Vive

• Arm motion tracking
• Controller button interface
• HMD visual interface
• Vibration interface
• Sound interface

Tablis

• Arm motion tracking
• Controller button interface
• HMD visual interface
• Vibration interface
• Sound interface

SpaceNav (3D mouse)

• Arm motion tracking
• Controller button interface

Oculus

• Arm motion tracking
• Controller button interface
• HMD visual interface
• Vibration interface
• Sound interface

Installation

Dependency installation

Install dependencies (for Vive)

sudo apt-get install --reinstall xserver-xorg-video-intel-hwe-18.04 libgl1-mesa-glx libgl1-mesa-dri xserver-xorg-core
sudo dpkg-reconfigure xserver-xorg

Install nvidia-driver (for Vive)

# for melodic, run command below
sudo apt install nvidia-driver-390
# for kinetic, install cuda-9.2 deb (local) manually and run command below.
sudo apt install nvidia-396

Install OpenVR, steam and steamVR (for Vive)

Follow here

Remove libcurl packages in Steam (for Vive)

rm -r ~/.steam/steam/ubuntu12_32/steam-runtime/usr/lib/x86_64-linux-gnu/libcurl* 

Install kodak 4k pro camera and ELP usb camera udev (for Baxter)

sudo cp udev/99-kodak.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger

ROS Workspace build

Build jsk_apc workspace (Only for baxter users)

source /opt/ros/$ROS_DISTRO/setup.bash
mkdir ~/jsk_apc_ws/src -p
cd ~/jsk_apc_ws/src
wstool init . https://raw.githubusercontent.com/start-jsk/jsk_apc/master/fc.rosinstall.${ROS_DISTRO}
wstool up
rosdep install -y -r --from-paths .

cd ~/jsk_apc_ws 
catkin config
catkin build

Build eus_vive workspace

source /opt/ros/$ROS_DISTRO/setup.bash
mkdir ~/vive_ws/src -p
cd ~/vive_ws/src
wstool init . https://raw.githubusercontent.com/knorth55/eus_vive/master/fc.rosinstall

# Only for baxter users
wstool merge https://raw.githubusercontent.com/knorth55/eus_vive/master/baxter.rosinstall
wstool merge https://raw.githubusercontent.com/knorth55/eus_vive/master/baxter.rosinstall.$ROS_DISTRO

# Only for dragon users
wstool merge https://raw.githubusercontent.com/knorth55/eus_vive/master/dragon.rosinstall

wstool up
rosdep install --ignore-src --from-path . -y -r -i

# Only for baxter users
source ~/jsk_apc_ws/devel/setup.bash

cd ~/vive_ws
catkin config
catkin build

How to start

Start procedure

• Connect Vive HMD and Lighthouse and power on the controller.
• Place Lighthouse B in front of you.
• Start one of launch files below.
• If you don't use HMD, press Calibrate button and do arm calibration.
• Press Enable button in GUI to start teleoperation.

Arm calibration (Only for no HMD mode)

The arm calibration is required for no HMD mode in order to measure your position and your arm length.

• Press Calibrate l/rarm button in GUI
• Listen to the voice instruction.
• Stretch your left/right arm and Press Trigger after the first instruction.
• Listen to the voice instruction again.
• Fold your left/right arm and Press Trigger after the second instruction.
• If calibration failed, please try again.

PR2 + Vive

pr2_vive_control.mp4

PR2 + Vive in JSK 73B2 or 610

rossetip
rossetmaster pr1040
# HMD mode
roslaunch eus_vive pr2_vive.launch
# No HMD mode
roslaunch eus_vive pr2_vive.launch head:=false

PR2 + Vive in Gazebo

roslaunch pr2_gazebo pr2_empty_world.launch
roslaunch eus_vive pr2_vive_gazebo.launch

PR2 + Tablis

pr2_tablis_eus_vive_compressed.mp4

PR2 + Tablis in JSK 73B2 or 610

rossetip
rossetmaster pr1040
roslaunch eus_vive pr2_tablis.launch

PR2 + Tablis in Gazebo

Launch Tablis in Choreonoid

roscd eus_vive/scripts/tablis
./start-tablis-sim.sh

roscd eus_vive/scripts/tablis
ipython -i tablis_setup.py
hcf.servoOn()
hcf.hc_svc.startHapticsController()

Launch bridge

roscd eus_vive/scripts/tablis
./start-bridge-sim.sh

Launch PR2 in Gazebo

roslaunch pr2_gazebo pr2_empty_world.launch
roslaunch eus_vive pr2_tablis_gazebo.launch

PR2 + SpaceNav

pr2_spacenav_base_control2.mp4

PR2 + SpaceNav in JSK 73B2

rossetip
rossetmaster pr1040
roslaunch eus_vive pr2_spacenav.launch

PR2 + SpaceNav in Gazebo

roslaunch pr2_gazebo pr2_empty_world.launch
roslaunch eus_vive pr2_spacenav_gazebo.launch

Baxter + Vive

baxter_vive_control.mp4

Baxter + Vive in JSK 73B2

Baxter control PC

rossetip
rossetmaster baxter
roslaunch eus_vive baxter_remote.launch

Vive PC

rossetip
rossetmaster baxter
# No HMD mode
roslaunch eus_vive baxter_vive_remote.launch
# for display
roslaunch eus_vive baxter_display_remote.launch

Baxter + Vive in Gazebo

roslaunch baxter_gazebo baxter_world.launch
roslaunch eus_vive baxter_vive_gazebo.launch

Baxter + Tablis

baxter_tablis_eus_vive_compressed.mp4

Baxter + Tablis in JSK 73B2 or 610

rossetip
rossetmaster baxter
roslaunch eus_vive baxter_tablis.launch

Baxter + Tablis in Gazebo

Launch Tablis in Choreonoid

roscd eus_vive/scripts/tablis
./start-tablis-sim.sh

roscd eus_vive/scripts/tablis
ipython -i tablis_setup.py
hcf.servoOn()
hcf.hc_svc.startHapticsController()

Launch bridge

roscd eus_vive/scripts/tablis
./start-bridge-sim.sh

Launch Baxter in Choreonoid

roslaunch baxter_gazebo baxter_world.launch
roslaunch eus_vive baxter_tablis_gazebo.launch

Baxter + SpaceNav

Baxter + SpaceNav in JSK 73B2

rossetip
rossetmaster baxter
roslaunch eus_vive baxter_spacenav.launch

Baxter + SpaceNav in Gazebo

roslaunch baxter_gazebo baxter_world.launch
roslaunch eus_vive baxter_spacenav_gazebo.launch

Baxter + MoveIt!

Real Robot in JSK 73B2

rossetip
rossetmaster baxter
roslaunch eus_vive baxter_moveit.launch

Dragon + SpaceNav

dragon_eus_vive_spacenav.mp4

Dragon + SpaceNav in Gazebo

roslaunch dragon bringup.launch simulation:=true real_machine:=false headless:=false
roslaunch eus_vive dragon_spacenav_gazebo.launch

JAXON + SpaceNav

jaxon_eus_vive_control3.mp4

JAXON + SpaceNav in Choreonoid

Launch JAXON in Choreonoid

roscd eus_vive/scripts/jaxon
./start-jaxon_with_rhp3hand-sim.sh

roscd eus_vive/scripts/jaxon
ipython -i jaxon_with_rhp3hand_setup.py
hcf.ast_svc.startAutoBalancer()
hcf.ast_svc.startStabilizer()
hcf.ast_svc.startWholeBodyMasterSlave()

Launch eus_vive for SpaceNav

roslaunch eus_vive jaxon_spacenav_choreonoid.launch

JAXON + Tablis

jaxon_eus_vive_tablis.mp4

JAXON + Tablis in Choreonoid

Launch Tablis in Choreonoid

roscd eus_vive/scripts/tablis
./start-tablis-sim.sh

roscd eus_vive/scripts/tablis
ipython -i tablis_setup.py
hcf.servoOn()
hcf.hc_svc.startHapticsController()

Launch JAXON in Choreonoid

roscd eus_vive/scripts/jaxon
./start-jaxon_with_rhp3hand-sim.sh

roscd eus_vive/scripts/jaxon
ipython -i jaxon_with_rhp3hand_setup.py
hcf.ast_svc.startAutoBalancer()
hcf.ast_svc.startStabilizer()
hcf.ast_svc.startWholeBodyMasterSlave()

Launch bridge and wbms core

roscd eus_vive/scripts/jaxon
./start-jaxon-eus-vive-sim.sh

Launch eus_vive for Tablis

roslaunch eus_vive jaxon_tablis_choreonoid.launch

Demo & Experiments

Miraikan Demo 2019/08/23-24

rossetip
rossetmaster baxter
roslaunch eus_vive baxter_vive_mirror.launch

Miraikan Demo 2020/09/11-13

Robot control PC (Robot side)

rossetip
rossetmaster baxter
roslaunch eus_vive baxter_remote.launch

Vive control PC (Pilot side)

rossetip
rossetmaster baxter
roslaunch eus_vive baxter_vive_remote.launch

Visualization, display and feedback PC (Pilot side)

rossetip
rossetmaster baxter
roslaunch eus_vive baxter_display_remote.launch

How to use Vive controller

PR2

Button	Usage
1 / Menu	Control toggle: base/arm (Default: base)
3 / Stream Menu	Steam Menu
8 / Grip	Not used

Arm mode

You can enable arm mode of right and left arm separately.

Command	Usage
7 / Trigger	Gripper toggle: open/close in Toggle grasp mode (Default: open)
7 / Trigger	Gripper toggle: open only when trigger is pressed in Hold grasp mode
Controller pose	robot end effector's pose

Base mode

Base mode is enabled when both arms are disabled in Arm mode.

Command	Usage
2 / Trackpad	Torso control: right: down / left: up
2 / Trackpad + 7 / Trigger (right)	Safe base control: right: x, y / left: w
2 / Trackpad + 7 / Trigger (right + left)	Unsafe base control: right: x, y / left: w

Baxter

Button	Usage
2 / Trackpad	Control toggle: stop / arm (Default: stop)
3 / Stream Menu	Steam Menu
8 / Grip	Not used

Arm mode

You can enable arm mode of right and left arm separately.

Command	Usage
7 / Trigger	Gripper toggle: open/close (Default: open)
Controller pose	robot end effector's pose

How to use GUI control

Teleoperation control buttons

Button	Usage
Reset	Diable and reset robot to initial posture
Enable	Enable robot
Disable	Disable robot
Calibrate larm/rarm	Calibrate with left/right arm

Arm & Gripper control buttons

Button	Usage
Reset larm/rarm	Reset only left/right arm
Enable larm/rarm	Enable only left/right arm
Disable larm/rarm	Disable only left/right arm
Start grasp lgripper/rgripper	Start grasp only left/right gripper
Stop grasp lgripper/rgripper	Stop grasp only left/right gripper
Toggle grasp mode	Change to Toggle grasp mode
Hold grasp mode	Change to Hold grasp mode

Tips

Baxter Network Configuration

Open Field Service Menu and change network configuration

VPNC Install

sudo apt install libgcrypt20-dev libgnutls28-dev
git clone https://github.com/streambinder/vpnc.git
cd vpnc
git checkout 1cf24ed6aa4a04b4b01cc9ebfacbad723eed04f5
make
sudo make install
cd ..
git clone git://git.infradead.org/users/dwmw2/vpnc-scripts.git
cd vpnc-scripts
sudo cp vpnc-script /etc/vpnc

VPNC Command for Fortigate

sudo vpnc --local-port 0 --gateway <gateway> --id ipsecvpn --username <username> --pfs dh5 --dh dh5 --auth-mode psk --no-detach --vendor fortigate --dh dh5

Demo Video

PR2 Fridge demo

• Trial 1
• Trial 2
• Trial 3

Baxter APC demo

• Trial 1
• Trial 2
• Trial 3
• Trial 4
• Trial 5