diff --git a/_pages/sts-il.md b/_pages/sts-il.md index a9b0655..b6ecc79 100644 --- a/_pages/sts-il.md +++ b/_pages/sts-il.md @@ -9,6 +9,7 @@ usemathjax: true --- # Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor +{: .no_toc } [ arXiv ](https://arxiv.org/abs/2311.01248){: .btn .btn-blue } [ Github](https://github.com/SAIC-MONTREAL/tactile-il){: .btn .btn-green } @@ -16,8 +17,9 @@ usemathjax: true ### Trevor Ablett1, Oliver Limoyo1, Adam Sigal2, Affan Jilani3, Jonathan Kelly1 , Kaleem Siddiqi3, Francois Hogan2, Gregory Dudek2,3 +{: .no_toc } -
+
1University of Toronto, 2Samsung AI Centre, Montreal, QC, Canada, 3McGill University @@ -26,12 +28,14 @@ usemathjax: true \**work completed while Trevor Ablett, Oliver Limoyo, and Affan Jilani were on internship at Samsung AI Centre, Montreal* #### Submitted to IEEE Transactions on Robotics (T-RO): Special Section on Tactile Robotics +{: .no_toc } --- ## Summary +{: .no_toc } -{::nomarkdown} +{::nomarkdown}
A summary of our system for multimodal and force-matched imitation learning.
@@ -39,8 +43,8 @@ usemathjax: true {:/} -{::nomarkdown} -
+{::nomarkdown} +
{:/} -## Motivation +## Table of contents +{: .no_toc } + -When we open doors, our fingers roll, slip, use normal and shear forces, respond to dense tactile feedback, move quickly, and can manipulate small, challenging knobs. In this paper, we combined tactile sensing with imitation learning to attempt to allow robots to achieve some of these same desirable qualities. +1. TOC +{:toc} -Kinesthetic teaching is a popular approach to collecting expert robotic demonstrations of contact-rich tasks for imitation learning (IL), but it typically only measures motion, ignoring the force placed on the environment by the robot. Furthermore, contact-rich tasks require accurate sensing of both reaching and touching, which can be difficult to provide with conventional sensing modalities. We address these challenges with a see-through-your-skin (STS) visuotactile sensor, using the sensor both (i) as a measurement tool to improve kinesthetic teaching, and (ii) as a policy input in contact-rich door manipulation tasks. +## Motivation + +The conventional approach to manipulating articulated objects such as doors and drawers with robots relies on a firm, stable grasp of the handle followed by a large arm motion to complete the opening/closing task. +
+
+ +
Grasp and rotate.
+
+
+ +
Press and pull.
+
+
+ +
A tiny human combines both.
+
+
+In contrast, humans are capable of opening and closing doors with minimal arm motions, by relaxing their grasp on the handle and allowing for relative motion between their fingers and the handle. +This work aims to learn robot policies for door opening that are more in line with human manipulation, by leveraging high-resolution visual and tactile feedback to control the contact interactions between the robot end-effector and the handle. ### Why Kinesthetic Teaching? -{::nomarkdown} +{::nomarkdown}
{:/} -Teleoperation is a viable strategy for collecting demonstrations, but it requires a **proxy** for contact feedback. In the teleoperation example above, we used the vibration motor in the VR controller and the force-torque sensor to provide coarse feedback, but demonstrations were still difficult to collect. Even with considerable practice, we would still often cause the robot to enter into a protective stopped state for generating excessive force against the environment. +Teleoperation is a viable strategy for collecting demonstrations, but it requires a **proxy** for contact feedback. In the teleoperation example above, we used the vibration motor in the VR controller and the force-torque sensor to provide coarse feedback, but demonstrations were still difficult to collect. Even with considerable practice, we would still often cause the robot to enter into a protective stopped state for generating excessive force against the environment.
-{::nomarkdown} +{::nomarkdown}
- - -
Left: Kinesthetic teaching.   Right: Replays of kinesthetic teaching without force matching.
+
+ +
Kinesthetic Teaching.
+
+
+ +
Replays of kinesthetic teaching without force matching.
+
{:/} @@ -101,8 +134,14 @@ Kinesthetic teaching has a significant downside, however: the observation and ac ### See-through-Your-Skin (STS) Tactile Sensor
- -
Left: 2022 version of STS sensor1.   Right: Version from this work, in visual mode (LEDs off) and tactile mode (LEDs on).
+
+ +
2022 version of STS sensor

[^1]

.
+
+
+ +
Version from this work, in visual mode (LEDs off) and tactile mode (LEDs on).
+
An STS sensor[^1] is a visuotactile sensor, comparable to other gel-based tactile sensors[^2], that can be switched between visual and tactile modes by leveraging a semi-transparent surface and controllable lighting, allowing for both pre-contact visual sensing and during-contact tactile sensing with a single sensor. In this work, we use the sensor in tactile mode to record a signal linearly related to force during demonstrations, and show its value in both visual and tactile mode as an input to learned policies through extensive experiments. @@ -110,12 +149,17 @@ An STS sensor[^1] is a visuotactile sensor, comparable to other gel-based tactil ### Tactile Force Matching
- - -
Left: Our force-matched trajectories are generated using a spring relationship between desired poses and recorded poses, matching the functionality of our Cartesian impedance controller. Right: Kinesthetic teaching while reading the signal linearly related to force with an STS sensor.
+
+ +
Our force-matched trajectories are generated using a spring relationship between desired poses and recorded poses.
+
+
+ +
Kinesthetic teaching while reading the signal linearly related to force with an STS sensor.
+
Our force-matched poses were generated by measuring a signal linearly related to force with an STS sensor, which was then used to modify the desired poses before they were input back into our Cartesian impedance controller. For more details, see our paper. @@ -123,24 +167,29 @@ Our force-matched poses were generated by measuring a signal linearly related to ### Learned STS Mode Switching
- - -
Left: Our learned policy includes sensor mode as an output, in addition to robot motion. Right: Mode switch labels are provided during the force matched replay by the demonstrator, who pushes a key when the robot makes contact with the environment.
+
+ +
Our learned policy includes sensor mode as an output, in addition to robot motion.
+
+
+ +
Mode switch labels are provided during the force matched replay by the demonstrator, who pushes a key when the robot makes contact with the environment.
+
To further leverage an STS sensor for imitation learning, we add mode switching as a policy output, allowing the policy to learn the appropriate moment to switch an STS from its visual to its tactile mode. ## Experiments -To verify the efficacy of force matching, learned mode switching, and tactile sensing in general, we study multiple observation configurations to compare and contrast the value of visual and tactile data from an STS with visual data from a wrist-mounted eye-in-hand camera. +To verify the efficacy of force matching, learned mode switching, and tactile sensing in general, we study multiple observation configurations to compare and contrast the value of visual and tactile data from an STS with visual data from a wrist-mounted eye-in-hand camera. ### Experimental Tasks -{::nomarkdown} -
+{::nomarkdown} +