Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trajectory Execution Failed Sliently due to "Sending data through socket failed." #556

Closed
shuobh opened this issue Jul 5, 2022 · 8 comments · May be fixed by #615
Closed

Trajectory Execution Failed Sliently due to "Sending data through socket failed." #556

shuobh opened this issue Jul 5, 2022 · 8 comments · May be fixed by #615
Labels
close if no update question Further information is requested

Comments

@shuobh
Copy link

shuobh commented Jul 5, 2022

Summary

Introduction to the issue

When executing multiple trajectories sequentially using remote control, some trajectory may fail silently due to "Sending data through socket failed.". For example, when executing A->B->C. If B failed, then C will directly be executed after A which out any error throwing.

Versions

ROS Driver version: ROS Noetic
Affected Robot Software Version(s): 5.11.1
Affected Robot Hardware Version(s): UR3e

Impact

If a trajectory is skipped, then it might cause the robot to blindly move from the end of the previous trajectory to the beginning of the next trajectory which my cause a big hazard.

Project status at point of discovered

When did you first observe the issue?
This happens by chance with relatively low probability.

Workaround Suggestion

By searching in this repo, the cause to this failure of "Sending data through socket failed." might be due to that we are not using a real time kernal. We will be testing it following https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/real_time.md. However, if this case happens, the driver should either retry, or throw an error instead of fail silently.
@shuobh
Copy link
Author

shuobh commented Jul 14, 2022

Ran into another trajectory execution issue:

Jul 14 14:24:44 nuc-19-robot-14 bluehill-start[2271640]: [ WARN] [1657833884.788138173]: Dropping all 1 trajectory point(s), as they occur before the current time.
Jul 14 14:24:44 nuc-19-robot-14 bluehill-start[2271640]: Last point is 1.555738s in the past.

@fmauch
Copy link
Collaborator

fmauch commented Jul 26, 2022

I cannot quite follow what you are trying to accomplish. Could you please elaborate what (and with which timing) you are sending to the driver's interfaces?

Sending data through socket failed should not happen in a normal usage, so this would be something to further inspect.

And the comment from your second post comes from the joint trajectory controller directly and means that the trajectory sent to the controller is illegal as (as the output states) the last point in the trajectory is in the past. Timestamps of waypoints must be increasing in time.

@fmauch fmauch added the question Further information is requested label Jul 26, 2022
@shuobh
Copy link
Author

shuobh commented Aug 5, 2022

We've tried to install a realtime kernel but it failed to compile. However, by upgrading to low-latency kernel, this problem seems to be resolved as well. @fmauch Do you have any insight on the difference between low latency kernel and realtime kernel and how it might impact the driver?

@github-actions
Copy link

This issue has not been updated for a long time. If no further updates are added, this will be closed automatically. Comment on the issue to prevent automatic closing.

@fmauch
Copy link
Collaborator

fmauch commented Jan 24, 2023

We are currently looking deeper into this and hope to provide some more insight soon.

@fmauch
Copy link
Collaborator

fmauch commented Jan 30, 2023

The deeper look as mentioned above is now taking shape in #615.

@fmauch fmauch linked a pull request Jan 30, 2023 that will close this issue
Real time benchmarking / reasoning #615
Draft
@github-actions
Copy link

This issue has not been updated for a long time. If no further updates are added, this will be closed automatically. Comment on the issue to prevent automatic closing.

@github-actions
Copy link

This issue has been closed due to inactivity. Feel free to comment or reopen if this is still relevant.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
close if no update question Further information is requested
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Real time benchmarking / reasoning fmauch/Universal_Robots_ROS_Driver
2 participants
@fmauch @shuobh