Trajectory stopped mid motion

[simonschmeisser]

We sometimes observe trajectories being stopped (softly) in the middle of execution. We have not found any error messages or other reasons for this and have a hard time debugging. We then re-send a new and shortened trajectory containing the remaining way points and it succeeds.

We send trajectories that can be quite sparse (eg 90° motion) at times and dense when simulating a cartesian segment. Could that be an issue, should we interpolate more?

Where does the printf in MotoPlus go to?

Any ideas and hints what to look out for?

[gavanderhoorn]

(for now I've converted this to a discussion. If it turns out this is something that needs to be fixed, we'll create a more detailed issue)

printf(..) goes to stdout, which is either the console (if you have a monitor attached -- you'd need a special cable for that), or to an active telnet session.

The M+ SDK manual explains how to enable telnet (you need to update a parameter and reboot the controller) and how to log-in. Make sure to log-in before you expect the failure to happen, or you'll be too late (basically: log-in as soon as the controller lets you).

We sometimes observe trajectories being stopped (softly) in the middle of execution. We have not found any error messages or other reasons for this and have a hard time debugging.

there is nothing on the ROS-side? Not even if you start the driver with roslaunch --screen ...? Could you create a wireshark capture including traffic when the problem occurs?

Is the action goal cancelled at that point, or does it just hang?

[simonschmeisser]

Thanks for the info, we'll try to open a telnet session then and see if we get anything.

The driver checks for reaching the goal position in a loop and then at some point the action will time out due to MoveIt's execution threshold

[gavanderhoorn]

@simonschmeisser: have you been able to make any progress?

[cjue]

The issue has not come up anymore, I think we can close this discussion for now.

My interpretation is that the lowered Shock Detection Level set up by the customers led to some unexpected stops which were immediately cleared by calling /robot_enable as soon as our old script saw that remote mode was active but motion was not possible. This had the unintended side effect of clearing the error immediately, leaving it "hidden" in the Alarm History which is not visible at the default Pendant security level.

[EricMarcil]

This might be caused by the FSU interaction. There is a note about it here:
HC-Notes.pdf

FSU Interaction
The FSU (Functional Safety Unit) works independently from the controller, therefore all FSU function will work properly whether MotoROS is used or not. When activated, there is no explicit feedback to ROS. In the case that alarms are generated by the FSU, the ROS side will be notified of the alarm state like any other alarm. The alarm will need to be reset, and motion reinitialized from the current position.
The issue is when a function such as Speed Limit is enabled, any motion sent by ROS at speed higher than the speed limit will be reduced. This will affect the robot motion and the robot will not reach it’s expected end position. The only feedback to ROS is the standard position-monitor stream. There is no explicit feedback that the speed was dynamically reduced. This is a limitation that we are aware of. So
it is not recommended to use FSU speed limitation with ROS. If you do need to use it, be careful to properly monitor feedback position to detect if the robot trajectory is not following the expected commanded trajectory.

I'm not sure what model of robot you are using but this is not limited to the HC robot. Though the HC robot always have a default speed limit for PFL mode, users may also define FSU speed limits for other robot models and may experience the same issue.

[EricMarcil]

Also see Issue #247

[cjue]

Thank you for the pointer to #247, we encountered that problem in another application.

There is no FSU-slowdown in the application described by @simonschmeisser

[cjue]

There is no FSU-slowdown in the application

Let me correct that: there actually was FSU slowdown. #247 was the source of the issues described by @simonschmeisser here.

There is no reason for FSU-slowdown in the application, but there actually was a random 10 second slowdown triggered by "everything safe again" transitions. We really need to play close attention to any possible causes of speed limitation in the future.