Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection is too slow - communication_constraints_violation #174

Open
Minyoung1005 opened this issue Jan 13, 2025 · 1 comment
Open

Connection is too slow - communication_constraints_violation #174

Minyoung1005 opened this issue Jan 13, 2025 · 1 comment

Comments

@Minyoung1005
Copy link

Minyoung1005 commented Jan 13, 2025

Hi,

I installed franka ros2 and libfranka on an asus nuc to control franka panda research 3 robot. However, the connection seems to be slow even using real time kernel and following some advices in https://frankaemika.github.io/docs/troubleshooting.html#motion-stopped-due-to-discontinuities-or-communication-constraints-violation. We can successfully retrieve the robot's state, control the robot via gui, but can't run control scripts because they would suddenly stop due to connection problem. We've tried with or without using a network switch, but both cases raise same connection problems. We eventually want to use a gpu workstation to control the robot, so using a network switch would be necessary. Could you please help me fix this connection issue? If I can't reduce connection time, is there a way to increase the maximum time threshold?

System log is attached as a file.
logs-295341-2320006-2025-01-13T20-36-47.zip

Environment:

  • Ubuntu 22.04 installed on asus nuc 14 pro U7 - intel core ultra U7 155H - 32GB Memory and 1TB SSD
  • 5 port gigabit ethernet switch (TP-Link TL-SG105)
  • Cat8 ethernet cables, lengths are in the range of 3 to 6ft
  • libfranka 0.14.1

Here are some logs for running commands:

  1. connection stats
sudo ping 172.16.0.2 -i 0.001 -D -c 10000 -s 1200

--- 172.16.0.2 ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 10017ms
rtt min/avg/max/mdev = 0.202/0.830/1.076/0.185 ms

  1. communication test occurs communication_constraints_violation
sudo ./examples/communication_test 172.16.0.2

WARNING: This example will move the robot! Please make sure to have the user stop button at hand!
Press Enter to continue...

Finished moving to initial joint configuration.

Starting communication test.
#100 Current success rate: 0.14
#200 Current success rate: 0.20
#300 Current success rate: 0.19
#400 Current success rate: 0.16
#500 Current success rate: 0.16
#600 Current success rate: 0.17
#700 Current success rate: 0.16
#800 Current success rate: 0.15
libfranka: Move command aborted: motion aborted by reflex! ["communication_constraints_violation"]
  1. joint_impedance_example_controller.launch.py stops while running
ros2 launch franka_bringup joint_impedance_example_controller.launch.py arm_id:=fr3 robot_ip:=172.16.0.2

[INFO] [launch]: All log files can be found below /home/clear/.ros/log/2025-01-13-15-28-41-217496-clearnuc-9360
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [joint_state_publisher-1]: process started with pid [9361]
[INFO] [robot_state_publisher-2]: process started with pid [9363]
[INFO] [ros2_control_node-3]: process started with pid [9365]
[INFO] [spawner-4]: process started with pid [9367]
[INFO] [spawner-5]: process started with pid [9369]
[INFO] [franka_gripper_node-6]: process started with pid [9371]
[INFO] [spawner-7]: process started with pid [9373]
[ros2_control_node-3] [WARN] [1736800121.520665321] [controller_manager]: [Deprecated] Passing the robot description parameter directly to the control_manager node is deprecated. Use '~/robot_description' topic from 'robot_state_publisher' instead.
[ros2_control_node-3] [INFO] [1736800121.520890768] [resource_manager]: Loading hardware 'FrankaHardwareInterface' 
[franka_gripper_node-6] [INFO] [1736800121.530114112] [fr3_gripper]: Trying to establish a connection with the gripper
[franka_gripper_node-6] [INFO] [1736800121.531454168] [fr3_gripper]: Connected to gripper
[ros2_control_node-3] [INFO] [1736800121.540168305] [resource_manager]: Initialize hardware 'FrankaHardwareInterface' 
[ros2_control_node-3] [INFO] [1736800121.540212987] [FrankaHardwareInterface]: Connecting to robot at "172.16.0.2" ...
[robot_state_publisher-2] [INFO] [1736800121.544211110] [robot_state_publisher]: got segment base
[robot_state_publisher-2] [INFO] [1736800121.544305369] [robot_state_publisher]: got segment fr3_hand
[robot_state_publisher-2] [INFO] [1736800121.544310114] [robot_state_publisher]: got segment fr3_hand_tcp
[robot_state_publisher-2] [INFO] [1736800121.544312791] [robot_state_publisher]: got segment fr3_leftfinger
[robot_state_publisher-2] [INFO] [1736800121.544315199] [robot_state_publisher]: got segment fr3_link0
[robot_state_publisher-2] [INFO] [1736800121.544317537] [robot_state_publisher]: got segment fr3_link1
[robot_state_publisher-2] [INFO] [1736800121.544319712] [robot_state_publisher]: got segment fr3_link2
[robot_state_publisher-2] [INFO] [1736800121.544321821] [robot_state_publisher]: got segment fr3_link3
[robot_state_publisher-2] [INFO] [1736800121.544323772] [robot_state_publisher]: got segment fr3_link4
[robot_state_publisher-2] [INFO] [1736800121.544325833] [robot_state_publisher]: got segment fr3_link5
[robot_state_publisher-2] [INFO] [1736800121.544327833] [robot_state_publisher]: got segment fr3_link6
[robot_state_publisher-2] [INFO] [1736800121.544329756] [robot_state_publisher]: got segment fr3_link7
[robot_state_publisher-2] [INFO] [1736800121.544331754] [robot_state_publisher]: got segment fr3_link8
[robot_state_publisher-2] [INFO] [1736800121.544333774] [robot_state_publisher]: got segment fr3_rightfinger
[ros2_control_node-3] [INFO] [1736800121.550732022] [FrankaHardwareInterface]: Successfully connected to robot
[ros2_control_node-3] [INFO] [1736800121.554116427] [service_server]: Service started
[ros2_control_node-3] [INFO] [1736800121.662154166] [action_server]: Action server started
[ros2_control_node-3] [INFO] [1736800121.662225629] [resource_manager]: Successful initialization of hardware 'FrankaHardwareInterface'
[ros2_control_node-3] [INFO] [1736800121.662375612] [resource_manager]: 'configure' hardware 'FrankaHardwareInterface' 
[ros2_control_node-3] [INFO] [1736800121.662387609] [resource_manager]: Successful 'configure' of hardware 'FrankaHardwareInterface'
[ros2_control_node-3] [INFO] [1736800121.662406754] [resource_manager]: 'activate' hardware 'FrankaHardwareInterface' 
[ros2_control_node-3] [INFO] [1736800121.662894373] [FrankaHardwareInterface]: Started
[ros2_control_node-3] [INFO] [1736800121.662904533] [resource_manager]: Successful 'activate' of hardware 'FrankaHardwareInterface'
[ros2_control_node-3] [INFO] [1736800121.667584080] [controller_manager]: update rate is 1000 Hz
[ros2_control_node-3] [WARN] [1736800121.667768583] [controller_manager]: Could not enable FIFO RT scheduling policy: with error number <1>(Operation not permitted). See [https://control.ros.org/master/doc/ros2_control/controller_manager/doc/userdoc.html] for details on how to enable realtime scheduling.
[ros2_control_node-3] [INFO] [1736800121.706090170] [controller_manager]: Loading controller 'joint_state_broadcaster'
[joint_state_publisher-1] [INFO] [1736800121.717250794] [joint_state_publisher]: Waiting for robot_description to be published on the robot_description topic...
[spawner-4] [INFO] [1736800121.728051145] [spawner_joint_state_broadcaster]: Loaded joint_state_broadcaster
[ros2_control_node-3] [INFO] [1736800121.728917102] [controller_manager]: Configuring controller 'joint_state_broadcaster'
[ros2_control_node-3] [INFO] [1736800121.728997854] [joint_state_broadcaster]: 'joints' or 'interfaces' parameter is empty. All available state interfaces will be published
[spawner-4] [INFO] [1736800121.734495071] [spawner_joint_state_broadcaster]: Configured and activated joint_state_broadcaster
[ros2_control_node-3] [INFO] [1736800121.774505306] [controller_manager]: Loading controller 'franka_robot_state_broadcaster'
[spawner-5] [INFO] [1736800121.790818270] [spawner_franka_robot_state_broadcaster]: Loaded franka_robot_state_broadcaster
[ros2_control_node-3] [INFO] [1736800121.791579320] [controller_manager]: Configuring controller 'franka_robot_state_broadcaster'
[spawner-5] [INFO] [1736800121.820467374] [spawner_franka_robot_state_broadcaster]: Configured and activated franka_robot_state_broadcaster
[INFO] [spawner-4]: process has finished cleanly [pid 9367]
[ros2_control_node-3] [INFO] [1736800121.901577961] [controller_manager]: Loading controller 'joint_impedance_example_controller'
[spawner-7] [INFO] [1736800121.932934811] [spawner_joint_impedance_example_controller]: Loaded joint_impedance_example_controller
[ros2_control_node-3] [INFO] [1736800121.934790252] [controller_manager]: Configuring controller 'joint_impedance_example_controller'
[ros2_control_node-3] [INFO] [1736800121.937445537] [joint_impedance_example_controller]: Extracted Robot Name: fr3
[spawner-7] [INFO] [1736800121.947118921] [spawner_joint_impedance_example_controller]: Configured and activated joint_impedance_example_controller
[INFO] [spawner-5]: process has finished cleanly [pid 9369]
[INFO] [spawner-7]: process has finished cleanly [pid 9373]
[ros2_control_node-3] terminate called after throwing an instance of 'franka::ControlException'
[ros2_control_node-3]   what():  libfranka: Move command aborted: motion aborted by reflex! ["communication_constraints_violation"]
[ERROR] [ros2_control_node-3]: process has died [pid 9365, exit code -6, cmd '/opt/ros/humble/lib/controller_manager/ros2_control_node --ros-args --params-file /home/clear/franka_ros2_ws/install/franka_bringup/share/franka_bringup/config/controllers.yaml --params-file /tmp/launch_params_idnqalsu --params-file /tmp/launch_params_lzgic1sy --params-file /tmp/launch_params_s3kh663_ -r joint_states:=franka/joint_states'].
[INFO] [launch]: process[ros2_control_node-3] was required: shutting down launched system
[INFO] [franka_gripper_node-6]: sending signal 'SIGINT' to process[franka_gripper_node-6]
[INFO] [robot_state_publisher-2]: sending signal 'SIGINT' to process[robot_state_publisher-2]
[INFO] [joint_state_publisher-1]: sending signal 'SIGINT' to process[joint_state_publisher-1]
[franka_gripper_node-6] [INFO] [1736800124.202270534] [rclcpp]: signal_handler(signum=2)
[robot_state_publisher-2] [INFO] [1736800124.202766154] [rclcpp]: signal_handler(signum=2)
[INFO] [robot_state_publisher-2]: process has finished cleanly [pid 9363]
[INFO] [joint_state_publisher-1]: process has finished cleanly [pid 9361]
[INFO] [franka_gripper_node-6]: process has finished cleanly [pid 9371]
  1. cpu-freq
cpufreq-info

cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.40 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.69 GHz.
analyzing CPU 1:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 1
  CPUs which need to have their frequency coordinated by software: 1
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.70 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.70 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 4.07 GHz.
analyzing CPU 2:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 2
  CPUs which need to have their frequency coordinated by software: 2
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.70 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.70 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 4.01 GHz.
analyzing CPU 3:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 3
  CPUs which need to have their frequency coordinated by software: 3
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.70 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.70 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 4.05 GHz.
analyzing CPU 4:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 4
  CPUs which need to have their frequency coordinated by software: 4
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.70 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.70 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 4.06 GHz.
analyzing CPU 5:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 5
  CPUs which need to have their frequency coordinated by software: 5
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.40 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.28 GHz.
analyzing CPU 6:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 6
  CPUs which need to have their frequency coordinated by software: 6
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.40 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.58 GHz.
analyzing CPU 7:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 7
  CPUs which need to have their frequency coordinated by software: 7
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.40 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.23 GHz.
analyzing CPU 8:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 8
  CPUs which need to have their frequency coordinated by software: 8
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.40 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.51 GHz.
analyzing CPU 9:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 9
  CPUs which need to have their frequency coordinated by software: 9
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.40 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.38 GHz.
analyzing CPU 10:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 10
  CPUs which need to have their frequency coordinated by software: 10
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.40 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.67 GHz.
analyzing CPU 11:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 11
  CPUs which need to have their frequency coordinated by software: 11
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.40 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.62 GHz.
analyzing CPU 12:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 12
  CPUs which need to have their frequency coordinated by software: 12
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 3.80 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 2.55 GHz.
analyzing CPU 13:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 13
  CPUs which need to have their frequency coordinated by software: 13
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 3.80 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 2.49 GHz.
analyzing CPU 14:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 14
  CPUs which need to have their frequency coordinated by software: 14
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 3.80 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 2.49 GHz.
analyzing CPU 15:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 15
  CPUs which need to have their frequency coordinated by software: 15
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 3.80 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 2.74 GHz.
analyzing CPU 16:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 16
  CPUs which need to have their frequency coordinated by software: 16
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 3.80 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 2.52 GHz.
analyzing CPU 17:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 17
  CPUs which need to have their frequency coordinated by software: 17
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 3.80 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 2.48 GHz.
analyzing CPU 18:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 18
  CPUs which need to have their frequency coordinated by software: 18
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 3.80 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 2.70 GHz.
analyzing CPU 19:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 19
  CPUs which need to have their frequency coordinated by software: 19
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 3.80 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 2.39 GHz.
analyzing CPU 20:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 20
  CPUs which need to have their frequency coordinated by software: 20
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 2.50 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 2.50 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 1.60 GHz.
analyzing CPU 21:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 21
  CPUs which need to have their frequency coordinated by software: 21
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 2.50 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 2.50 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 1.60 GHz.

Thanks

@AndreasKuhner
Copy link
Member

Hi @Minyoung1005 ,
I guess the problem is the round-trip time:

--- 172.16.0.2 ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 10017ms
rtt min/avg/max/mdev = 0.202/0.830/1.076/0.185 ms

We need a round-time time of at max 0.3 to 0.4ms -> you are in mean at 0.8ms. So, I think you need to get another network card or network connection from your PC. Maybe you could try to start with a 'simple' notebook running an RT to get it to run but you need to get this round-trip time down by a lot.

Cheers,
Andreas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants