Mucojo simulated Franka Gripper have weird open/close values

[Harimus]

Hi I just finished assembling GELLO and was running the sim_franka version of the environment to check for everything working properly. After some re-calibration It worked fine.

However I noticed that Franka gripper value in the simulator were mapped a bit weirdly. When checking the panda/finger_joint values it fluctuates between [8e-06, 0.000153] #gripper: (open, close) and the panda/actuator for the fingers, between [0.0541, 0.994]. It seems like the actuator values are fine but not the gripper joint values? The gripper is constantly closed.

I double-checked on Dynamixel-wizard the Gripper position value, to make sure it wasn't mis-match of calibration and the calibration value I got (191, 149) seemed to correspond pretty well with what the dynamixel-wizard was showing.

Also:
on Line 124

gello_software/experiments/run_env.py

Line 124 in e6a660d

agent = GelloAgent(port=gello_port, start_joints=args.start_joints)

I think it is supposed to be start_joints=reset_joints and in line 123 reset_joints = np.array(args.start_joints) (otherwise the code errors when start_joints are handed through command-line args, since it is given as tuple and not np.ndarray)

[johnny-wang16]

Hi there,
Im facing the same problem here. The gripper is always closed for some reason. are you able to solve it?

[Harimus]

For the simulator, No. I might look in to it in the future but I only used the sim to double-check the joints.

In the real robot gripper worked just fine; Checking polymetis (library that wrap around libfranka API) the values that GELLO output seems to match what that expect. (normalized to [0,1]) My guess is that Mujuoco, which (probably) uses urdf file for specifying the robot, expect the values to be meter or force (newton) and this not matching what GELLO gives to it.

For the real robot, I might also add that your gripper might freeze when it grasp objects. (If you're using standard gripper of Franka) I haven't fixed it but I'm pretty sure this is GELLO using the wrong API call to polymetis/libfranka; I might up a PR if that's the case

[johnny-wang16]

Got it. Thanks for the info! I'll try to deploy on a real franka and see if I have the same problem.

[johnny-wang16]

Hi there, may I ask how you get gello to control a real Franka ? In my setup, I have an intel NUC connected to Franka with polymetis installed. The gello code is sitting on another ubuntu laptop, which is connected to the NUC via a network switch.

To run gello, I installed polymetis in the same conda envrionement as gello on the laptop. I also changed the IP specified in launch_nodes.py . But somehow its giving me connection errors even though Im able to ping the robot.

[Harimus]

I have similar setup. The process of starting it (for me) is:

• Activate Franka (yellow light turns blue on robot)
• launch launch_robot.py robot_client=franka_hardware on NUC (polymetis launch file for franka arm) Edit: this code asks for sudo access and you need to provide sudo password
• launch launch_gripper.py gripper=franka_hand on NUC (polymetis launch file for franka default arm)
• on my ubuntu computer connected to GELLO hardware, first do sudo chmod 666 /dev/path/to/usb to allow GELLO code to access the USB (hardware)
• on my ubuntu computer connected to GELLO hardware, python3 experiments/launch_nodes.py --robot panda --robot_ip="your.ip.address"
• on my ubuntu computer connected to GELLO hardware python3 experiments/run_env.py --agent=gello. This code actually often fails, and I need to rerun it. The reason for failing is
  • The initial position of GELLO hardware is too faraway from franka arm (polymetis) home position, and run_env.py check for difference in position and crashes if it's too far away.
  • everytime I reconnect the GELLO Hardware with USB, some of the calibration value seems to have changed. The above "check for difference in position" triggers with an difference-value and I usually have to modify (manually) the calibration value so it reflects the real situation.

When I bump/get too far outside the workspace defined in polymetis, polymetis doesn't crash but above GELLO two launch processes crashes, so I need to restart them. launch_nodes.py always first goes back to homing position, so be aware of that when you restart those processes.

I hope that helps

Edit: I also remember that the GELLO launch_nodes does crash if the gripper server on polymetis is not properly running, it was slightly confusing because looking at the code, it is supposed to run without it.

Edit2: For the default gripper, there seems to be an error in the GELLO software code that I needed to modify. Namely this line in gello/robots/panda.py

gello_software/gello/robots/panda.py

Line 21 in e6a660d

ip_address="localhost",

This default value localhost is supposed to be robot_ip. The localhost will probably crash the launch_node by not being able to connect to the gripper server.

[Harimus]

sorry added two edits later

[johnny-wang16]

Thanks! Following your instructions works! However, I do have the same problem: to get Gello running, I have to manually match the joint positions of Gello with those of Franka's which is kind of tedious.

[Harimus]

yep it is indeed.... I assume it is an error somewhere in-between the DynamixelSDK and the calibration code so I might also check that later because I intend to modify some parts there for my own use.

If you have the default franka gripper and end up getting in to the same issue with grasping an object -> Gripper freezes/become unresponsive, I'd like to know if you come up with any solution. I managed to fix it but in a slightly non-ideal way.

The issue is (probably) related to polymetis and should be easily solved when this PR get merged on polymetis

[chisarie]

Hi @Harimus , could you let me know how did you solve the freezing gripper issue in the real franka? When I grasp an object, the gripper does not open anymore.

I already tried adding the changes suggested in the PR you linked, but it is still not working for me. (I checked already that the changes are there, I rebuilt the code and added a print statement in the cpp to double check).

[chisarie]

Ah I think I solved it. If anybody is interested, my solution was to modify the command_joint_state function in panda.py as following:

    def command_joint_state(self, joint_state: np.ndarray) -> None:
        """Command the leader robot to a given state.

        Args:
            joint_state (np.ndarray): The state to command the leader robot to.
        """
        self.robot.update_desired_joint_positions(torch.tensor(joint_state[:-1]))
        gripper_closed = joint_state[-1] > 0.25
        if gripper_closed and not self.gripper_closed:
            self.gripper_closed = True
            self.gripper.grasp(speed=0.1, force=1.0)
        elif not gripper_closed and self.gripper_closed:
            self.gripper_closed = False
            self.gripper.stop()
            self.gripper.goto(width=MAX_OPEN, speed=1, force=1)
        return

[Harimus]

Yes I solved in a similar way, by using the grasp function instead. However after some testing, it seems like that function ignores the force value and grasp things with constant force whatever the value.

Did you observe the same thing?

[chisarie]

I didn't look into the grasp force myself, but I have heard from some other colleagues that they had this problem. They were using franka_ros instead of polymetis, so it must be an issue with the franka api itself.

[obiza-theaiinstitute]

Yes, it seems that precise finger control is not possible with the Franka gripper? The authors avoid this by using the Robotiq gripper.