⚖️ How will fairness be ensured for the competition?

[Victorlouisdg]

ℹ️ I'm reposting a question I got from a participants via email here, because I believe the answer can be relevant to all participants.

Question

Firstly, during the competition, is the crumpled condition of the clothes on the table a random configuration?

Then, we have doubts about the fairness of the competition. In this task, different initial grasping points of the left arm have already determined the upper limit of the final unfolded area. For example, for a T-shirt, the gripper may grasp the cuff or the chest respectively, in the assumption that the best unfolding state can be achieved, the unfolding coverage of the former must be higher than the latter. So how do you ensure the fairness of the competition? I think that if you evaluate the performance of different participants' algorithms under different initial grasp positions in step 2, evaluation based entirely on the metric unfolding coverage is not reasonable.

As we all know, for the operation of deformable objects, especially cloth, due to the flexibility and extensibility of cloth, the greater tension is used by robot arms when unfolding, the larger final unfolding coverage of the cloth is achieved. Therefore, how do you ensure that the tension from robot arms in different participants' tests is the same? Even in case you implement exactly the same drag for the same cloth in different participants' tests, different grasp positions can bear different forces. After all, how do you ensure that the unfolding operation will not cause unfair competition with the above questions?

[Victorlouisdg]

Answer

Thank you for your thoughtful questions. Here's a breakdown of your concerns and how we’re addressing them:

Randomness of initialization

Rationale: The current initialization procedure, i.e. starting crumpled, grasping the highest point and then the lowest point, was chosen to ensure all grasps and consequently the hanging configuration are realistic and the result of fully autonomous robot execution.

We considered alternative initialization schemes, such as using human assistance to grasp specific points like sleeve cuffs. However, we decided against this approach. Human-assisted initialization would slow down the process. Additionally, while more controlled, it can introduce biases through artificially perfect grasps. Furthermore, reproducing consistent initial hanging configurations across multiple trials remains challenging even with human involvement.

Coverage as the (only) metric

Coverage is objective, easy to understand, broadly applicable to cloth categories, and aligns reasonably well with human perception of a successfully unfolded garment. However, we acknowledge that on its own it does not capture the difficulty arising from specific initial states. This challenge partially motivates the competition - to evaluate grasp selection methods under realistic conditions that are as similar as possible for the participating teams.

Extensibility of the cloth

Two aspects are important to consider:

• Limited effect on surface area: While some cloth items stretch slightly under tension, this stretching is mostly along the line between the grasp points, and has a relatively minor effect on the overall surface area.
• Tension control: Our UR5e robots have force-torque sensors in their wrists. We use these to approximate the tension in the cloth during the final stretch. We aim for a threshold of 4N, which is low enough to prevent damage to the cloth. However, achieving this exact tension consistently is difficult due to sensor noise, drift, and imperfect gravity compensation by the control system.

Fairness of the competition

You’re absolutely right that the initialization procedure can produce hanging configurations of varying difficulty. Here's how we're aiming for a fair competition:

• Multiple trials: Each team will face multiple cloth items and initializations. This helps balance out the impact of lucky vs. unlucky starting states.
• Improving the initialization: We will attempt to tune the lowest point grasp to grasp less of the cloth. Grasping less cloth generally allows it to drape more freely and increases the potential for unfolding.
• Variance reduction: The lowest point grasp helps reduce some of the randomness in the hanging configuration. Specific points (extrema like sleeve cuffs or waistline) are naturally more likely to be the lowest.
• Focus on maximization: While perfect unfolding might not always be possible, the goal remains to maximize the achieved surface area under the given constraints.

Conclusion

We are trying to guarantee fairness to the best of our ability. The initialization, while potentially limiting full unfolding, simulates real-world challenges. Coverage, though imperfect, is a quantifiable and objective metric. We control stretch as reasonably as possible. Multiple trials help mitigate the impact of variation.

Future outlook

For this competition, we’ve chosen a more constrained focus, with participants providing the second grasp on the hanging cloth, which is arguably the most critical for maximizing unfolding. In the future, we will expand the scope of participant control to include grasps during the initial cloth manipulation stages, or even grant full control over the robots.