add ciou and diou

[fsx950223]

1.refact giou loss.
2.add ciou loss and diou loss.
3.expose iou apis as public api.

[facaiy]

@fsx950223 Hi, I was not active on github these two month, could you please link to the related issue (feature request or some discussion)?

[fsx950223]

@fsx950223 Hi, I was not active on github these two month, could you please link to the related issue (feature request or some discussion)?

Related PR: #477
Related paper: https://arxiv.org/abs/1911.08287

[zhen8838]

你好，我看了你实现的ciou，有几个小问题想交流下

1. ciou里面，是不是应该加一个clip by value好点, 因为像yolo里面的无效区域的wh标签是0，除0的时候会出现nan值。

ver = 4 * (((tf.atan(b1_width / b1_height) - tf.atan(b2_width / b2_height)) / math.pi)**2)

如果出现nan值，普通乘mask没法消去，还得用tf.where来消除，感觉会比较蛋疼。

boxes1 = tf.constant([[4.0, 3.0, 7.0, 5.0], [5.0, 6.0, 10.0, 7.0]])
boxes2 = tf.constant([[3.0, 4.0, 6.0, 8.0], [.0, .0, .0, .0]])
_common_iou(boxes1, boxes2, 'ciou')
# [-0.42209864,         nan]
tf.zeros([2]) * _common_iou(boxes1, boxes2, 'ciou')
# [-0., nan]
1 - tf.zeros([2]) * _common_iou(boxes1, boxes2, 'ciou')
# [ 1., nan]

感觉可以直接在求arctan的时候加div_no_zero，还有就是clip by value，我看作者原本的实现是直接clip的。

2. 还有就是要不要在ciou里面屏蔽部分梯度，作者pytorch里面是把求系数的部分都梯度屏蔽了 ，请问有实验过不屏蔽梯度效果怎么样？

[fsx950223]

你好，我看了你实现的ciou，有几个小问题想交流下

1. ciou里面，是不是应该加一个clip by value好点, 因为像yolo里面的无效区域的wh标签是0，除0的时候会出现nan值。

ver = 4 * (((tf.atan(b1_width / b1_height) - tf.atan(b2_width / b2_height)) / math.pi)**2)

如果出现nan值，普通乘mask没法消去，还得用tf.where来消除，感觉会比较蛋疼。

boxes1 = tf.constant([[4.0, 3.0, 7.0, 5.0], [5.0, 6.0, 10.0, 7.0]])
boxes2 = tf.constant([[3.0, 4.0, 6.0, 8.0], [.0, .0, .0, .0]])
_common_iou(boxes1, boxes2, 'ciou')
# [-0.42209864,         nan]
tf.zeros([2]) * _common_iou(boxes1, boxes2, 'ciou')
# [-0., nan]
1 - tf.zeros([2]) * _common_iou(boxes1, boxes2, 'ciou')
# [ 1., nan]

感觉可以直接在求arctan的时候加div_no_zero，还有就是clip by value，我看作者原本的实现是直接clip的。

1. 还有就是要不要在ciou里面屏蔽部分梯度，作者pytorch里面是把求系数的部分都梯度屏蔽了 ，请问有实验过不屏蔽梯度效果怎么样？

1.tf.atan的值域在[-1.6,1.6]之间,所以我觉得不用clip
2.

论文给出的loss公式如上图所示，不明白为什么official实现是这个样子的

[zhen8838]

@fsx950223

1. tf.atan值域虽然是[-1.6,1.6],但万一里面的w/h是nan他的结果也会是nan了。。
2. 我参考作者和知乎链接，里面说要把v的梯度缩小，避免爆炸。用stop grad之后梯度上数值大小的确小了一些但也有几个值变大了，感觉他用stop grad之后并不是把数值减小，而是想把梯度都放到预测bbox上去。我测试代码里面发现，alpha的梯度没有了，把ar的梯度都集中在预测bbox的宽度和高度上。
3. 不过他为什么还有加上(8 / tf.square(np.pi))我就不知道了。。

我的测试代码：

def test_grad():
    offset = 0.
    a = tf.Variable([[4.0, 3.0, 7.0, 5.0], [5.0, 6.0, 10.0, 7.0]])
    b = tf.Variable([[3.0, 4.0, 6.0, 8.0], [14.0, 14.0, 15.0, 15.0]])
    with tf.GradientTape(True) as tape:
        a = a[..., None, :]
        b = b[..., None, :]
        tl = tf.maximum(a[..., :2], b[..., :2])
        br = tf.minimum(a[..., 2:4], b[..., 2:4])

        area_i = tf.reduce_prod(tf.maximum(br - tl, 0) + offset, axis=-1)
        area_a = tf.reduce_prod(a[..., 2:4] - a[..., :2] + offset, axis=-1)
        area_b = tf.reduce_prod(b[..., 2:4] - b[..., :2] + offset, axis=-1)
        iou = area_i / (area_a + area_b - area_i)

        outer_tl = tf.minimum(a[..., :2], b[..., :2])
        outer_br = tf.maximum(a[..., 2:4], b[..., 2:4])
        # two bbox center distance sum((b_cent-a_cent)^2)
        inter_diag = tf.reduce_sum(tf.square((b[..., :2] + b[..., 2:]) / 2
                                             - (a[..., :2] + a[..., 2:]) / 2 + offset), -1)
        # two bbox diagonal distance
        outer_diag = tf.reduce_sum(tf.square(outer_tl - outer_br + offset), -1)
        # calc ciou alpha paramter

        arctan = ((tf.math.atan((b[..., 2] - b[..., 0]) / (b[..., 3] - b[..., 1])))
                  - tf.math.atan((a[..., 2] - a[..., 0]) / (a[..., 3] - a[..., 1])))

        v = tf.math.square(2 / np.pi) * tf.square(arctan)
        alpha = v / ((1 - iou) + v)
        w_temp = 2 * (a[..., 2] - a[..., 0])

        ar = (8 / tf.square(np.pi)) * arctan * ((a[..., 2] - a[..., 0] - w_temp) * (a[..., 3] - a[..., 1]))

        ciou = iou - (inter_diag / outer_diag) - (alpha * ar)

    tape.gradient(v, [a, b])
    """ [[[-0.04231081,  0.06346621,  0.04231081, -0.06346621]],
        [[-0.01833142,  0.09165711,  0.01833142, -0.09165711]]]
        
        [[[ 0.04400324, -0.03300243, -0.04400324,  0.03300243]],
        [[ 0.23830847, -0.23830847, -0.23830847,  0.23830847]]]"""
    tape.gradient(ciou, [a, b])
    """ 
        [[[ 0.06351729, -0.08257562, -0.15244228,  0.27827212]],
        [[ 0.08103281,  0.01117781, -0.07266921,  0.01513398]]],
   
        [[[-0.0851735 , -0.06970734,  0.17409849, -0.12598914]],
        [[-0.7246207 ,  0.6417478 ,  0.7162571 , -0.6680595 ]]]
    """

    offset = 0.
    a = tf.Variable([[4.0, 3.0, 7.0, 5.0], [5.0, 6.0, 10.0, 7.0]])
    b = tf.Variable([[3.0, 4.0, 6.0, 8.0], [14.0, 14.0, 15.0, 15.0]])
    with tf.GradientTape(True) as tape:
        a = a[..., None, :]
        b = b[..., None, :]
        tl = tf.maximum(a[..., :2], b[..., :2])
        br = tf.minimum(a[..., 2:4], b[..., 2:4])

        area_i = tf.reduce_prod(tf.maximum(br - tl, 0) + offset, axis=-1)
        area_a = tf.reduce_prod(a[..., 2:4] - a[..., :2] + offset, axis=-1)
        area_b = tf.reduce_prod(b[..., 2:4] - b[..., :2] + offset, axis=-1)
        iou = area_i / (area_a + area_b - area_i)

        outer_tl = tf.minimum(a[..., :2], b[..., :2])
        outer_br = tf.maximum(a[..., 2:4], b[..., 2:4])
        # two bbox center distance sum((b_cent-a_cent)^2)
        inter_diag = tf.reduce_sum(tf.square((b[..., :2] + b[..., 2:]) / 2
                                             - (a[..., :2] + a[..., 2:]) / 2 + offset), -1)
        # two bbox diagonal distance
        outer_diag = tf.reduce_sum(tf.square(outer_tl - outer_br + offset), -1)
        # calc ciou alpha paramter

        arctan = tf.stop_gradient(
            (tf.math.atan((b[..., 2] - b[..., 0]) / (b[..., 3] - b[..., 1])))
            - tf.math.atan((a[..., 2] - a[..., 0]) / (a[..., 3] - a[..., 1])))

        v = tf.stop_gradient(tf.math.square(2 / np.pi) * tf.square(arctan))
        alpha = tf.stop_gradient(v / ((1 - iou) + v))
        w_temp = tf.stop_gradient(2 * (a[..., 2] - a[..., 0]))
        ar = (8 / tf.square(np.pi)) * arctan * ((a[..., 2] - a[..., 0] - w_temp) * (a[..., 3] - a[..., 1]))

        ciou = iou - (inter_diag / outer_diag) - (alpha * ar)
    
    tape.gradient(ar, [a, b])
    """ 
        [[[ 0.5500405 , -0.8250607 , -0.5500405 ,  0.8250607 ]],
        [[ 0.47661695, -2.3830848 , -0.47661695,  2.3830848 ]]],
        
        None
     """
    tape.gradient(v, [a, b])
    """ [None, None] """
    tape.gradient(ciou, [a, b])
    """ 
        [[[-0.10692195,  0.08424009,  0.01162432,  0.12420168]],
        [[-0.08888854,  0.27500343,  0.09725215, -0.24869165]]]
        
        [[[ 0.03184488, -0.16596799,  0.06345274, -0.04247379]],
        [[-0.03867403, -0.0441989 ,  0.03031043,  0.01788712]]]
    """

[fsx950223]

@fsx950223

1. tf.atan值域虽然是[-1.6,1.6],但万一里面的w/h是nan他的结果也会是nan了。。
2. 我参考作者和知乎链接，里面说要把v的梯度缩小，避免爆炸。用stop grad之后梯度上数值大小的确小了一些但也有几个值变大了，感觉他用stop grad之后并不是把数值减小，而是想把梯度都放到预测bbox上去。我测试代码里面发现，alpha的梯度没有了，把ar的梯度都集中在预测bbox的宽度和高度上。
3. 不过他为什么还有加上(8 / tf.square(np.pi))我就不知道了。。

我的测试代码：

def test_grad():
    offset = 0.
    a = tf.Variable([[4.0, 3.0, 7.0, 5.0], [5.0, 6.0, 10.0, 7.0]])
    b = tf.Variable([[3.0, 4.0, 6.0, 8.0], [14.0, 14.0, 15.0, 15.0]])
    with tf.GradientTape(True) as tape:
        a = a[..., None, :]
        b = b[..., None, :]
        tl = tf.maximum(a[..., :2], b[..., :2])
        br = tf.minimum(a[..., 2:4], b[..., 2:4])

        area_i = tf.reduce_prod(tf.maximum(br - tl, 0) + offset, axis=-1)
        area_a = tf.reduce_prod(a[..., 2:4] - a[..., :2] + offset, axis=-1)
        area_b = tf.reduce_prod(b[..., 2:4] - b[..., :2] + offset, axis=-1)
        iou = area_i / (area_a + area_b - area_i)

        outer_tl = tf.minimum(a[..., :2], b[..., :2])
        outer_br = tf.maximum(a[..., 2:4], b[..., 2:4])
        # two bbox center distance sum((b_cent-a_cent)^2)
        inter_diag = tf.reduce_sum(tf.square((b[..., :2] + b[..., 2:]) / 2
                                             - (a[..., :2] + a[..., 2:]) / 2 + offset), -1)
        # two bbox diagonal distance
        outer_diag = tf.reduce_sum(tf.square(outer_tl - outer_br + offset), -1)
        # calc ciou alpha paramter

        arctan = ((tf.math.atan((b[..., 2] - b[..., 0]) / (b[..., 3] - b[..., 1])))
                  - tf.math.atan((a[..., 2] - a[..., 0]) / (a[..., 3] - a[..., 1])))

        v = tf.math.square(2 / np.pi) * tf.square(arctan)
        alpha = v / ((1 - iou) + v)
        w_temp = 2 * (a[..., 2] - a[..., 0])

        ar = (8 / tf.square(np.pi)) * arctan * ((a[..., 2] - a[..., 0] - w_temp) * (a[..., 3] - a[..., 1]))

        ciou = iou - (inter_diag / outer_diag) - (alpha * ar)

    tape.gradient(v, [a, b])
    """ [[[-0.04231081,  0.06346621,  0.04231081, -0.06346621]],
        [[-0.01833142,  0.09165711,  0.01833142, -0.09165711]]]
        
        [[[ 0.04400324, -0.03300243, -0.04400324,  0.03300243]],
        [[ 0.23830847, -0.23830847, -0.23830847,  0.23830847]]]"""
    tape.gradient(ciou, [a, b])
    """ 
        [[[ 0.06351729, -0.08257562, -0.15244228,  0.27827212]],
        [[ 0.08103281,  0.01117781, -0.07266921,  0.01513398]]],
   
        [[[-0.0851735 , -0.06970734,  0.17409849, -0.12598914]],
        [[-0.7246207 ,  0.6417478 ,  0.7162571 , -0.6680595 ]]]
    """

    offset = 0.
    a = tf.Variable([[4.0, 3.0, 7.0, 5.0], [5.0, 6.0, 10.0, 7.0]])
    b = tf.Variable([[3.0, 4.0, 6.0, 8.0], [14.0, 14.0, 15.0, 15.0]])
    with tf.GradientTape(True) as tape:
        a = a[..., None, :]
        b = b[..., None, :]
        tl = tf.maximum(a[..., :2], b[..., :2])
        br = tf.minimum(a[..., 2:4], b[..., 2:4])

        area_i = tf.reduce_prod(tf.maximum(br - tl, 0) + offset, axis=-1)
        area_a = tf.reduce_prod(a[..., 2:4] - a[..., :2] + offset, axis=-1)
        area_b = tf.reduce_prod(b[..., 2:4] - b[..., :2] + offset, axis=-1)
        iou = area_i / (area_a + area_b - area_i)

        outer_tl = tf.minimum(a[..., :2], b[..., :2])
        outer_br = tf.maximum(a[..., 2:4], b[..., 2:4])
        # two bbox center distance sum((b_cent-a_cent)^2)
        inter_diag = tf.reduce_sum(tf.square((b[..., :2] + b[..., 2:]) / 2
                                             - (a[..., :2] + a[..., 2:]) / 2 + offset), -1)
        # two bbox diagonal distance
        outer_diag = tf.reduce_sum(tf.square(outer_tl - outer_br + offset), -1)
        # calc ciou alpha paramter

        arctan = tf.stop_gradient(
            (tf.math.atan((b[..., 2] - b[..., 0]) / (b[..., 3] - b[..., 1])))
            - tf.math.atan((a[..., 2] - a[..., 0]) / (a[..., 3] - a[..., 1])))

        v = tf.stop_gradient(tf.math.square(2 / np.pi) * tf.square(arctan))
        alpha = tf.stop_gradient(v / ((1 - iou) + v))
        w_temp = tf.stop_gradient(2 * (a[..., 2] - a[..., 0]))
        ar = (8 / tf.square(np.pi)) * arctan * ((a[..., 2] - a[..., 0] - w_temp) * (a[..., 3] - a[..., 1]))

        ciou = iou - (inter_diag / outer_diag) - (alpha * ar)
    
    tape.gradient(ar, [a, b])
    """ 
        [[[ 0.5500405 , -0.8250607 , -0.5500405 ,  0.8250607 ]],
        [[ 0.47661695, -2.3830848 , -0.47661695,  2.3830848 ]]],
        
        None
     """
    tape.gradient(v, [a, b])
    """ [None, None] """
    tape.gradient(ciou, [a, b])
    """ 
        [[[-0.10692195,  0.08424009,  0.01162432,  0.12420168]],
        [[-0.08888854,  0.27500343,  0.09725215, -0.24869165]]]
        
        [[[ 0.03184488, -0.16596799,  0.06345274, -0.04247379]],
        [[-0.03867403, -0.0441989 ,  0.03031043,  0.01788712]]]
    """

我的实现与c版本的类似，pytorch版本为什么要把倒数部分加到前向传播里？实现出错了吗？

[zhen8838]

@fsx950223 我按照作者pytorch实现的ciou训练了一下，30mb的mbnetv1 yolo在voc上map到69.7了，提升4%左右，的确有效。不过我发现diou-nms好像提升不大，就零点几个点。还没有试过如果不屏蔽梯度会怎么样。

[Squadrick]

@fsx950223 @zhen8838 Thanks for the contribution and helping with the review. Could you use English for all public discussion? It would be beneficial to the rest of the community. For some reason, this page can't be translated by Chrome.

[fsx950223]

@zhen8838 This is the test result of ap0.5? The average ap0.5-0.95 and ap0.75 test indicators were used by the author

[zhen8838]

@zhen8838 This is the test result of ap0.5? The average ap0.5-0.95 and ap0.75 test indicators were used by the author

yes ap 0.5. others haven't been tested yet.

Now I am troubled by other problems, and I find that if I use ciou to train from scratch, the model will not converge. At the same time, if I use a lot of target data sets, such as face data sets, using pre-train model but ciou will still cause gradient explosion.

[fsx950223]

@zhen8838 This is the test result of ap0.5? The average ap0.5-0.95 and ap0.75 test indicators were used by the author

yes ap 0.5. others haven't been tested yet.

Now I am troubled by other problems, and I find that if I use ciou to train from scratch, the model will not converge. At the same time, if I use a lot of target data sets, such as face data sets, using pre-train model but ciou will still cause gradient explosion.

I got better performance on ap0.75 with current implementation, could you try it?

[zhen8838]

Now get better Map than before, I get 70.3Map at ap0.5 in voc.

[fsx950223]

Could config typeguard to disable the default __init__ rule?

[googlebot]

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

[gabrieldemarmiesse]

@googlebot I consent

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[gabrieldemarmiesse]

I've merged master into your branch to update it and fixed any formatting/conflicts it might have. If you need to do some more modifications, please do git pull beforehand.

[gabrieldemarmiesse]

@fsx950223 what's the status with this pull request? Do you need any help on the technical side? if this pull request if left without changes for around a week, we'll close it, but feel free to reopen after.

[Cospel]

This looks very cool, is anyone still working on this? Can this be merged?

[bhack]

This kind of things are landing in the keras-cv perimeter https://github.com/keras-team/keras-cv/pull/3/files#r441198652
Do we want still to invest contributors/reviewers time here or is it better to route features like this one in keras-cv?

/cc @seanpmorgan @gabrieldemarmiesse.

[bhack]

/cc @tanzhenyu

[tanzhenyu]

This kind of things are landing in the keras-cv perimeter https://github.com/keras-team/keras-cv/pull/3/files#r441198652
Do we want still to invest contributors/reviewers time here or is it better to route features like this one in keras-cv?

/cc @seanpmorgan @gabrieldemarmiesse.

I actually do think it might be preferrable to have iou/giou/ciou/diou live here (at least for now). What Keras-CV should have is popular losses used from mainstream models, so we're aiming to have things like focal loss, ssd loss (in a hard negative mining way), etc.

But if these losses are drop-in replacement and bring significant benefits to the main models today, we'd be happy to accept it :-)

[tanzhenyu]

/cc @tanzhenyu

Thanks @bhack for looping me in!!

[bhack]

@tanzhenyu Can you help us to review tensorflow/community#241 on the Keras side so that we have a better shared process to routing our (voluntary/limited) resources. Cause if the upstreaming process cannot work for Keras we need to patch that PR with another alternative process.

P.s.
e.g. please search focal in the repository cause we have some open issues/PRs and as it is mentioned in your target list we will not spent time on that we will just deprecate when it will be ready (upstreamed?) in keras-cv.

[tanzhenyu]

@tanzhenyu Can you help us to review tensorflow/community#241 on the Keras side so that we have a better shared process to routing our (voluntary/limited) resources. Cause if the upstreaming process cannot work for Keras we need to patch that PR with another alternative process.

P.s.
e.g. please search focal in the repository cause we have some open issues/PRs and as it is mentioned in your target list we will not spent time on that we will just deprecate when it will be ready (upstreamed?) in keras-cv.

Thanks I will take a look at the RFC.
Meanwhile yes we should move focal loss, I could do that, or ask the original contributor to help on that. As a matter of fact we need quite some help on implementing metrics related things (such as COCO metrics, mAP etc), so it'd be great to start re-directing external contribution in this topic.
I'd also be happy to help review image related PRs, but I don't think I'm a member to have review permissions?

[bhack]

I'd also be happy to help review image related PRs, but I don't think I'm a member to have review permissions?

For this /cc @seanpmorgan @WindQAQ

[AakashKumarNain]

@tanzhenyu Can you help us to review tensorflow/community#241 on the Keras side so that we have a better shared process to routing our (voluntary/limited) resources. Cause if the upstreaming process cannot work for Keras we need to patch that PR with another alternative process.
P.s.
e.g. please search focal in the repository cause we have some open issues/PRs and as it is mentioned in your target list we will not spent time on that we will just deprecate when it will be ready (upstreamed?) in keras-cv.

Thanks I will take a look at the RFC.
Meanwhile yes we should move focal loss, I could do that, or ask the original contributor to help on that. As a matter of fact we need quite some help on implementing metrics related things (such as COCO metrics, mAP etc), so it'd be great to start re-directing external contribution in this topic.
I'd also be happy to help review image related PRs, but I don't think I'm a member to have review permissions?

Hi @tanzhenyu . Let's move focal loss. Let me know what are all the changes that are required. Maybe we should open a separate issue and start discussion there?

[tanzhenyu]

@tanzhenyu Can you help us to review tensorflow/community#241 on the Keras side so that we have a better shared process to routing our (voluntary/limited) resources. Cause if the upstreaming process cannot work for Keras we need to patch that PR with another alternative process.
P.s.
e.g. please search focal in the repository cause we have some open issues/PRs and as it is mentioned in your target list we will not spent time on that we will just deprecate when it will be ready (upstreamed?) in keras-cv.

Thanks I will take a look at the RFC.
Meanwhile yes we should move focal loss, I could do that, or ask the original contributor to help on that. As a matter of fact we need quite some help on implementing metrics related things (such as COCO metrics, mAP etc), so it'd be great to start re-directing external contribution in this topic.
I'd also be happy to help review image related PRs, but I don't think I'm a member to have review permissions?

Hi @tanzhenyu . Let's move focal loss. Let me know what are all the changes that are required. Maybe we should open a separate issue and start discussion there?

SGTM. Please make sure it supports multi-classing, able to handle predictions w/ and w/o background label, etc.

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

[bhack]

tensorflow/models#9083