Test data leakage issue

[dschaehi]

Hi, can you confirm the following claim in [1]?

KBAT incorporates ConvKB in the last layer of its model
architecture, which should be affected by different evaluation
protocols. But we find another bug on the leakage of test
triples during negative sampling in the reported model, which
results in more significant performance degradation.

[1] Z. Sun, S. Vashishth, S. Sanyal, P. Talukdar, Y. Yang, A Re-evaluation of Knowledge Graph Completion Methods, ArXiv:1911.03903 [Cs]. (2019). http://arxiv.org/abs/1911.03903 (accessed November 14, 2019).

[hermangalio]

I would also be interested in this. Apparently the MRR score goes from .518 to .157 for the FB15k-237 dataset.

[LemonQC]

Hi, can you confirm the following claim in [1]?

KBAT incorporates ConvKB in the last layer of its model
architecture, which should be affected by different evaluation
protocols. But we find another bug on the leakage of test
triples during negative sampling in the reported model, which
results in more significant performance degradation.

[1] Z. Sun, S. Vashishth, S. Sanyal, P. Talukdar, Y. Yang, A Re-evaluation of Knowledge Graph Completion Methods, ArXiv:1911.03903 [Cs]. (2019). http://arxiv.org/abs/1911.03903 (accessed November 14, 2019).

But how to find the leak data in the source code. I have mailed to Sun, while I got no reply

[chauhanjatin10]

Hi @dschaehi , @SakerLiu , @hgalioulline ,
Thanks for pointing the bug. You can follow the updates in issue #28 .
Thanks
Jatin