-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
133 lines (121 loc) · 13.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
<!DOCTYPE html>
<html lang="en-us">
<head>
<meta charset="UTF-8">
<title>XINFOTABS</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#157879">
<link rel="stylesheet" href="css/normalize.css">
<link href='https://fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css'>
<link rel="stylesheet" href="css/cayman.css">
</head>
<body>
<section class="page-header">
<h1><img src="figures/xinfotabs_logo2.png" style="max-width:50%;"></h1>
<a href="https://aclanthology.org/2022.fever-1.7.pdf" class="btn">Paper</a>
<a href="https://github.com/XInfoTabS/dataset" class="btn">Data+Code</a>
<a href="https://infotabs.github.io/" class="btn">InfoTabS</a>
<a href="https://youtu.be/A9aS7sXY3BA" class="btn">Video</a>
<a href="https://vgupta123.github.io/docs/XInfoTabS_Poster.pdf" class="btn">Poster</a>
<a href="https://vgupta123.github.io/docs/XInfoTabS_PPT.pdf" class="btn">PPT</a>
</section>
<section class="main-content">
<h1>Evaluating Multilingual Tabular Natural Language Inference</h1>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>About</h2><p style="text-align: justify;"> The ability to reason about tabular or semi-structured knowledge is a fundamental problem for today's Natural Language Processing (NLP) systems. While significant progress has been achieved in the direction of tabular reasoning, these advances are limited to English due to the absence of multilingual benchmark datasets for semi-structured data. In this paper, we use machine translation methods to construct a multilingual tabular natural language inference (TNLI) dataset, namely XInfoTabS, which expands the English TNLI dataset of InfoTabS to ten diverse languages. We also present several baselines for multilingual tabular reasoning, e.g., machine translation-based methods and cross-lingual TNLI.<p>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>The Tabular Inference Problem</h2>
<p style="text-align: justify; display:inline;"> Given a premise table, the task is to determine whether given hypothesis is true (<span style="color: darkgreen">entailment</span>), false (<span style="color: red">contradiction</span>), or undetermined (<span style="color: blue">neutral</span>, i.e. tabular natural language inference. Below is an example from the <a href="infotabs.github.io">INFOTABS </a>dataset:</p>
<p style="margin-left:10%; margin-right:10%;"><img src="figures/example.png" style="max-width:65%;"></p>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>The XInfoTabS dataset</h2>
<p style="text-align: justify; display:inline;">
XInfoTabS is a multi-lingual extension of InfoTabS dataset.
The XInfoTabS dataset consists of ten languages, namely English (`en'), German (`de'), French (`fr'), Spanish (`es'),
Afrikaans (`af'), Russian (`ru'), Chinese (`zh'), Korean (`ko'), Hindi (`hi') and Arabic (`ar'), which belong to seven
distinct language families and six unique writing scripts. Furthermore, these languages are the majority spoken in all seven continents
covering 2.76 billion native speakers in comparison to 360 million English language InfoTabS speakers.
</p>
<p>
To create XInfoTabS, we leverage machine translation models which provide high-quality translations of tabular data. Below is an example from the dataset.
</p>
<p style="margin-left:10%; margin-right:10%;"><img src="figures/xinfotabs_example.png" style="max-width:100%;"></p>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Motivation</h2>
<!--<p style="text-align: justify;"><ul>
<li>Recent work mostly focuses on <b>building</b> sophisticated <b>neural models</b></li>
<li>How will models designed for <b>raw text adapt for tabular data</b>?</li>
<li>How to <b>represent data</b> and <b>incorporate knowledge</b> into the model?
<li>Can better <b>pre-processing</b> of <b>tabular information</b> enhance table comprehension?
</ul></p>-->
<p style="text-align: justify;">To date, <b> no work</b> has been done in the field of <b>multilingual tabular inference</b>. All existing works are done entirely in English language.
. We study the following questions in regards to the multilingual tabular reasoning problem: How can we <b>create a dataset</b> that can be leveraged to train and evaluate <b>multilingual models</b> for the task? How well can multilingual models (for example, XLM-RoBERTa and mBERT) reason about <b>multilingual tabular inference</b>?
</ul></p>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Key Challenges</h2>
<p style="text-align: justify;"> The following are the key challenges encountered while generating a multilingual dataset using machine translation:
<ul>
<li>Tabular data that is semi-structured contains succinct, non-sentential implicit information. As a result, translation is difficult.
</li>
<li>Translation quality is not universal. Quality varies with multilingual models (e..g mBART, M2M, MarianMT), 11 languages and data format (i.e. table, hypothesis)</li>
<li>How to measure the translations quality using automatic metric and human rating especially for tabular semi-structured data.
</ul>
</p>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a><span>◈</span> Table Translation Pipeline</h2>
<h3><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Challenge</h3>
<p style="text-align: justify;"> Translating tabular data is more challenging than normal sentences because translation models are trained to translate proper sentences but our data is in the form of key value pairs which makes it difficult to incorporate context for good quality translation</p>
<h3><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Solution: </h3>
<p style="text-align: justify;">We devised a translation pipeline which incorporates context addition thorugh table category and methods for handling named entitities </p>
<p style="margin-left:5%; margin-right:5%;"><img src="figures/pipeline.png" style="width: 1000px;"></p>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a><span>◈</span> Measuring the Quality of translations</h2>
<h3><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Challenge</h3>
<p style="text-align: justify;"> Since we are generating the dataset using machine translation, it is very important to ensure that the quality of translations are good. Thus we need to have a set of methods using which can judge the quality of the data. </p>
<h3><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Solution: </h3>
<p style="text-align: justify;">We use a combination of both human verification and automatic methods to determine the accuracy of the translations. The different scores are mentioned below</p>
<p style="margin-left:10%; margin-right:10%;"><img src="figures/accuracy.png" style="width: 1000px;"></p>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a><span>◈</span> Choosing Models for Translation</h2>
<h3><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Challenge</h3>
<p style="text-align: justify;"> There are multiple open source models available for machine translation, including both bilingual and multilingual models. To ensure maximum translation quality, it is essential to choose the best translation model for each language. </p>
<h3><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Solution: </h3>
<p style="text-align: justify;"> To solve this problem we generate a subset of the InfoTabS dataset and perform translation using different bilingual and multilingual models for all languages </p>
<p style="text-align: justify;"> Finally, the model with the best human evaluation score is selected for a given language.</p>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Experimental Results</h2>
<p style="text-align: justify; display:inline;">Finally we perform the taskl of tabular NLI on the dataset created. We use a total of 5 different finetuning methods. The task wise and language wise results are shown below:</p>
<p style="margin-left:7%; margin-right:7%;"><img src="figures/result-task-wise.png" style="width: 1000px;"></p>
<p style="margin-left:7%; margin-right:7%;"><img src="figures/result-language-wise.png" style="width: 1000px;"></p>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>People</h2>
<p style="text-align: justify;"> The following people have worked on the paper<a href="https://aclanthology.org/2022.fever-1.7.pdf"> "XINFOTABS: Evaluating Multilingual Tabular Natural Language Inference"</a>: </p>
<figure>
<img src="figures/bhavnick.jpeg" style="width:32.5%;">
<img src="figures/anant.jpeg" style="width:32.5%;">
<img src="figures/vivekg.jpg" style="width:32.5%;">
<figcaption>From left to right, <a href="https://www.linkedin.com/in/bhavnicksm/">Bhavnick Minhas</a>, <a href="https://www.linkedin.com/in/anantshankhdhar/">Anant Shankhdhar</a> and <a href="https://vgupta123.github.io">Vivek Gupta</a> </figcaption>
</figure>
<figure>
<img src="figures/divyanshu.jpeg" style="width:32.5%;">
<img src="figures/shuo.jpeg" style="width:32.5%;">
<figcaption>From left to right, <a href="https://www.linkedin.com/in/divyanshu-aggarwal-498894163/">Divyanshu Aggarwal</a> and <a href="https://imsure318.github.io/">Shuo Zhang</a> </figcaption>
</figure>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Citation</h2>
<p style="text-align: justify;"> Please cite our paper as below.</p>
<pre><code>@inproceedings{minhas-etal-2022-xinfotabs,
title = "{XI}nfo{T}ab{S}: Evaluating Multilingual Tabular Natural Language Inference",
author = "Minhas, Bhavnick and
Shankhdhar, Anant and
Gupta, Vivek and
Aggarwal, Divyanshu and
Zhang, Shuo",
booktitle = "Proceedings of the Fifth Fact Extraction and VERification Workshop (FEVER)",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.fever-1.7",
pages = "59--77",
abstract = "The ability to reason about tabular or semi-structured knowledge is a fundamental problem for today{'}s Natural Language Processing (NLP) systems. While significant progress has been achieved in the direction of tabular reasoning, these advances are limited to English due to the absence of multilingual benchmark datasets for semi-structured data. In this paper, we use machine translation methods to construct a multilingual tabular NLI dataset, namely XINFOTABS, which expands the English tabular NLI dataset of INFOTABS to ten diverse languages. We also present several baselines for multilingual tabular reasoning, e.g., machine translation-based methods and cross-lingual. We discover that the XINFOTABS evaluation suite is both practical and challenging. As a result, this dataset will contribute to increased linguistic inclusion in tabular reasoning research and applications.",
}</code></pre>
<h2><a id="user-content-header-2" class="anchor" href="#header-2" aria-hidden="true"><span class="octicon octicon-link"></span></a>Acknowledgement</h2>
<p style="text-align: justify;">Authors thank members of the <a href="https://svivek.com/">Utah NLP group</a> for their valuable insights and
suggestions at various stages of the project; and reviewers for their helpful comments. Additionally, we appreciate the inputs provided by Vivek Srikumar and Ellen Riloff. Vivek Gupta acknowledges support from Bloomberg's Data Science Ph.D. Fellowship.</p>
<footer class="site-footer">
<span class="site-footer-owner"><a href="https://xinfotabs.github.io/">XINFOTABS</a> is maintained by <a href="https://www.linkedin.com/in/anantshankhdhar/">Anant Shankhdhar</a>.</span>
<span class="site-footer-credits">This page was generated by <a href="https://pages.github.com">GitHub Pages</a> using the <a href="https://github.com/jasonlong/cayman-theme">Cayman</a> theme by <a href="https://github.com/jasonlong">jasonlong</a>.</span>
</footer>
</section>
</body>
</html>