-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
351 lines (317 loc) · 19.9 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="description" content="Project page for MultiON EAI Challenge CVPR 2024.">
<meta name="author" content="SFU">
<meta name="keywords" content="embodied ai workshop, vision navigation, multion, multion challenge, cvpr 2024 ">
<title>MultiON EAI Challenge CVPR 2024</title>
<!--FACEBOOK-->
<meta property="og:type" content="website" />
<meta property="og:url" content="http://multion-challenge.cs.sfu.ca" />
<meta property="og:title" content="MultiON EAI Challenge CVPR 2024" />
<meta property="og:description" content="Project page for MultiON EAI Challenge CVPR 2024." />
<!--TWITTER-->
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:title" content="MultiON EAI Challenge CVPR 2024" />
<meta name="twitter:description" content="Project page for MultiON EAI Challenge CVPR 2024." />
<!-- Bootstrap core CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/codemirror.min.css">
<link href="https://fonts.googleapis.com/css?family=Lato:400,400i|PT+Serif:700" rel="stylesheet">
<link rel="stylesheet" href="dist/css/style.css">
<link rel="stylesheet" href="dist/css/new_style.css">
</head>
<body class="is-boxed has-animations">
<div class="body-wrap boxed-container">
<header class="site-header">
<div class="btn-group" style="float:right;">
<button class="btn btn-danger dropdown-toggle" type="button" id="dropdownMenuButton" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Challenges
</button>
<div class="dropdown-menu" aria-labelledby="dropdownMenuButton">
<a class="dropdown-item" href="./index.html">2024</a><br>
<a class="dropdown-item" href="./2023.html">2023</a><br>
<a class="dropdown-item" href="./2022.html">2022</a><br>
<a class="dropdown-item" href="./2021.html">2021</a>
</div>
</div>
</header>
<main>
<section class="hero text-light text-center">
<div class="container-sm">
<div class="hero-inner">
<h1 class="hero-title h2-mobile mt-0 is-revealing">
Multi-Object Navigation (MultiON) Challenge
</h1>
<div>
Held in conjunction with the <a href="https://embodied-ai.org/" target="_blank">Embodied AI
Workshop</a> at CVPR 2024
</div>
<br>
</div>
</div>
</section>
<div class="container">
<div class="row">
<div class="has-top-divider"> </div>
<div class="col-lg-12 col-md-12 col-sm-12 col-xs-12">
<div class="col-lg-12 col-md-12 col-sm-12 col-xs-12 text-center">
<image src="videos/2024/rgb.gif" class="img-fluid img-thumbnail rounded"
alt="cylinder-track"></image>
<figcaption class="text-center gif-img-caption">
An example episode from the MultiON challenge 2024.
</figcaption>
</div>
<div class="col-lg-12 col-md-12 col-sm-12 col-xs-12 text-center">
<div class="col-lg-12 col-md-12 col-sm-12 col-xs-12 text-center goal-img">
Goals:
<div class="img-placeholder" style="display: inline-block; margin: 0 5px;">
<div class="square" style="width: 140px; height: 50px; background-color: white; border: 1px solid black; margin-bottom: 5px;">Go to the kitchen sink cabinet.</div>
</div>
⟶
<div class="img-placeholder" style="display: inline-block; margin: 0 5px;">
<div class="square" style="width: 100px; height: 50px; background-color: white; border: 1px solid black; margin-bottom: 5px;">Find the pet house.</div>
</div>
⟶
<div class="img-placeholder" style="display: inline-block; margin: 0 5px;">
<div class="square" style="width: 200px; height: 50px; background-color: white; border: 1px solid black; margin-bottom: 5px;">Go to the window curtain on the sliding door.</div>
</div>
</div>
</div>
</div>
</div>
</div>
<section class="overview section text-center">
<div class="has-top-divider"> </div>
<div class="container">
<div class="overview-inner section-inner">
<h2 class="section-title mt-0">Overview</h2>
Multi-Object Navigation (MultiON) challenge is hosted at Embodied AI workshop, CVPR 2024. This
challenge is built upon the task introduced in <a
href="https://shivanshpatel35.github.io/multi-ON/">MultiON: Benchmarking Semantic Map Memory
using Multi-Object Navigation</a> (NeurIPS 2020). MultiON is a long-horizon generalization
of the object-navigation task, i.e., where the agent has to navigate to multiple goals.
</div>
</div>
</section>
<section class="citation section text-left">
<div class="container">
<div class="participation-inner section-inner">
<h2 class="section-title mt-0">Dates</h2>
<div>
<table class="dates-table">
<tr>
<td>Challenge starts <br>(Dataset and starter code <a href="https://github.com/3dlg-hcvc/multion-challenge" target="_blank"> available </a> and <br>EvalAI opens up for
Minival Phase submissions)</td>
<td>April 19, 2024</td>
</tr>
<tr>
<td><a href="https://eval.ai/web/challenges/challenge-page/2276/leaderboard">Leaderboard</a> opens <br>(Test Standard Phase and <br>Test Challenge Phase submissions)
</td>
<td>April 19, 2024</td>
</tr>
<tr>
<td>Challenge submission deadline</td>
<td>June 3, 2024 (AoE)</td>
</tr>
<tr>
<td>Winner announcement at <a href="https://embodied-ai.org/">Embodied AI Workshop, CVPR
2024</a></td>
<td>June, 2024</td>
</tr>
</table>
</div>
</div>
</div>
</section>
<section class="task section text-left">
<div class="container">
<div class="section-inner">
<h2 class="section-title mt-0">Task</h2>
In MultiON, an agent is tasked with navigating to a sequence of objects. These objects are placed into a realistic 3D environment. The task is based on the <a
href="http://aihabitat.org/">Habitat platform</a> and <a
href="https://3dlg-hcvc.github.io/hssd/">Habitat Synthetic Scenes Dataset (HSSD)</a> scenes.
<br><br>
Each episode contains 3 target objects randomly sampled from the objects present in the scene. Unlike previous challenges, this time the objects are described by a language instruction such as ‘Find the mantel clock on the chest of drawers’ instead of an object category. Moreover, the set of goal objects is not known apriori, making it closer to a real-world setting where the agent might be asked to ‘any’ object in the environment.
Each language instruction may contain a coarse description of the object (‘Go to the candle’) or a fine-grained description (‘Find the mini spa candle’). In the case of the former, there might exist multiple valid goal objects and navigating to any of them is considered successful. Instructions may also contain spatial relations between objects such as ‘Find the red short pillar candle on the grey nightstand’.
<br><br>
In summary, in each episode, the agent is initialized at a random starting position and orientation in
an unseen environment and provided a sequence of 3 target objects.
The agent must navigate to each target object in the sequence (in the given
order) and call the FOUND action to indicate discovery.
The agent has access to an RGB-D camera and a noiseless GPS+Compass sensor. GPS+Compass sensor provides
the agent's current location and orientation information relative to the start of the episode.
</div>
</div>
</section>
<section class="evaluation section text-left has-top-divider">
<div class="container">
<div class="evaluation-inner section-inner">
<h2 class="section-title mt-0">Evaluation Details</h2>
The episode terminates when an agent discovers all objects in the sequence of the current episode or
when it calls an incorrect FOUND action. A FOUND action is incorrect if it is called when the agent is
not within a 1.5m from its current target object. Note that this does not require the
agent to be viewing the object at the time of calling FOUND. After the episode terminates, the agent is
evaluated using the Progress and PPL metrics that are defined below.<br>
<strong>Progress</strong>: The proportion of objects correctly found in the episode.<br>
<strong>PPL</strong>: Progress weighted by path length. PPL quantifies the efficiency of agent's
trajectory with respect to the optimal trajectory.<br>
</div>
</div>
</section>
<section class="citation section text-left has-top-divider">
<div class="container">
<div class="participation-inner section-inner">
<h2 class="section-title mt-0">Submission Guidelines</h2>
Participants must make submission through our <a
href="https://eval.ai/web/challenges/challenge-page/2276/overview">EvalAI</a> page. There are three
phases in the challenge.
<h4 class="section-title mt-8">Phase 1: Minival Phase</h4>
This phase evaluates MultiON agents on the minival set of the MultiON dataset. This phase is meant to be used for sanity checking the
results of remote evaluation against your local evaluations.
<h4 class="section-title mt-8">Phase 2: Test Standard Phase</h4>
This results of this phase will be used to prepare the public leaderboard for the challenge. We suggest using this
phase for reporting results in papers and for comparing with other methods. Each team is allowed
a maximum of 3 submissions per day for this phase.
<h4 class="section-title mt-8">Phase 3: Test Challenge Phase</h4>
Only submissions made in this phase are considered as entries to the MultiON
Challenge since this will be used to decide the winners. Each team is allowed a
total of 3 submissions to this phase until the end of this phase.
For detailed submission instruction, please refer <a
href="https://github.com/3dlg-hcvc/multion-challenge">this</a>.
</div>
</div>
</section>
<section class="citation section text-left has-top-divider">
<div class="container">
<div class="participation-inner section-inner">
<h2 class="section-title mt-0">Challenge Updates</h2>
Any updates related to the challenge will be posted here. Please join the Google Group email list to receive updates about the challenge: <a href="https://groups.google.com/g/multion-challenge-2023/">click here</a> to join or send an
email to
<a
href="mailto:multion-challenge-2023+subscribe@googlegroups.com?&subject=Subscribe to email list&body=Subscribe">multion-challenge-2023+subscribe@googlegroups.com</a>.
</div>
</div>
</section>
<section class="citation section text-left">
<div class="container">
<div class="participation-inner section-inner">
<h2 class="section-title mt-0">Terms and Conditions</h2>
The Habitat-Sim is released under <a
href="https://github.com/facebookresearch/habitat-sim/blob/master/LICENSE" target="_blank">MIT
license</a>.
To use HSSD dataset, please refer <a href="https://3dlg-hcvc.github.io/hssd/"
target="_blank">here</a>.
If you use Habitat-Sim or the MultiON dataset in a paper, please consider citing the following
publications:
<div>
<pre style="background-color: rgb(189, 211, 207);">
<code style="background-color: rgb(189, 211, 207); color: black">@inproceedings{habitat19iccv,
title={Habitat: {A} {P}latform for {E}mbodied {AI} {R}esearch},
author={Manolis Savva and Abhishek Kadian and Oleksandr Maksymets and Yili Zhao and Erik Wijmans and Bhavana Jain and Julian Straub and Jia Liu and Vladlen Koltun and Jitendra Malik and Devi Parikh and Dhruv Batra},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year={2019}
}</code></pre>
<pre style="background-color: rgb(189, 211, 207);"><code style="background-color: rgb(189, 211, 207); color: black">@inproceedings{wani2020multion,
title={Multi-ON: Benchmarking Semantic Map Memory using Multi-Object Navigation},
author={Saim Wani and Shivansh Patel and Unnat Jain and Angel X. Chang and Manolis Savva},
booktitle={Neural Information Processing Systems (NeurIPS)},
year={2020},
}</code></pre>
If you used the proposed Baseline:
<pre style="background-color: rgb(189, 211, 207);"><code style="background-color: rgb(189, 211, 207); color: black">@inproceedings{raychaudhuri2024mopa,
title={MOPA: Modular Object Navigation With PointGoal Agents},
author={Raychaudhuri, Sonia and Campari, Tommaso and Jain, Unnat and Savva, Manolis and Chang, Angel X},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={5763--5773},
year={2024}
}</code></pre>
If you used the HSSD dataset:
<pre style="background-color: rgb(189, 211, 207);"><code style="background-color: rgb(189, 211, 207); color: black">@article{khanna2023habitat,
title={Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation},
author={Khanna, Mukul and Mao, Yongsen and Jiang, Hanxiao and Haresh, Sanjay and Schacklett, Brennan and Batra, Dhruv and Clegg, Alexander and Undersander, Eric and Chang, Angel X and Savva, Manolis},
journal={arXiv preprint arXiv:2306.11290},
year={2023}
}</code></pre>
</div>
</div>
</div>
</section>
<section class="organizers section text-left">
<div class="container">
<div class="organizers-inner section-inner">
<h2 class="section-title mt-0">Organizers</h2>
<div class="row" id="organizers-container">
<div class="organizers-column">
<a href="https://sonia-raychaudhuri.github.io/">
<img src="images/sonia.jpg" alt="Sonia Raychaudhuri" style="width:180px">
</a>
<div class="text-center">
<a href="https://sonia-raychaudhuri.github.io/">Sonia Raychaudhuri</a><br>
SFU
</div>
</div>
<div class="organizers-column">
<a href="http://hit.psy.unipd.it/enrico-cancelli">
<img src="images/enrico.png" alt="Enrico Cancelli" style="width:180px">
</a>
<div class="text-center">
<a href="http://hit.psy.unipd.it/enrico-cancelli">Enrico Cancelli</a><br>
U of Padova
</div>
</div>
<div class="organizers-column">
<a href="https://www.tommasocampari.com/">
<img src="images/campari.jpg" alt="Tommaso Campari" style="width:180px">
</a>
<div class="text-center">
<a href="https://www.tommasocampari.com/">Tommaso Campari</a><br>
FBK
</div>
</div>
<div class="organizers-column">
<a href="http://lambertoballan.net/">
<img src="images/lamberto.jpeg" alt="Lamberto Ballan" style="width:180px">
</a>
<div class="text-center">
<a href="http://lambertoballan.net/">Lamberto Ballan</a><br>
U of Padova
</div>
</div>
<div class="organizers-column">
<a href="https://angelxuanchang.github.io/">
<img src="images/angel.jpg" alt="Angel X. Chang" style="width:180px">
</a>
<div class="text-center">
<a href="https://angelxuanchang.github.io/">Angel X. Chang</a><br>
SFU
</div>
</div>
<div class="organizers-column">
<a href="https://msavva.github.io/">
<img src="images/manolis.jpg" alt="Manolis Savva" style="width:180px">
</a>
<div class="text-center">
<a href="https://msavva.github.io/">Manolis Savva</a><br>
SFU
</div>
</div>
</div>
</div>
</div>
</section>
</main>
</div>
<!-- Bootstrap core JavaScript
================================================== -->
<!-- Placed at the end of the document so the pages load faster -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/popper.js@1.12.9/dist/umd/popper.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
</body>
</html>