OSM Multilingual data directly from wiki (wikibase data items) #5647

nyurik · 2018-12-22T05:30:40Z

This completes the work by @bhousel (thanks for getting me started!)

Takes data directly from the Wikibase data items (OSM Wiki)
https://wiki.openstreetmap.org/wiki/OpenStreetMap:Data_Items
Understands the difference in regions - e.g. will show different
images depending on the local settings
Perf: Single request will get both the tag and key description
Slightly different wording to encourage description editing/translation

RU has a custom image	EN has no image

* Takes data directly from the Wikibase data items (OSM Wiki) https://wiki.openstreetmap.org/wiki/OpenStreetMap:Data_Items * Understands the difference in regions - e.g. will show different images depending on the local settings * Perf: Single request will get both the tag and key description

1ec5

Excited to see this project come together. 🎉

1ec5 · 2018-12-22T07:11:40Z

modules/services/osm_wikibase.js

+
+
+    /** List of data items representing language regions.
+     *  To regenerate, use Sophox query:  http://tinyurl.com/y6v9ne2c (every instance of Q6999)


This seems like a list that will grow over time, perhaps even as frequently as once per release cycle. Consider splitting it out into a separate file (JSON perhaps) that can be updated as part of iD’s build process.

Why can't the service just fetch these on init() ?

Does init() support async? In any case, this would be a fast-enough call to do on the first request to the service, and cache it. Patch is on its way...

The taginfo interface does a similar lookup on init() to fetch popular keys. Just try to make sure that if it fails iD will still work ok (maybe just without documentation lookups)

iD/modules/services/taginfo.js

Lines 202 to 208 in e51f2eb

this.keys(params, function(err, data) {

if (err) return;

data.forEach(function(d) {

if (d.value === 'opening_hours') return; // exception

_popularKeys[d.value] = true;

});

});

1ec5 · 2018-12-22T07:14:32Z

modules/services/osm_wikibase.js

+        gl: 'Q7793', hr: 'Q7794', ht: 'Q7795', hu: 'Q7796', id: 'Q7797', it: 'Q7798', ja: 'Q7799',
+        ko: 'Q7800', lt: 'Q7801', lv: 'Q7802', ms: 'Q7803', nl: 'Q7804', no: 'Q7805', pl: 'Q7806',
+        pt: 'Q7807', ro: 'Q7808', ru: 'Q7809', sk: 'Q7810', sq: 'Q7811', sv: 'Q7812', tr: 'Q7813',
+        uk: 'Q7814', vi: 'Q7815', yue: 'Q7816', 'zh-hans': 'Q7817', 'zh-hant': 'Q7818',


It would be nice to have all of iD’s current localizations represented here if possible.

I would actually want to shrink this list, not grow it :) The reason for this list is to support all the existing discrepancies in the wiki, e.g. status = in-use in some languages, and defacto in other. This makes very little sense to encourage such differentiation, so I created the smallest possible list (only the ones that actually conflicted)

1ec5 · 2018-12-22T07:18:25Z

modules/services/osm_wikibase.js

+     *  A less accurate list can be seen here (everything that links to Q6999):
+     *  https://wiki.openstreetmap.org/w/index.php?title=Special%3AWhatLinksHere&target=Item%3AQ6999&namespace=120
+     */
+    regionCodes: {


It looks like this is a list of locales, rather than regions, to use IETF language tag terminology. Other parts of iD use the term “locale” to refer to this concept, so I think we should be consistent here too.

modules/services/osm_wikibase.js

1ec5 · 2018-12-22T07:27:04Z

modules/ui/tag_reference.js

-        var locale = utilDetect().locale.toLowerCase();
-        var localized;
-
-        if (locale !== 'pt-br') {  // see #3776, prefer 'pt' over 'pt-br'


For the benefit of others reading this PR, I’m pretty sure the MediaWiki API handles #3776 and other language fallbacks implicitly.

modules/ui/tag_reference.js

1ec5 · 2018-12-22T07:50:46Z

data/core.yaml

@@ -418,6 +417,7 @@ en:
    choose: Select feature type
    results: "{n} results for {search}"
    reference: View on OpenStreetMap Wiki
+    edit_reference: Edit or translate on OSM Wiki


Is this string used anywhere? I see it in the screenshots above, but my browser’s in-page find isn’t finding any references to it.

Good catch! It got lost in a merge accident :)

bhousel · 2018-12-22T18:18:31Z

modules/services/osm_wikibase.js

+
+
+    /** List of data items representing language regions.
+     *  To regenerate, use Sophox query:  http://tinyurl.com/y6v9ne2c (every instance of Q6999)


Why can't the service just fetch these on init() ?

Locale IDs are now fetched together with the other data when the language code is provided.

pnorman · 2018-12-23T00:43:47Z

Does this have any implications for editing when the wiki is offline?

bhousel · 2018-12-23T00:56:46Z

Does this have any implications for editing when the wiki is offline?

Not really - the built in tag documentation already comes from the wiki. So this PR is just to request the documentation directly from the wikibase API rather than going through taginfo API for it.

bhousel · 2018-12-26T19:48:16Z

Thanks @nyurik - looks good to merge 👍

mmd-osm · 2018-12-29T15:28:27Z

Not sure if this was changed in this PR. When I click on "Edit or translate on OSM Wiki", it takes me right to the Wikibase page (e.g. https://wiki.openstreetmap.org/wiki/Item:Q409).

Assuming that the average iD user may not be a Wiki guru (I am not one, btw), the contents on the page may be disturbing to some users.

I'd rather expect a link to take me to a Wiki page with more human-readable explanations as to what the key is about (e.g. https://wiki.openstreetmap.org/wiki/Key%3Amaxheight).

bhousel · 2018-12-29T15:58:48Z

I'd rather expect a link to take me to a Wiki page with more human-readable explanations as to what the key is about (e.g. https://wiki.openstreetmap.org/wiki/Key%3Amaxheight).

@1ec5 @nyurik - should we also leave an OSM wiki sitelink there, if exists a suitable one?

I thought we were keeping them, but then again I'm pretty 👎 about the OSM wiki these days, so I'm ok with not including it. 😕

nyurik · 2018-12-29T16:10:05Z

I do think we should have both -- a link to add/fix/translate short descriptions, and a link to the wiki page if it exists (could be two separate lines). To make it simple to detect existence in a given language. I will need to populate the existence of the wiki page in the data items.

nyurik · 2018-12-29T16:14:16Z

P.S. example of the data item with translation wiki links https://wiki.openstreetmap.org/wiki/Item:Q104#P31

bhousel and others added 2 commits December 22, 2018 00:23

WIP on replacing documentation with direct calls to OSM wikibase

22e36be

1ec5 reviewed Dec 22, 2018

View reviewed changes

bhousel suggested changes Dec 22, 2018

View reviewed changes

nyurik added 2 commits December 22, 2018 14:58

Data items - remove hardcoded locale IDs

aa78cd1

Locale IDs are now fetched together with the other data when the language code is provided.

Use new message to encourage editing and translation

8b931f3

Fix language fallback processing, caching bug.

9f7f4aa

nyurik force-pushed the osm_wikibase branch from e774a23 to 9f7f4aa Compare December 23, 2018 06:57

This was referenced Dec 24, 2018

highway=crossing has wrong "unmarked crossing" description #5652

Closed

Check for disambiguation page in wiki info #4414

Open

All Waterareas have same "i"-Link #4064

Closed

bhousel approved these changes Dec 26, 2018

View reviewed changes

bhousel merged commit bc72c38 into openstreetmap:master Dec 26, 2018

nyurik deleted the osm_wikibase branch December 26, 2018 19:55

bhousel mentioned this pull request Jan 4, 2019

Use multilingual description and images from OSM Wikibase #5321

Closed

quincylvania mentioned this pull request Jan 23, 2019

Update to iD v2.13.0 openstreetmap/openstreetmap-website#2122

Merged

my-na-me mentioned this pull request Jan 24, 2019

Issues with OSM Wikibase links #5753

Closed

nyurik mentioned this pull request Mar 21, 2020

Include information from "Data items" geometalab/OSMTagFinder#7

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OSM Multilingual data directly from wiki (wikibase data items) #5647

OSM Multilingual data directly from wiki (wikibase data items) #5647

nyurik commented Dec 22, 2018

1ec5 left a comment

1ec5 Dec 22, 2018

bhousel Dec 22, 2018

nyurik Dec 22, 2018

bhousel Dec 22, 2018

1ec5 Dec 22, 2018

nyurik Dec 22, 2018

1ec5 Dec 22, 2018

1ec5 Dec 22, 2018

1ec5 Dec 22, 2018 •

edited

Loading

nyurik Dec 22, 2018

bhousel Dec 22, 2018

pnorman commented Dec 23, 2018

bhousel commented Dec 23, 2018

bhousel commented Dec 26, 2018

mmd-osm commented Dec 29, 2018

bhousel commented Dec 29, 2018

nyurik commented Dec 29, 2018 •

edited

Loading

nyurik commented Dec 29, 2018 •

edited

Loading



		/** List of data items representing language regions.
		* To regenerate, use Sophox query: http://tinyurl.com/y6v9ne2c (every instance of Q6999)

	this.keys(params, function(err, data) {
	if (err) return;
	data.forEach(function(d) {
	if (d.value === 'opening_hours') return; // exception
	_popularKeys[d.value] = true;
	});
	});

OSM Multilingual data directly from wiki (wikibase data items) #5647

OSM Multilingual data directly from wiki (wikibase data items) #5647

Conversation

nyurik commented Dec 22, 2018

1ec5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1ec5 Dec 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pnorman commented Dec 23, 2018

bhousel commented Dec 23, 2018

bhousel commented Dec 26, 2018

mmd-osm commented Dec 29, 2018

bhousel commented Dec 29, 2018

nyurik commented Dec 29, 2018 • edited Loading

nyurik commented Dec 29, 2018 • edited Loading

1ec5 Dec 22, 2018 •

edited

Loading

nyurik commented Dec 29, 2018 •

edited

Loading

nyurik commented Dec 29, 2018 •

edited

Loading