Extract surface and smoothness example image list in shared repository

[tordans]

I really like the work that is done in #3257. I think it will improve the data in OSM a lot. And I want to promote the list of example images to other apps as well.

For that reasons, it would be great to extract the information in a share project that can be used by other apps more easily. Similar to what NSI, ELI and id-tagging-schema does, but on a much smaller scale.

To give an idea of what that could look like IMO, I create this gist:
https://gist.github.com/tordans/c0ba3935575864235c932ee6f5a65510

Its based on the table from #3257 (comment).

---

Here are a few projects that could use this data:

• MapComplete in a separate theme Smoothness theme based on surface-tags pietervdvn/MapComplete#545
• GoMap could more easily extract and show the images, similar to what wtimme prototyped in Show example images for "bicycle_parking" bryceco/GoMap#425 (comment)
• iD could use the data as well.
• Create a website based on the JSON that allows to easily look at the data, similar to the temporary spreadsheet. Which would be something to link to from a wiki page as well.

---

A few thinks that we should think about

• Versioning and release process of this file
• Should this file already think about allowing for regional images vs. the default image? Eg. other gravel images in countries with mainly mountain roads?…
• What would a good place for this repo be? github.com/osmlabs? github.com/ideditor?, right here in streetcomplete?

[Helium314]

If you are interested in the original files and licenses you can have a look at https://github.com/Helium314/StreetComplete/blob/656aa329646749e569169182773cbb425c6839da/app/src/main/res/authors.txt#L164-L186
(I'm not happy with some of the current images but that's not the point here...)

[matkoniecz]

general:

1. is there anyone interested in using this data, except StreetComplete? I commented for now on Smoothness theme based on surface-tags pietervdvn/MapComplete#545 (comment) (other potential users also should be asked to judge is there benefit from increased complexity). If there would be multiple projects using this then it would make sense, if it would be just StreetComplete I am more skeptical about increasing complexity

2. how complex would be pulling this data into StreetComplete, instead of storing it in usual way? Are you interested in writing necessary code for that (likely as gradle tasks)

3. who would be controlling and maintaining this new repository? Who would be added as backup owner? (if repo can have multiple owners)

about file format:

I would store images itself and just have links/info about original sources. This way fetching third part links of already external repository and we would not be vulnerable to renames of files on external sites and everything relevant would be fetched by cloning repository

[pietervdvn]

@tordans Instead of setting up new infrastructure to do this, why not reuse the wikidata/wikibase setup that is powering the OSM-wiki?

[matkoniecz]

@pietervdvn

wikidata/wikibase setup is suffering from lack of review of changes - it is not even possible to effectively watchlist data items (if you try you will get notifications about label changes in all languages, this makes watchlisting ineffective unlike for wiki pages).

And in general data quality and edit quality there is not very high.

If it would be used I am not going to neither maintain related data items, nor review edits made there (what would be necessary before each release) and would be against using it in StreetComplete.

inability to store files itself is a lesser issue, there is also problem with fact that there is no policy which files should be preferred - and in some cases preferred image between different use cases would be different

very minor issue is that wikibase API is really annoying (I still have not figured out how to parse for example P46 property "combination", see say https://wiki.openstreetmap.org/wiki/Item:Q5007). Editor is heavily JS based and it lags even on modern laptop. Android emulation + Android Studio is more responsive than data item editor.

Yes, here I am old man yelling at clouds data items - I tried data items and I dislike them heavily for reasons mentioned above and also some other. Maybe I am wrong here but I dislike idea of using data items in any way for anything.

[tordans]

Instead of setting up new infrastructure to do this, why not reuse the wikidata/wikibase setup that is powering the OSM-wiki?

(I first answered in pietervdvn/MapComplete#545 (comment), but lets keep the thread here.)

Wikidata is awesome, but not for complex presets and structures this.

• I am not aware of a way to model the data structure of this json in Wikidata items. But I will be happy to see it working in an example. It might be possible with a good collaboration with the Wikidata admins, that create new properties and qualifiers. – I looked into it a while back and did not get very far even after quite an invest of time.
• Wikidata has no review process. Which is great for some UseCases, but the complex and long (and good + important) discussion of the SC thread shows, that it's not about adding some/any image, but a great one. A PR->Review-Flow is pretty good for this, like NSI, ELI and id-tagging-schema shows.
• An external, shared repository can provide a standard release process based on semantic versioning and – for the node ecosystem npm updates. The issues that iD has with ELI compared to the other external resources that do provide releases and semantic versions show how important that is for a hassle free integration of external data.

---

Update: Just to clarify, I personally disagree with matkoniecz' general dislike of Wikidata. It would love to see it used more in other situations. But I don't see it working here. — But let's try to stay on topic of the issue at hand :-).

[matkoniecz]

I am not aware of a way to model the data structure of this json in Wikidata items. But I will be happy to see it working in an example

It can be handled by adding P28 to items representing given values, see say https://wiki.openstreetmap.org/wiki/Item:Q21492 (I just added it to demonstrate how it can be done)

no review process. (...) A PR->Review-Flow is pretty good for this, like NSI, ELI and id-tagging-schema shows.

That is also my fundamental objection (not my personal dislike that may be misguided)

Just to clarify, I personally disagree with matkoniecz' general dislike of Wikidata.

Note that Wikidata is a different project ( https://www.wikidata.org/ - excellent for storing interwiki links between Wikipedia articles in different languages, quite good for unique ids assigned to notable objects).

OSM data items are using Wikibase software powering it.

[tordans]

Re #3592 (comment)

The effort that it would make to pull this data in SC is certainly a factor. Should it be too complex, this project should be postponed. I cannot even make a guess about this nor would I be able to help develop this part.

However, for my own projects I would need to extract the data from SC anyways, so we could just as well do it together. The MC theme I linked would need this data, even if it does not get pull automatically in MC but is just used when developing the theme.

And a web- or wiki-overview table of the images use in SC would help increase adoption (even when looking up tags manually) and acceptance of the SC work IMO.

GoMap already is happy to pull good presets and data from iD and SC. So if the information is accessible easily, that would help a lot to get the conversation about an integration going. – The later is true for iD as well IMO.

Repo ownership: I don't know, yet (see my list of possible repo holding organisations). But I don't expect this small piece of shared tagging knowledge to be changing too often – its a lot simpler than ELI or id-tagging-schema.

[matkoniecz]

I added "feedback required" as

is there anyone interested in using this data, except StreetComplete?

remains open. It is not extremely complex but it is noticeable more complex and doing it just in case seems a poor idea.

[tordans]

It is not extremely complex but it is noticeable more complex and doing it just in case seems a poor idea.

Just to clarify: I will do it for myself to create a webpage (and probably wiki page) of the mapping table for reference outside of SC. And to extract the data for a MC-theme.

I would prefer we joined forces on this so I don't have to monitor if changes in SC are relevant to this list. (However, I read the release notes, so I would likely be informed, so that might be fine.)

[tordans]

Offtopic: Wikidata vs Wikibase

Note that Wikidata is a different project ( https://www.wikidata.org/ - excellent for storing interwiki links between Wikipedia articles in different languages, quite good for unique ids assigned to notable objects).
OSM data items are using Wikibase software powering it.

That is very confusing to me. The tag we use is wikidata=<Data Item Q123> (and that wiki page does not even mention Wikibase). And our own osm wiki pages use the term Wikidata more than they use Wikibase (eg the Data item page with 7x 'wikibase' and 11x 'wikidata'). And the page I use to view wikipedia-data-items is named Wikidata, eg. https://www.wikidata.org/wiki/Q2013.

But not to get too far off topic here, let's just clarify for this thread, that we are talking about those data structures https://wiki.openstreetmap.org/wiki/Item:Q746 (Data Item 'surface')

[matkoniecz]

Just to clarify: I will do it for myself to create a webpage (and probably wiki page) of the mapping table for reference outside of SC. And to extract the data for a MC-theme.

In such case there is an actual user, great!

That is very confusing to me.

I created https://wiki.openstreetmap.org/wiki/Wikibase in attempt to explain it. If it is still confusing - can you write at https://wiki.openstreetmap.org/w/index.php?title=Talk:Wikibase&action=edit ? (requires wiki account)

[pietervdvn]

In such case there is an actual user, great!

And if this implies an (automatic) translation from StreetComplete into MapComplete, I'd be a user too!

Very thrilled to see the outcome, especially if it opens up the StreetComplete-themes...

[matkoniecz]

And if this implies an (automatic) translation from StreetComplete into MapComplete, I'd be a user too!

This can be done already by parsing language files (XML in predictable locations so relatively easy, may be useful already for surface names).

For how this idea can be implemented I see following options:

(A) this new project has all this image data, StreetComplete would fetch credits, amend authors file (replacing part between lines --here credits from smoothness library start and --here credits from smoothness library end) and fetch and resize images. The problem is that it would require script to resize images - either in pure Kotlin or relying on something installed on system (and supporting just Linux is likely not OK)

(B) this new project would contain already resized images, including all sizes needed by StreetComplete. So resizing would not be needed

(C) invert idea: it would be script that extracts StreetComplete content making it more structured and easier to reuse (for example parsing authors file to get image links, parsing translation file and producing nice json). No additional complexity in StreetComplete, translations would be included, any contributions would need to go through StreetComplete. For parsing authors file I actually have Kotlin code doing this but it would be a bit fragile and acts as detector of any missing authorship information.

The largest negative would be that building smoothness example library could not be done step by step (unless StreetComplete would accept images to be used by not yet implemented quest).

[westnordost]

You are right, all the effort and contributions that have flown into the implementation of #3617 should not only profit StreetComplete. After all, along the way, much more example photos were searched and created that didn't make it into the quest because after all, only one picture is displayed for each combination of smoothness and surface. We don't have this limitation in the wiki.

So alright, I went through the whole PR and its contributions again, uploaded all the pictures I found most representative to the openstreetmap wiki that weren't yet and weren't yet on wikimedia commons and created this page:

https://wiki.openstreetmap.org/wiki/Key:smoothness/Gallery

It also includes the source pictures chosen in the end for #3617, but not only

[westnordost]

And in case you want to use exactly the same pictures as StreetComplete for MapComplete, @tordans already created that spreadsheet https://docs.google.com/spreadsheets/d/1-JiRgPPSByqyt7qQagFYJ3O7OvVh2DTJY_2-x9oeExs/edit#gid=0 in which I also added the descriptions for each.

[tordans]

Thanks a lot @westnordost for this work. I still think it helpful to extract the data from SC/the Google Doc to a shared repository with a JSON file in it (see gist) and will likely do myself in the near future. Having the images in the Wiki is a great solution IMO that we can link to from there. Feedback on the JSON-/repo-structure would be very helpful.
I would then archive the Google doc and generate wiki + html tables based on that data.
It would be ideal, if we kept it in sync with the code and wording here. I think manually at first.
I will ask for a repo at https://github.com/osmlab.