Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Users with old osm editors ( Old-style OSM editor statistics: created_by | 2017 ) #27

Closed
ImreSamu opened this issue Mar 26, 2017 · 12 comments

Comments

@ImreSamu
Copy link

ImreSamu commented Mar 26, 2017

I have analyzed the 2017.01.01... - OSM history statistics for creating old-style multipolygons.

As I see some users still using very old OSM editors and creating old-style multi-polygons in 2017 ...

OSM Editor statistics ( created_by= ) for old-style multipolygons

Based on OSM History file ( 2017.01.01 -2017.03.20 ) filter for type=multipolygon + 2017
osmium cat planet.osh.pbf -t relation -f opl | grep " Ttype=multipolygon " | grep " v1 " | egrep " t2017-" + JOIN created_by= from changesets ..

Detailed data with links : https://gist.github.com/ImreSamu/ea6f81856ceaf6e9092469490239fdf1

FREQ:

   1589 Potlatch 2
    659 iD 2.0.2
    570 iD 2.1.3
    493 iD 2.1.2
     78 JOSM/1.5 (11427 en)
     70 JOSM/1.5 (11425 ru)
     62 JOSM/1.5 (11526 en)
     62 JOSM/1.5 (11427 fr)
     50 iD 2.1.0
     43 JOSM/1.5 (11223 en)
     39 JOSM/1.5 (9979 de)
     39 JOSM/1.5 (10966 fr)
     36 JOSM/1.5 (11639 ru)
     33 JOSM/1.5 (10327 es)
     30 JOSM/1.5 (11639 fr)
     30 JOSM/1.5 (11526 fr)
     30 JOSM/1.5 (11427 it)
     28 JOSM/1.5 (11639 pl)
     27 Merkaartor 0.18.3 (de)
     27 JOSM/1.5 (11427 de)
     26 JOSM/1.5 (11427 ru)
     21 JOSM/1.5 (11526 it)
     21 JOSM/1.5 (11526 de)
     20 JOSM/1.5 (11639 en)
     20 JOSM/1.5 (11223 Debian ru)
     20 JOSM/1.5 (11223 de)
     20 JOSM/1.5 (10966 ru)
     16 JOSM/1.5 (11425 en)
     16 JOSM/1.5 (10966 cs)
     15 JOSM/1.5 (11526 es)
     15 JOSM/1.5 (10966 en)
     14 JOSM/1.5 (11526 ru)
     14 JOSM/1.5 (10966 de)
     13 JOSM/1.5 (9329 Debian fr)
     13 JOSM/1.5 (11514 de)
     12 iD 2.1.1
     11 JOSM/1.5 (11639 it)
     10 JOSM/1.5 (11427 uk)
     10 JOSM/1.5 (10327 pl)
      9 JOSM/1.5 (11526 pl)
      9 JOSM/1.5 (11526 en_GB)
      9 JOSM/1.5 (11427 pl)
      9 JOSM/1.5 (11223 es)
      8 JOSM/1.5 (7643 uk)
      8 JOSM/1.5 (11639 es)
      7 JOSM/1.5 (7643 de)
      7 JOSM/1.5 (11639 en_GB)
      7 JOSM/1.5 (11639 de)
      7 JOSM/1.5 (10327 en_GB)
      7 iD 1.8.5-slide
      6 JOSM/1.5 (7643 en_GB)
      6 JOSM/1.5 (11427 es)
      6 JOSM/1.5 (11427 en_GB)
      6 JOSM/1.5 (11223 fr)
      6 JOSM/1.5 (11223 en_GB)
      6 JOSM/1.5 (10786 fr)
      5 JOSM/1.5 (11526 hu)
      5 JOSM/1.5 (11427 ja)
      5 JOSM/1.5 (11223 ja)
      5 JOSM/1.5 (10786 en_GB)
      4 Merkaartor 0.18.3 (ru)
      4 JOSM/1.5 (5267 uk)
      4 JOSM/1.5 (11526 ja)
      4 JOSM/1.5 (10786 en)
      3 Potlatch 2;JOSM/1.5 (10327 en)
      3 Merkaartor 0.18 (en)
      3 JOSM/1.5 (11639 sv)
      3 JOSM/1.5 (11526 id)
      3 JOSM/1.5 (11514 ru)
      3 JOSM/1.5 (11465 fr)
      3 JOSM/1.5 (11462 de)
      3 JOSM/1.5 (11427 pt_BR)
      3 JOSM/1.5 (11427 Debian en)
      3 JOSM/1.5 (11425 fr)
      3 JOSM/1.5 (11223 ru)
      3 JOSM/1.5 (10966 en_GB)
      2 Vespucci (F-Droid) 0.9.8.1.1228-fdroid
      2 Vespucci 0.9.8.1.1216
      2 Potlatch 2;JOSM/1.5 (11526 en)
      2 JOSM/1.5 (9329 fr)
      2 JOSM/1.5 (4878 de)
      2 JOSM/1.5 (11731 ru)
      2 JOSM/1.5 (11526 zh_CN)
      2 JOSM/1.5 (11526 pt_BR)
      2 JOSM/1.5 (11526 nl)
      2 JOSM/1.5 (11514 it)
      2 JOSM/1.5 (11514 fr)
      2 JOSM/1.5 (11514 es)
      2 JOSM/1.5 (11514 en)
      2 JOSM/1.5 (11488 de)
      2 JOSM/1.5 (11461 ru)
      2 JOSM/1.5 (11441 de)
      2 JOSM/1.5 (11405 fr)
      2 JOSM/1.5 (11223 zh_CN)
      2 JOSM/1.5 (11223 it)
      2 JOSM/1.5 (10327 en)
      2 JOSM/1.5 (10168 en_GB)
      1 Vespucci 0.9.8.1.1228
      1 reverter;JOSM/1.5 (11526 en)
      1 reverter;JOSM/1.5 (10327 pl)
      1 Merkaartor 0.18.3.31 (en)
      1 Merkaartor 0.18.3 (en)
      1 JOSM/1.5 (9329 Debian en)
      1 JOSM/1.5 (9060 en)
      1 JOSM/1.5 (7643 es)
      1 JOSM/1.5 (7287 sk)
      1 JOSM/1.5 (11714 en)
      1 JOSM/1.5 (11686 de)
      1 JOSM/1.5 (11652 de)
      1 JOSM/1.5 (11639 nl)
      1 JOSM/1.5 (11639 hu)
      1 JOSM/1.5 (11639 fi)
      1 JOSM/1.5 (11631 pt_BR)
      1 JOSM/1.5 (11598 ru)
      1 JOSM/1.5 (11582 uk)
      1 JOSM/1.5 (11568 ru)
      1 JOSM/1.5 (11568 fr)
      1 JOSM/1.5 (11548 ru)
      1 JOSM/1.5 (11541 en)
      1 JOSM/1.5 (11526 uk)
      1 JOSM/1.5 (11526 ca)
      1 JOSM/1.5 (11514 pt_BR)
      1 JOSM/1.5 (11514 ja)
      1 JOSM/1.5 (11512 da)
      1 JOSM/1.5 (11507 sv)
      1 JOSM/1.5 (11490 de)
      1 JOSM/1.5 (11463 sv)
      1 JOSM/1.5 (11463 de)
      1 JOSM/1.5 (11461 de)
      1 JOSM/1.5 (11429 sk)
      1 JOSM/1.5 (11429 ru)
      1 JOSM/1.5 (11427 zh_CN)
      1 JOSM/1.5 (11427 sv)
      1 JOSM/1.5 (11427 ca)
      1 JOSM/1.5 (11425 sv)
      1 JOSM/1.5 (11425 it)
      1 JOSM/1.5 (11425 hu)
      1 JOSM/1.5 (11425 de)
      1 JOSM/1.5 (11425 cs)
      1 JOSM/1.5 (11223 Debian en)
      1 JOSM/1.5 (11223 ca)
      1 JOSM/1.5 (10966 pt_BR)
      1 JOSM/1.5 (10786 pl)
      1 JOSM/1.5 (10786 Debian hu)
      1 JOSM/1.5 (10526 hu)
      1 JOSM/1.5 (10526 fr)
      1 JOSM/1.5 (10327 SVN ru)
      1 JOSM/1.5 (10327 fr)
      1 JOSM/1.5 (10168 fr)
      1 JOSM/1.5 (10168 en)
      1 
@joto
Copy link
Collaborator

joto commented Mar 28, 2017

Thanks @ImreSamu for looking at this. Potlatch2 is in the process of getting fixed to work with new style multipolygons. I don't think many people are running that outside the main osm.org page, so that is getting taken care of. It would be interesting to find out who is running old iD instances. The JOSM edits and others are probably to small to warrent any action.

I would expect this to sort itself out medium term after the osm2pgsql on the main OSMF servers is switched over to an upcoming new software version that doesn't handle old style multipolygons any more. At least it will not be worse than the many other problems our data has.

@ImreSamu
Copy link
Author

It would be interesting to find out who is running old iD instances.

Since (openstreetmap/iD#2449)
We have a

  • host=..... tag in the iD Editor Changesets, so we can detect the hosts ...
    • <tag k="host" v="http://strava.github.io/iD/"/>

example:

<changeset id="45313264" created_at="2017-01-19T23:08:36Z"   ....  >
  ....
  <tag k="locale" v="de-DE"/>
  <tag k="host" v="http://strava.github.io/iD/"/>
  <tag k="imagery_used" v="Bing;Strava Global Heat"/>
  <tag k="created_by" v="iD 1.8.5-slide"/>
</changeset>

I will try to create a statistics about this. ( about: live iD Editor hosts + iD Editor versions - in 2017 )

@ImreSamu
Copy link
Author

It would be interesting to find out who is running old iD instances.

Changesets>10 ( 2017.01.01 - 2017.03.22 )

  • probably the iD Editor 1.x.x is old.
  • sometimes the "host" tag is missing

Changesets FREQ +  iD Editor Host
-------------------------------------------    
 320377 https://www.openstreetmap.org/id # iD 2.0.2
 212558 http://www.openstreetmap.org/id # iD 2.0.2
 188491 https://www.openstreetmap.org/id # iD 2.1.3
 148849 https://www.openstreetmap.org/id # iD 2.1.2
 106355 http://www.openstreetmap.org/id # iD 2.1.2
 103819 http://www.openstreetmap.org/id # iD 2.1.3
  23211 https://www.openstreetmap.org/id # iD 2.1.0
  11760 http://www.openstreetmap.org/id # iD 2.1.0
   7155 https://www.openstreetmap.org/id # iD 2.1.1
   5992 http://www.openstreetmap.org/id # iD 2.1.1
   4130 http://strava.github.io/iD/ # iD 1.8.5-slide
   1574 undefined/id # iD 2.0.2
   1430 http://improveosm.org/ # iD 1.8.5
   1092 https://strava.github.io/iD/ # iD 1.8.5-slide
    420  # iD 1.6.2
    314 undefined/id # iD 2.1.3
    285 https://openstreetmap.us/iD/release/ # iD 2.0.2
    280 https://osm.naturalatlas.com/edit/ # iD 1.8.5
    259 http://localhost:8080/ # iD 2.0.1
    256 undefined/id # iD 2.1.2
    207  # iD 1.8.5
    194 http://projets.pavie.info/id-indoor/ # iD-indoor (iD 1.9.6)
    193  # iD 2.0.2
    190 http://editor.skitrails.info/ # iD 2.0.1
    123  # iD 1.8.1
    106  # iD 2.1.2
     94 http://id.softek.ir/ # iD 2.0.1
     64 https://openstreetmap.us/iD/release/ # iD 2.1.3
     64 http://id.loggingroads.org/ # iD 1.7.0
     61 https://id.opensidewalks.com/ # iD 2.0.0-alpha.1
     49 https://crowdid.osmcanada.com/ # iD 2.0.0
     40 https://openstreetmap.us/iD/release/ # iD 2.1.2
     39 https://www.juicytrails.com/id/index.html # iD 1.8.3-slide
     35 http://id.openstreetmap.us.s3-website-us-east-1.amazonaws.com/e0157f4/ # iD e0157f4
     29 http://localhost:8080/ # iD 2.1.3
     27 undefined/id # iD 2.1.0
     26 https://www.openstreetmap.org/id # iD 2.0.1
     14 http://localhost:8080/ # iD 2.1.2
     14 http://id.openstreetmap.us.s3-website-us-east-1.amazonaws.com/61bd8fc/index.html # iD 61bd8fc
     13  # iD 2.1.1
     12 http://192.168.1.27:8080/ # iD 2.0.1
     11  # iD 2.1.0
     11 http://www.openstreetmap.org/id # iD 2.0.1
     .....
    . 

full list with the long tail : https://gist.github.com/ImreSamu/fb9840c02212373552d84ac5a7bed709

@joto
Copy link
Collaborator

joto commented Mar 28, 2017

Maybe somebody wants to talk to the people running those iD instances and encouraging them to update. But it seems these sites are mostly for editing special things and they have changes in the software for them. So a) they might not be a big problem for our purposes here and b) they probably can't just update their software easily. So unless there is some evidence of massive problems from these sites, I think we can ignore this for now.

@tyrasd
Copy link
Member

tyrasd commented Mar 28, 2017

Are there specific problems with those old versions of iD regarding multipolgyon tagging? Or why are we talking about them here? Because, if I remember correctly, iD did always support the creation of properly tagged multipolygon relations. Legacy support for old style mps was and is still present in iD, but that shouldn't be of concern here, right? See also openstreetmap/iD#3908 (comment)

Or am I overlooking or recollecting something wrongly?

@ImreSamu
Copy link
Author

ImreSamu commented Mar 28, 2017

@tyrasd

Because, if I remember correctly,
iD did always support the creation of properly tagged multipolygon relations.
Legacy support for old style mps was and is still present in iD, but that shouldn't be of concern here, right?
Or am I overlooking or recollecting something wrongly?

please re-check my numbers and examples ..

in 2017 about a same number of old-style multipolygons created
with Potlach2 (Sum=1589 ) and iD (659+570+493 = 1722 )

   1589 Potlatch 2
    659 iD 2.0.2
    570 iD 2.1.3    <- Latest iD release ( on Feb 24 )
    493 iD 2.1.2    

full list: https://gist.github.com/ImreSamu/ea6f81856ceaf6e9092469490239fdf1 Search for iD - and check the relation first version. ( version="1" ) , so it is created with iD.
As I remember it is ordered , so See examples at the bottom, it is more refresh.

Imho: We have at least 1722 iD Editor examples for creating old-style polygons this year ( in the history, most of them corrected now )

5 examples:

@tyrasd
Copy link
Member

tyrasd commented Mar 28, 2017

@ImreSamu yes, I've read your posts. Btw: thanks for digging into this and providing the stats!

What I'm asking is: Is this a bug in (current and/or old versions of) iD that has to be fixed?[*] Or is it "just" mappers manually adding old-style multipolygons (like the also substantial number of 1171 old style multipolygons that have been added by JOSM users). If it's the former, a bug report to iD should be written (a fix could potentially also be backported to patch legacy versions of iD), but if it's the latter, we should ask what the reasons for this are: Maybe the users aren't aware of the deprecation of old style MPs, or they're following out of date documentation, etc.

PS: [*] I've tried to create multipolygons in iD (even in the 1.8.5 "strava slide" fork) and it works as expected: no old style multipolygons were created by using the default "merge areas" operation. The only way to actually create such an old stlye multipolygon from within iD (at least as far as I am aware of) is by manually creating an untagged multipolygon relation and manually adding the members one by one. Please correct me if I'm wrong. :)

@ImreSamu
Copy link
Author

@tyrasd

What I'm asking is: Is this a bug in ...
Or is it "just" mappers manually adding old-style multipolygons

honestly : I don't know .. I just checking the output.

but if it's the latter, we should ask what the reasons for this are:
Maybe the users aren't aware of the deprecation of old style MPs, or they're
following out of date documentation, etc.

or iD editor should add an extra validation step ? for detecting polygon problems/errors ...

My last example is a 5 role=outer -
imho : maybe it should be catch in the iD Editor side.

<osm version="0.6" generator="OpenStreetMap server" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
<relation id="7086042" changeset="46983054" timestamp="2017-03-19T13:53:45Z" version="1" visible="true" user="...." uid="494937">
  <member type="way" ref="481374655" role="outer"/>
  <member type="way" ref="481374653" role="outer"/>
  <member type="way" ref="481374656" role="outer"/>
  <member type="way" ref="481374657" role="outer"/>
  <member type="way" ref="481374658" role="outer"/>
  <tag k="type" v="multipolygon"/>
</relation>
<relation id="7086042" changeset="47104075" timestamp="2017-03-23T18:23:43Z" version="2" visible="false" user="...." uid="665748"/>
</osm>

@tyrasd
Copy link
Member

tyrasd commented Mar 28, 2017

iD editor should add an extra validation step ?

Very good point! I've opened the feature request in openstreetmap/iD#3933

@ImreSamu
Copy link
Author

Side notes for "post monitoring" topic

osmcha has an "Unknown iD instance" detection filter.

idea: maybe a new filter in osmcha for "Old-style multipolygons created" will be also useful, so we can easily monitor this info.

Bonus: the "fix old-style multipolygon" changesets comments can be monitor via osmcha:

@ImreSamu
Copy link
Author

ImreSamu commented May 9, 2017

2017-04

I have created a statistics for 2017-04 - a Fresh NEW ( v1 ) old_style polygons

osmium cat history-latest.osm.pbf -t relation -f opl | grep " Ttype=multipolygon " | grep " v1 " | egrep " t2017-04" + created_by -> old_style_created.opl

And a simple statistics FREQ by changeset editor ( created_by )

  • 2017-04-0x ( 2017-04-01 --- 2017-04-09 )
  • 2017-04-1x ( 2017-04-10 --- 2017-04-19 )
  • 2017-04-2x ( 2017-04-20 --- 2017-04-29 )
$ cat  old_style_created.opl| grep 2017-04-0  | cut -d' ' -f10  | sed -e 's/%20/ /' | cut -d' ' -f1 | sort |  uniq -c |  sort -nr
    114 changeset_created_by=JOSM/1.5
    102 changeset_created_by=iD
     57 changeset_created_by=Potlatch
      5 changeset_created_by=Vespucci
      2 changeset_created_by=
      1 changeset_created_by=Merkaartor
$ cat  old_style_created.opl| grep 2017-04-1  | cut -d' ' -f10  | sed -e 's/%20/ /' | cut -d' ' -f1 | sort |  uniq -c |  sort -nr
    220 changeset_created_by=JOSM/1.5
    200 changeset_created_by=iD
     93 changeset_created_by=Potlatch
      3 changeset_created_by=Level0
      1 changeset_created_by=Vespucci
$ cat  old_style_created.opl| grep 2017-04-2  | cut -d' ' -f10  | sed -e 's/%20/ /' | cut -d' ' -f1 | sort |  uniq -c |  sort -nr
    399 changeset_created_by=JOSM/1.5
    267 changeset_created_by=iD
     42 changeset_created_by=Potlatch
      1 changeset_created_by=Vespucci
      1 changeset_created_by=reverter;JOSM/1.5
      1 changeset_created_by=Merkaartor

JOSM versions for 2017-04-2x

$cat  old_style_created.opl| grep JOSM | grep 2017-04-2  | cut -d' ' -f10  | sed -e "s#%20%# #g" | cut -d' ' -f1-2 | sort |  uniq -c |  sort -nr
    274 changeset_created_by=JOSM/1.5 (11826
     74 changeset_created_by=JOSM/1.5 (11639
     16 changeset_created_by=JOSM/1.5 (9979
     10 changeset_created_by=JOSM/1.5 (11833
      5 changeset_created_by=JOSM/1.5 (11223
      5 changeset_created_by=JOSM/1.5 (10966
      3 changeset_created_by=JOSM/1.5 (11427
      2 changeset_created_by=JOSM/1.5 (12015
      2 changeset_created_by=JOSM/1.5 (11915
      2 changeset_created_by=JOSM/1.5 (11526
      1 changeset_created_by=reverter;JOSM/1.5 (11826
      1 changeset_created_by=JOSM/1.5 (12012
      1 changeset_created_by=JOSM/1.5 (12006
      1 changeset_created_by=JOSM/1.5 (11828
      1 changeset_created_by=JOSM/1.5 (11514
      1 changeset_created_by=JOSM/1.5 (10786
      1 changeset_created_by=JOSM/1.5 (10168

TOP10 users (with manual Data anonymization for privacy -> uAnonymized1 ) ( 2017-04- )
Like a Pareto principle
The top1 user created 279 old-style polygon in April.

$cat  old_style_created.opl| grep JOSM | grep 2017-04  | cut -d' ' -f7 |  sort |  uniq -c |  sort -nr | head 
    279 uAnonymized1
     60 uAnonymized2
     40 uAnonymized3
     17 uAnonymized4
     16 uAnonymized5
     15 uAnonymized6
     15 uAnonymized7
     14 uAnonymized8
     14 uAnonymized9
     13 uAnonymized10

@ImreSamu
Copy link
Author

ImreSamu commented Jul 4, 2017

2017-06

OSM history statistics for creating old-style multipolygons.

  • 2017-06-01 ... 2017-06-25 (25day)
  • osmium cat history-latest.osm.pbf -t relation -f opl | grep " Ttype=multipolygon " | grep " v1 " | egrep " t2017-06" + created_by -> old_type201706.opl
$ cat old_type201706.opl | wc -l
1195

$ cat old_type201706.opl | cut -d' ' -f 10 |  sed -e "s#%20%# #g" | cut -d' ' -f 1 |  sort  | uniq -c | sort -nr
    695 changeset_created_by=JOSM/1.5
    395 changeset_created_by=iD
    103 changeset_created_by=Potlatch
      2 changeset_created_by=

# TOP15
$ cat old_type201706.opl | cut -d' ' -f 10 | sort  | uniq -c | sort -nr | sed -e "s#%20%# #g" | head -15
    506 changeset_created_by=JOSM/1.5 (12275 en)
    242 changeset_created_by=iD 2.2.1
    150 changeset_created_by=iD 2.2.2
    103 changeset_created_by=Potlatch 2
     21 changeset_created_by=JOSM/1.5 (12275 ru)
     20 changeset_created_by=JOSM/1.5 (11826 ru)
     15 changeset_created_by=JOSM/1.5 (12409 en_AU)
     12 changeset_created_by=JOSM/1.5 (11826 en)
      9 changeset_created_by=JOSM/1.5 (12275 en_GB)
      8 changeset_created_by=JOSM/1.5 (12275 de)
      6 changeset_created_by=JOSM/1.5 (12275 fr)
      6 changeset_created_by=JOSM/1.5 (12271 hu)
      6 changeset_created_by=JOSM/1.5 (11223 Debian es)
      5 changeset_created_by=JOSM/1.5 (12275 it)
      5 changeset_created_by=JOSM/1.5 (12275 es)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants