Cleanup of old data may result in incomplete ways

[Helium314]

related to #4980

I received a crash report for SCEE about a NullPointerException in

StreetComplete/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSource.kt

Line 415 in 7dbfc88

val nodes = getWayNodes(way)!!

which can only occur if not all nodes are available.
It's SCEE, but I'm quite sure the same crash can happen in StreetComplete too, so it should be fixed here.

My suspicion is that old nodes got cleaned (more than 14 days old), but the way and at least one node was still there. As far as I understand, this can happen when both were edited and uploaded. Then the edited elements will have an updated LAST_SYNC timestamp after upload, but the non-edited nodes will keep the old timestamp.

Details leading to this idea were provided by the user:

• Before the crash, the overlay was not shown, despite being active (i.e. no highlighted data)
• The crash came up repeatedly until the user downloaded the same area again, so I think the issue is not connected to the map data cache
• The user downloads data manually

How to Reproduce
I did not try to reproduce it, but I assume it could be reproduced as follows:

• download an area
• wait 7 days
• edit a way and a node of that way and upload the edits
• wait more than 7, but less than 14 days
• edit the way again
• do something that calls getMapDataWithGeometry with the edited node inside the bbox

[Helium314]

Maybe use something like

    fun filterNodeIdsWithoutWays(nodeIds: Collection<Long>): Collection<Long> {
        val idsString = nodeIds.joinToString(",")
        val nodeIdsWithWays = db.query(NAME_NODES, where = "$NODE_ID IN ($idsString)", columns = arrayOf(NODE_ID)) { c -> c.getLong(NODE_ID) }
        return nodeIds - existingNodeIds.toSet()
    }

after elementDB.getIdsOlderThan in MapDataController.deleteOlderThan?

[westnordost]

Hm, so the solution should be to not delete nodes regardless of their age as long as a way that is not old is still in the database.

[Helium314]

Yes, that was the idea: filter the to-be deleted nodes as proposed above, so only nodes that are not part of any way are deleted.
This requires changing deleteOlderThan a little bit, as such filtering needs to happen after deleting old ways.