What does --middle-with-nodes do exactly?

[mboeringa]

I have been testing with the latest master, which includes the changes for the new 'middle', by setting both:

--middle-database-format=new

and

--middle-with-nodes

Now, the osm2pgsql manual is kind of brief and mildly ambiguous regarding the --middle-with-nodes option:

https://osm2pgsql.org/doc/manual.html#middle-command-line-options

Version >= 1.9.0 Used together with the new middle database format when a flat nodes file is used to force storing nodes with tags in the database, too.

Am I right that --middle-with-nodes will add any node that is part of a way or relation if is tagged to the 'planet_osm_point' table, where previous versions of osm2pgsql would simply ignore a tagged node if it was part of a way or relation? I vaguely remember reading about this change, but could not find the reference to this new option again.

If this is indeed the case, shouldn't this option be default if a user specifies the new 'middle' format? The main reason for not storing these nodes was a technical one if I understood it. Now that the technical challenge seems to have been overcome, it seems only logical to have this option on by default, I see little reason not to want to have this option on by default, it just makes the resulting data more complete.

This might even allow dropping the --middle-with-nodes command line option altogether from osm2pgsql, saving one extra command line option.

The only drawback might be that, as I witnessed, one may need extra storage, as I noticed osm2pgsql seems to create the --flat-nodes file and --middle-database-format=new specific 'planet_osm_nodes' table in parallel, while previously specifying --flat-nodes would prevent the creation of the 'planet_osm_nodes' table.

In terms of performance, the impact of specifying the 'new' 'middle' options, including --middle-with-nodes versus 'legacy' seems absolutely minimal. On Planet, it maybe just adds 15 minutes extra import time.

[joto]

I have updated the docs to make the description a bit clearer I hope. It is not as complex as you describe. It simply adds all nodes that have at least one tag to the planet_osm_nodes table even if a flat node file is used. Unlike the other data in the middle, this is not needed for osm2pgsql itself, it is only so that this data is there for users who want to do something special with the data.

Adding those nodes adds about 30 GB for the table plus 5 GB for the index to the database size. That's large enough that we don't want to do that by default.

[mboeringa]

@joto

Thanks for the explanation. However, do I understand a couple of things right now?:

• It doesn't matter whether you specify --middle-with-nodes or not, all nodes with tags will finally be in the 'planet_osm_point' table (well, I mean at least those selected by the style file).
• This option only makes sure you have a copy of the data in the 'planet_osm_nodes' table during import with --flat-nodes, so you can access it from within e.g. a flex LUA file to work with the data, whereas without this option, the data would be inaccessible to LUA because it would only be in the --flat-nodes file?

[joto]

@mboeringa The first one is right, the second one not quite. There is currently no way to get at the nodes data from Lua, you can only get at it using SQL queries in the database. But in the future we will probably have ways to access that data from Lua, through two-stage processing or so, but we are not quite there yet.