16-bit unsigned integer output of stream_link_identifier hindering creation of large stream networks

[kmsampson]

Thank you for these amazing tools. I am building a workflow for creating a vector hydro network with unique reach IDs. The process (in python) is as follows:

wbt.stream_link_identifier(fdir, strm, stream_id, esri_pntr=True, zero_background=False)
wbt.raster_streams_to_vector(stream_id, fdir, streams_vector, esri_pntr=True)

The problem is that I have a large domain with many stream reaches, more than 33,000. I see that the output of the ‘stream_link_identifier’ tool is a 16-bit signed integer raster. Because of the number of individual stream reaches, there are both positive and negative values in the output of ‘stream_link_identifier’. All reaches currently receive an ID value, but some are negative. However, when using the output of ‘stream_link_identifier’ as input to ‘raster_streams_to_vector’, only the reaches with a positive ID value (ID > 0.0) are converted to stream vectors.

Could a solution be as simple as to allow the output of ‘stream_link_identifier’ to be a 32-bit integer, so that no negative stream ID values need to be used to resolve all reach IDs? This would also allow for much larger stream networks to be derived.

I have attached a small test-case that illustrates my example, with input files, a small python script, and outputs of the above code.
Whitebox_Issue.zip

[jblindsay]

Yes, the fix for this should be as easy as updating the output raster type to an I32 instead of the I16. I have just committed code to deal with this. The hard part for me, is that I will undoubtedly have someone else complain that the output of this tool is now twice the size as it was before and request that it output to I16 instead. However, I personally think that have a larger limit on the possible number of streams is more important than file size.

[kmsampson]

I agree with you that the ability to create larger hydro networks is worth the file-size increase in the output. Users can always compress the outputs or convert to a smaller data type if necessary. Would it be possible to alter the code to 'count' the number of unique values in the output and then select the appropriate output data type (i16 or i32)?

[jblindsay]

Absolutely @katrinleinweber, that's a good idea. In fact, I'm not sure of why I didn't think of that myself. Anyhow, I've just updated the codebase to include this feature. It now tests whether the number of stream links is less than 32767 and if so outputs an i16, if not an i32 formatted raster. Thanks for the suggestion.

[kmsampson]

Thanks for the consideration of this enhancement request and for building this excellent toolset!