Remove Non-Native Processing Providers

[esnyder-rve]

QGIS Enhancement: Remove Non-Native Processing Providers

Date 2021/05/20

Author Ethan Snyder (@esnyder-rve)

Contact ethan dot snyder at rve dot com

Version QGIS 4

Summary

Frequently, there are issues with the external processing providers included in QGIS regardless of whether or not they are enabled. Thus, the processing plugin itself fails to load, leaving the users with nothing. This proposal is to remove the GRASS, SAGA and OTB providers from QGIS internally and to move them into downloadable plugins.

This does not affect or include the GDAL processing tools as GDAL is a dependency for QGIS.

Proposed Solution

Move GRASS, SAGA, and OTB tools to a plugin. (Hmm.. looks like part of that is done: https://plugins.qgis.org/plugins/processing_saga_nextgen/ and https://plugins.bruy.me/processing-saga.html for SAGA 7, not SAGA 2)

Also, since QGIS depends on GDAL, can we merge the GDAL tools in with the Native QGIS ones in the processing toolbox? The internal name can still remain "gdal:contour", but instead of being in a GDAL folder, put the contour tool in the Raster Terrain Analysis folder.

Affected Files

I don't know if there's anything special with these providers.

Performance Implications

This could actually speed up start-up times if the user never uses/needs SAGA, GRASS OTB tools.

Further Considerations/Improvements

There are several processing tool providers available as plugins: R, Whitebox, Fusion, QNeat3, TauDEM and LAStools to name a few. Since the official SAGA provider only supports SAGA 2, while the other plugins I referenced earlier support SAGA 7, it would make sense to deprecate this internal one (maybe ask the SAGA devs to maintain an "official" one?).

Backwards Compatibility

None needed.

Votes

(required)

EDITS

• Changed proposed QGIS version per gioman's comment regarding this as a target for QGIS 4.
• Added a note regarding the GDAL processing tools/algorithms.

[gioman]

@esnyder-rve this is already the more or less official plan (I guess for QGIS 4). My main concern is about the moment we will ship QGIS without those providers, because I think that by that moment we (QGIS) must guarantee that proper "official" plugins for SAGA and GRASS are available. I know there are reservations about making those providers as "official" plugins (meaning maintained by QGIS), which I understand. Nevertheless I think we should avoid duplication of efforts to avoid waste of resources and avoid confusion among users. For this reason i already gave (in some preliminary private conversation) gave my availability to support the work to move GRASS to a plugin. I guess for SAGA we should merge one of the already available plugins(?) and find someone available to maintain it (or support the work to). @alexbruy @nyalldawson

[esnyder-rve]

Why must QGIS guarantee official plugins for SAGA and GRASS? The purpose for this is to not make the Processing Plugin dependent on SAGA or GRASS. If, for whatever reason, something with SAGA or GRASS breaks, the whole Processing Plugin fails. Moving these outside of the Processing Plugin and into their own respective ones removes this (what I think is unnecessary) dependency.

[gioman]

Why must QGIS guarantee official plugins for SAGA and GRASS?

@esnyder-rve you have misread my answer. I didn't say QGIS should do it. I said that we should guarantee that from the moment we ship QGIS without those providers there should be a plugin replacing them. At the same time we should try avoid that >1 person or entity to do the same plugin for a specific provider, that is a waste of resources and will be a source of a lot of confusion for the users.

[esnyder-rve]

@esnyder-rve you have misread my answer. I didn't say QGIS should do it. I said that we should guarantee that from the moment we ship QGIS without those providers there should be a plugin replacing them.

@gioman Gotcha. I agree that before SAGA and GRASS get pulled, there should be a plugin ready to replace it. Don't really want to leave people high and dry.

[alexbruy]

I think most of the devs who are working on the Processing stuff, already reached an agreement to drop at least most of the 3rd party providers from Processing. However there are some opposition from other people and even new provider slipped in under their pressure (OTB, I'm looking at you now). Some of these providers almost not maintained and even caused CI failures time from time.

Personally I fully support removal of SAGA, GRASS and OTB from the QGIS core. Interested parties can maintain them as 3rd party provider plugins. Converting existing provider code into a plugin is quite easy task and in case this QEP adopted I will be happy to convert these providers into plugins to make it easier for community to pick them up.

As for porting GDAL algs to core, I started to work on it some time ago but had to put it aside because of lack of time and resources.

[esnyder-rve]

As for porting GDAL algs to core, I started to work on it some time ago but had to put it aside because of lack of time and resources.

@alexbruy To clarify what I said about moving GDAL tools into internal, I didn't mean porting. I would just hide the provider from the processing settings and distribute the tools in the appropriate QGIS Processing Tool folders and treat them as if they were internal.

[nyalldawson]

Thus, the processing plugin itself fails to load, leaving the users with nothing. This proposal is to remove the GRASS, SAGA and OTB providers from QGIS internally and to move them into downloadable plugins.

Just a note -- if the issue is the main processing plugin failing to load, then there's a potential solution which doesn't involve completely removing grass/otb/saga from QGIS. What I think we should do in the short term is move these three providers to be their own "core" plugins. I.e. they are included in the standard QGIS install, and are part of the main QGIS codebase, but are "sandboxed" as their own plugins which are loaded after the processing plugin. That way if they have issues during loading it will only be the one provider which is disabled, not the whole processing framework.

This would also be a good step toward forking them off into 3rd-party, non-official plugins, as they'd then be standalone plugins already...

[ytanguy]

Hello,
Speaking for OTB community, I think more and more users launch OTB applications through QGIS : we recently improved the plugin to ease the use of some applications (thank you troopa81 !).
Up to now, it was quite comfortable for us to have an official plugin in QGIS. But I understand your concerns and I know it's a hard-task to maintain properly this plugin wrt OTB latest versions.
I like the idea of having the plugin in its own sandbox. On our side, we have to improve plugin testing through our release process.

[ar-siddiqui]

Also, since QGIS depends on GDAL, can we merge the GDAL tools in with the Native QGIS ones in the processing toolbox? The internal name can still remain "gdal:contour", but instead of being in a GDAL folder, put the contour tool in the Raster Terrain Analysis folder.

On a personal level, I do like that GDAL functions are grouped separately, it let me know what is going on behind the scene and I can manipulate the input parameters easily.

[ar-siddiqui]

Sorry in advance if this is not the right place to ask.

I am developing a processing plugin for NOAA. I need to have some awareness that if I build a processing plugin that depends on GRASS and GDAL tools, will it be future proof with all the changes suggested here?

How do QGIS plans to handle current plugins that depend on GRASS/SAGA/GDAL tools, in the future, if/when these tools will be moved to thrid party plugins.

[esnyder-rve]

@ar-siddiqui

How do QGIS plans to handle current plugins that depend on GRASS/SAGA/GDAL tools, in the future, if/when these tools will be moved to third party plugins.

Currently, QGIS doesn't handle dependencies within plugins (as far as I know). It is up to the plugin author to state the dependencies in the plugin description, and the user to make sure they're satisfied.

I wouldn't worry about GDAL since QGIS depends on GDAL. I know running GRASS within QGIS has been funky. Depending on what you're doing and what you need, I'm not sure if implementing the tools you need yourself would be an option (GRASS is open source, so you can easily grab the source for the tools you need).

[ar-siddiqui]

@esnyder-rve

Currently, QGIS doesn't handle dependencies within plugins (as far as I know). It is up to the plugin author to state the dependencies in the plugin description, and the user to make sure they're satisfied.

I understand, my worry is that if a plugin is doing something like this:

outputs['Ibiomass'] = processing.run('grass7:i.biomass', alg_params, context=context, feedback=feedback, is_child_algorithm=True)

or this

Will QGIS will make sure that this will still work if the user has the grass7 plugin installed. Otherwise, everyone who has plugins, scripts, and models will have to update their things.

[esnyder-rve]

I suspect it would still work.

[esnyder-rve]

Can this QEP be closed?
https://changelog.qgis.org/en/qgis/version/3.22/#move-non-native-processing-providers-into-independ

[esnyder-rve]

Ping @alexbruy