Use modern Python import features for product enumeration

[d-maurer]

Fixes #1239.

Currently, Zope only supports file based module finders during product enumeration (see OFS.Application.get_products).
This PR replaces Zope's proprietary product enumeration by pkgutil.iter_modules and thereby supports all module finder types supported by Python.

To reduce the risk for incompatibilities the signature for product handling functions in OFS.Application is mostly left unchanged: product_dir is replaced by finder (though most functions do not use this parameter in their body).

[dataflake]

The code looks fine and the tests run, but how do we really know if the new way to load products will produce the same results as the old way, meaning it won't lead to any products not being loaded?

[d-maurer]

Jens Vagelpohl wrote at 2024-11-27 01:42 -0800:

The code looks fine and the tests run, but how do we really know if the new way to load products will produce the same results as the old way, meaning it won't lead to any products not being loaded?

I trust that `pkgutil.iter_modules` does the right thing for file loaders. The original code iterated over `Products.__path__` and selected in the resulting folders a subfolder as product if it contained an `__init__.p[yco]`. It disregarded path elements which were not folders; therefore, it worked only for file loaders. `pgkutil.iter_modules` promises to return a `ModuleInfo` object for each top level module (or package) indicated by `__path__`. If the path element corresponds to a file loader and a subfolder contains an `__init__.p[yco]`, then this subfolder represents a package and therefore should result in the generation of a corresponding `ModuleInfo`. Thus, the new approach should lead to at least the products detected earlier. The new approach may find more products: those that do not use a file loader and those that are implemented via an implicit namespace package.

[dataflake]

Thanks for the explanation!

[dwt]

That looks about how I would have expected. Thanks al lot!