Support external backends

README.md

@@ -33,8 +33,9 @@ A triplestore wrapper is created with the [`tripper.Triplestore`][Triplestore] c
 
 Documentation
 -------------
-* Getting started: Take a look at the [tutorial](docs/tutorial.md).
-* Reference manual: [API Reference]
+* Getting started: See the [tutorial](docs/tutorial.md)
+* [Discovery of custom backends](docs/backend_discovery.md)
+* [Reference manual]
 
 
 Installation
@@ -59,7 +60,7 @@ SINTEF.
 
 [rdflib]: https://rdflib.readthedocs.io/en/stable/
 [PyPI]: https://pypi.org/project/tripper
-[API Reference]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/
+[Reference manual]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/
 [Literal]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/#tripper.triplestore.Literal
 [Namespace]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/#tripper.triplestore.Namespace
 [Triplestore]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/#tripper.triplestore.Triplestore

docs/backend_discovery.md

@@ -0,0 +1,73 @@
+Discovery of custom backends
+============================
+A tripper backend is a normal importable Python module that defines the class `{Name}Strategy`, where `{Name}` is the name of the backend with the first letter capitalized (as it would be with the `str.title()` method).
+The methods they are supposed to define are documented in [tripper/interface.py].
+
+Tripper support several use cases for discovery of custom backends.
+
+
+Installed backend package
+-------------------------
+It is possible to create a pip installable Python package that provides new tripper backends that will be automatically discovered.
+
+The backend package should add the following to its `pyproject.toml` file:
+
+```toml
+[project.entry-points."tripper.backends"]
+mybackend1 = "subpackage.mybackend1"
+mybackend2 = "subpackage.mybackend2"
+```
+
+When your package is installed, this would make `mybackend1` and `mybackend2` automatically discovarable by tripper, such that you can write
+
+```python
+>>> from tripper import Triplestore
+>>> ts = Triplestore(backend="mybackend1")
+```
+
+
+Backend module
+--------------
+If you have a tripper backend that is specific to your application, or that you for some other reason don't want or feel the need to publish as a separate Python package, you can keep the backend as a module within your application.
+
+In this case you have two options, either specify explicitly backend module when you instantiate your triplestore or append your package to the `tripper.backend_packages` module variable:
+
+
+### Instantiate triplestore with explicit module path
+An explicit module path can either be absolute or relative as shown in the example below:
+
+```
+# Absolute
+>>> ts = Triplestore(backend="mypackage.backends.mybackend")
+
+# Relative to the `package` argument
+>>> ts = Triplestore(backend="backends.mybackend", package="mypackage")
+```
+
+A backend is considered to be specified explicitly if the `backend` argument contain a dot (.) or if the `package` argument is provided.
+
+
+### Append to `tripper.backend_packages`
+Finally you can insert/append the sub-package with your backend to the `tripper.backend_packages` list module variable:
+
+```python
+import tripper
+tripper.backend_packages.append("mypackage.backends")
+ts = Triplestore(backend="mybackend")
+```
+
+
+Search order for backends
+-------------------------
+Tripper backends are looked up in the following order:
+1. explicit specified backend modules
+2. backend packages
+3. checking `tripper.backend_packages`
+
+By default the built-in backends are looked up as the first element in `tripper.backend_packages` (but it is possible for the insert a custom backend sub-package before it).
+This means that backend packages are looked up before the built-in backends.
+Hence it is possible for a backend package to overwrite or extend a built-in backend.
+
+
+
+[tripper/interface.py]: https://emmc-asbl.github.io/tripper/latest/api_reference/interface/

docs/index.md

@@ -33,8 +33,9 @@ A triplestore wrapper is created with the [`tripper.Triplestore`][Triplestore] c
 
 Documentation
 -------------
-* Getting started: Take a look at the [tutorial](tutorial.md).
-* Reference manual: [API Reference]
+* Getting started: See the [tutorial](tutorial.md)
+* [Discovery of custom backends](backend_discovery.md)
+* [Reference manual]
 
 
 Installation
@@ -59,7 +60,7 @@ SINTEF.
 
 [rdflib]: https://rdflib.readthedocs.io/en/stable/
 [PyPI]: https://pypi.org/project/tripper
-[API Reference]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/
+[Reference manual]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/
 [Literal]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/#tripper.triplestore.Literal
 [Namespace]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/#tripper.triplestore.Namespace
 [Triplestore]: https://emmc-asbl.github.io/tripper/latest/api_reference/triplestore/#tripper.triplestore.Triplestore

docs/planned-backends.md

@@ -3,9 +3,6 @@ Planned backends
 In addition to the currently existing backends, the following
 additional backends may be supported in upcoming versions:
 
-  - OntoRec/OntoFlowKB
-  - Stardog
-  - DLite triplestore (based on Redland librdf)
   - Redland librdf
   - Apache Jena Fuseki
   - Allegrograph

tripper/__init__.py

@@ -21,7 +21,7 @@
     XSD,
     Namespace,
 )
-from .triplestore import Triplestore
+from .triplestore import Triplestore, backend_packages
 
 __version__ = "0.2.3"
 
@@ -45,4 +45,5 @@
     "Namespace",
     #
     "Triplestore",
+    "backend_packages",
 )

tripper/triplestore.py

@@ -14,11 +14,12 @@
 # pylint: disable=invalid-name,too-many-public-methods
 from __future__ import annotations  # Support Python 3.7 (PEP 585)
 
+import importlib
 import inspect
 import re
+import sys
 import warnings
 from collections.abc import Sequence
-from importlib import import_module
 from typing import TYPE_CHECKING
 
 from tripper.errors import NamespaceError, TriplestoreError, UniquenessError
@@ -44,6 +45,16 @@
 
     from tripper.utils import OptionalTriple, Triple
 
+try:
+    from importlib.metadata import entry_points
+except ImportError:
+    # Use importlib_metadata backport for Python 3.6 and 3.7
+    from importlib_metadata import entry_points
+
+
+# Default packages in which to look for tripper backends
+backend_packages = ["tripper.backends"]
+
 
 # Regular expression matching a prefixed IRI
 _MATCH_PREFIXED_IRI = re.compile(r"^([a-z]+):([^/]{2}.*)$")
@@ -74,21 +85,33 @@ def __init__(
         backend: str,
         base_iri: "Optional[str]" = None,
         database: "Optional[str]" = None,
+        package: "Optional[str]" = None,
         **kwargs,
     ) -> None:
         """Initialise triplestore using the backend with the given name.
 
         Parameters:
             backend: Name of the backend module.
+
+                For built-in backends or backends provided via a
+                backend package (using entrypoints), this should just
+                be the name of the backend with no dots (ex: "rdflib").
+
+                For a custom backend, you can provide the full module name,
+                including the dots (ex:"mypackage.mybackend").  If `package`
+                is given, `backend` is interpreted relative to `package`
+                (ex: ..mybackend).
             base_iri: Base IRI used by the add_function() method when adding
                 new triples. May also be used by the backend.
-            database: Name of database to connect to (for backends that supports it).
+            database: Name of database to connect to (for backends that
+                supports it).
+            package: Required when `backend` is a relative module.  In that
+                case, it is relative to `package`.
             kwargs: Keyword arguments passed to the backend's __init__()
                 method.
+
         """
-        module = import_module(
-            backend if "." in backend else f"tripper.backends.{backend}"
-        )
+        module = self._load_backend(backend, package)
         cls = getattr(module, f"{backend.title()}Strategy")
         self.base_iri = base_iri
         self.namespaces: "Dict[str, Namespace]" = {}
@@ -102,6 +125,50 @@ def __init__(
         for prefix, namespace in self.default_namespaces.items():
             self.bind(prefix, namespace)
 
+    @classmethod
+    def _load_backend(cls, backend: str, package: "Optional[str]" = None):
+        """Load and return backend module.  The arguments has the same meaning
+        as corresponding arguments to __init__().
+
+        If `backend` contains a dot or `package` is given, import `backend` using
+        `package` for relative imports.
+
+        Otherwise, if there in the "tripper.backends" entry point group exists
+        an entry point who's name matches `backend`, then the corresponding module
+        is loaded.
+
+        Otherwise, look for the `backend` in any of the (sub)packages listed
+        `backend_packages` module variable.
+        """
+        # Explicitly specified backend
+        if "." in backend or package:
+            return importlib.import_module(backend, package)
+
+        # Installed backend package
+        if (3, 8) <= sys.version_info < (3, 10):
+            # Fallback for Python 3.8 and 3.9
+            eps = entry_points().get("tripper.backends", ())
+        else:
+            # New entry_point interface from Python 3.10+, which is also
+            # implemented in the importlib_metadata backport for Python 3.6
+            # and 3.7.
+            eps = entry_points(group="tripper.backends")
+        for entry_point in eps:
+            if entry_point.name == backend:
+                return importlib.import_module(entry_point.module)
+
+        # Backend module
+        for pack in backend_packages:
+            try:
+                return importlib.import_module(f"{pack}.{backend}")
+            except ModuleNotFoundError:
+                pass
+
+        raise ModuleNotFoundError(
+            "No tripper backend named '{backend}'",
+            name=backend,
+        )
+
     # Methods implemented by backend
     # ------------------------------
     def triples(  # pylint: disable=redefined-builtin
@@ -352,11 +419,9 @@ def list_databases(cls, backend: str, **kwargs):
     # implemented by all backends.
 
     @classmethod
-    def _get_backend(cls, backend: str):
+    def _get_backend(cls, backend: str, package: "Optional[str]" = None):
         """Returns the class implementing the given backend."""
-        module = import_module(
-            backend if "." in backend else f"tripper.backends.{backend}"
-        )
+        module = cls._load_backend(backend, package=package)
         return getattr(module, f"{backend.title()}Strategy")
 
     @classmethod