Better biosphere install/update handling (#1145)

* Delete .github/workflows/install-canary.yaml

* Add testing file

* Get available AB versions per AB version from online file

* Limit ecoinvent importer to versions compatible with this AB version

* Implement choosing of version during biosphere update

* Implement choosing of version during defaults import

* Resolve offline bug + better updating

* move compatible versions file

* implement local file fallback

* implement legacy biosphere imports for any chosen version AB is compatible with

* update test

* update test

* update test

* update test

* update test

* update test

* resolve calculation setup bug where activites are checked when biosphere was updated.

* minor updates to test documentation

* GH action script for commenting on issues when related milestone is closed.

* update bw2io requirement

* Update readme install instructions

* Update meta.yaml

* Update README.md

* Change requirement to brightway2 instead of bw2io

* Update README.md

* Increases the flexibility of the excel importer so it can also handle the parameter scenario files.

* Update README.md

How to install using Mamba

* Set up signals infrascructure

* Implement dialog

* Implement finding of suitable alternatives and proper management of failing that.

* Implement actual relinking

* Minor documentation improvements + store exchanges to remove instead of to keep.

* Minor documentation + code improvements

* Resolve #782 + improve documentation of `Contributions` class

* Minor documentation + code improvements

* Fix type hinting error

* Update README.md

* Update README.md

* Update README.md

* Updates to the functions get_relevant_flows and get_relevant_activities (#1069)

* Updates to the functions from de Koning (get_relevant_flows and get_relevant_activities), avoiding use of pandas apply and using python map functionality for splitting pandas dataframes.

* Update the use of DataFrame.applymap to DataFrame.map in the excel file importer module.

* Multiple sdf update (#1083)

* Updates to the logging system to avoid the print statement. Merging with
commits for corrections to the uncertainty distributions that were
included within the same branch. Improves thread safety in logging
increasing stability in the multi-threaded processes.

* Changes the creation of log files. Amends the uncertainty wizard test to correct for use of logging over the standard console. Adds the .logs to .gitignore.

* 1) Simplifies the use of the python logging facility with QtThreads (as published on the python docs for logging), improving the stability of logging.\n2) Changes the generation of the logging object that is more compatible with pytest, supporting the existing test procedures.

* Provides an update to the Model for the calculation_setup Methods table, corrects for deletion of impact assessment methods. Updates the routine to the lca_setup module for calling the logger.

* Updates to the logger module, what was the ABLogger class is renamed to ABHandler. The module now contains the formats and settings for the Stream handlers. Calling of the methods in the other AB modules requires passing a logging.Logger instance and the name of the calling module. These are then incorporated into the wrapping routines in the class. Additions:\n 1) An error wrapper is provided and also explicitly provides a trace of the error.\n 2) A timestamp routine is provided and used for providing the file names for log files.\n 3) A standard location based on appdirs is provided and used for log file locations.

* Corrections to the setup of the logger in the test_uncertainty_wizard module

* Includes a change to the type used for the scenario columns when using multiple files with the combined (combinatoric) approach for the scenarios. Includes respective changes to the boolean tests applied to such Indexes.

* Minor corrections to local repository branch, to keep changes aligned

* Alterations to keep minor changes aligned with master branch and fork

---------

Co-authored-by: zoo <zoo@PK-home.kpn>

* Use node16 actions for main pipeline

* Use node16 actions in release pipeline

* Use latest version of release-changelog-builder

* Use node16 action for install canary pipeline

* Remove the special build for arm architecture

This was added before brightway2 supported multi-arch builds on
conda-forge. It is not needed anymore, because the normal AB is now also
installable on arm arch.

* Install canary updates (#1093)

* Increase timeout to 30 min

Originally the idea was that installation should never take longer than
12 min, otherwise we should get a warning with the failed pipeline. But
unfortuantely conda currently takes longer than 12 minutes to solve
the environment.

* Add mamba install canary

* Download artifacts to start implementing env comparison

* Looks like on linux it can take more than 30 minutes :-(

* 60min not enough on linux, increasing to 120

* Completely remove timeout, default is 6 hours

* Use solver libmamba option

* Add diff step to compare installations

* Use node16 actions

* Re-add the 12 timeout

* Split diff into separate steps

* Yq action only runs on linux

* debug yq formatting step

* Simplify

* more fighting with yq action

* artifacts are apparently directories in this case

* Try again with while loop

* Run on all os, but only 3.9

* ignore diff exit code

* Run canary install for 3.8 and 3.9 again

---------

Co-authored-by: haasad <haasad@users.noreply.github.com>

* Adding error message if no scenario file is loaded in a scenario LCA (#1085)

* Updated and improved contributing file

* Improve text on dependencies

* Fix canary link

* Add good/bad examples to CONTRIBUTING.md for commit messages and PR titles

* Add more specific requirements on communicating work on issues in CONTRIBUTING.md

* Minor updates to CONTRIBUTING.md and README.md

* Minor improvements to CONTRIBUTING.md

* Fix formatting error in PULL_REQUEST_TEMPLATE.md

* Resolve #1105

* Update copyright and license information to only link to one place.

* Resolve 3 bugs for location linking (#1051): 1) database of own activity was not always added (in case none of the exchanges were in own database) 2) improve handling of multiple locations. 3) replaced exchanges were deleted from the old process instead of the new one.

* Improve choosing and handling of alternative locations for location choosing

* Resolve readonly database bug for activity duplication to new activity.

* Store AB version in logfile

* Remove deprecated channels from conda envs

* Remove experimental conda environment creation and upload

I started this mostly as an experimental alternative install method with
`conda env create`. However since the channel handling isn't such a pain
anymore now that everything is on conda-forge this is obsolete.

* Move conda-envs to .github directory

* Move changelog config to .github

* Move dev recipe to .github directory

* Move stable recipe to recipe/meta.yaml

* Update dev recipe

* Use micromamba instead of miniconda to speed up tests

* Update install instructions to use libmamba solver with conda

* Fix path for dev recipe

* Added tooltip for table content

* Added docstring for data method

* Cleaning-up documentation

* Update README.md

Resolve #1127

* Add - copy on impact category copy (#1136)

* Add - copy on impact category copy #1133

* Reorder impact category tree context menu

* Resolve bug with switching project on delete (#1138)

* Switch project on delete #1075

* Cleaning up ProjectController.change_project

* Resolving comments by marc-vdm

* GH action script for commenting on issues when related milestone is closed.

* GH action script for commenting on issues when related milestone is closed.

* GH action script for commenting on issues when related milestone is closed.

* Add test for semantic version sort

* review instructions bot

* review instructions bot

* Update dialog text

* Passed main windows as parent to popup

* Update safe_link_fetch from utils

* Update safe_link_fetch from utils and update link to correct repo

* Update safe_link_fetch from utils and update link to correct repo

---------

Co-authored-by: bsteubing <33026150+bsteubing@users.noreply.github.com>
Co-authored-by: zoophobus <jonathan.kidner@gmail.com>
Co-authored-by: Jonathan Kidner <12627199+Zoophobus@users.noreply.github.com>
Co-authored-by: zoo <zoo@PK-home.kpn>
Co-authored-by: haasad <haasad@users.noreply.github.com>
Co-authored-by: Adrian Haas <11636405+haasad@users.noreply.github.com>
Co-authored-by: Marin Visscher <m.r.visscher@science.leidenuniv.nl>
Co-authored-by: Marin Visscher <103424764+mrvisscher@users.noreply.github.com>

activity_browser/bwutils/ecoinvent_biosphere_versions/compatible_ei_versions.txt

@@ -0,0 +1,3 @@
+{
+'2.9': ['3.4', '3.5', '3.6', '3.7', '3.7.1', '3.8', '3.9', '3.9.1'],
+}

activity_browser/bwutils/ecoinvent_biosphere_versions/ecospold2biosphereimporter.py

@@ -0,0 +1,75 @@
+from bw2io.importers import Ecospold2BiosphereImporter
+from bw2io.importers.ecospold2_biosphere import EMISSIONS_CATEGORIES
+from bw2data.utils import recursive_str_to_unicode
+from zipfile import ZipFile
+
+from ...info import __ei_versions__
+from ...utils import sort_semantic_versions
+
+import os
+from lxml import objectify
+
+import logging
+from activity_browser.logger import ABHandler
+
+logger = logging.getLogger('ab_logs')
+log = ABHandler.setup_with_logger(logger, __name__)
+
+def create_default_biosphere3(version) -> None:
+    """Reimplementation of bw.create_default_biosphere3 to allow import from older biosphere versions."""
+    # format version number to only Major/Minor
+    version = version[:3]
+
+    if version == sort_semantic_versions(__ei_versions__)[0]:
+        # most recent version
+        eb = Ecospold2BiosphereImporter()
+    else:
+        # not most recent version, import legacy biosphere from AB
+        eb = ABEcospold2BiosphereImporter(version=version)
+    eb.apply_strategies()
+    eb.write_database()
+
+
+class ABEcospold2BiosphereImporter(Ecospold2BiosphereImporter):
+    """Reimplementation of bw2io.importers Ecospold2BiosphereImporter to import legacy biosphere from AB data"""
+
+    def extract(self, version):
+        def extract_flow_data(o):
+            ds = {
+                "categories": (
+                    o.compartment.compartment.text,
+                    o.compartment.subcompartment.text,
+                ),
+                "code": o.get("id"),
+                "CAS number": o.get("casNumber"),
+                "name": o.name.text,
+                "database": self.db_name,
+                "exchanges": [],
+                "unit": o.unitName.text,
+            }
+            ds["type"] = EMISSIONS_CATEGORIES.get(
+                ds["categories"][0], ds["categories"][0]
+            )
+            return ds
+
+        lci_dirpath = os.path.join(os.path.dirname(__file__), 'legacy_biosphere')
+
+        # find the most recent legacy biosphere that is equal to or older than chosen version
+        for ei_version in sort_semantic_versions(__ei_versions__):
+            use_version = ei_version
+            fp = os.path.join(lci_dirpath, f'ecoinvent elementary flows {use_version}.xml.zip')
+            if sort_semantic_versions([version, ei_version])[0] == version and os.path.isfile(fp):
+                # this version is equal/lower and available
+                break
+
+        # extract the xml from the zip
+        with ZipFile(fp) as zipped_file:
+            with zipped_file.open(f'ecoinvent elementary flows {use_version}.xml') as file:
+                root = objectify.parse(file).getroot()
+
+        log.debug(f'Installing biosphere {use_version} for chosen version {version}')
+        flow_data = recursive_str_to_unicode(
+            [extract_flow_data(ds) for ds in root.iterchildren()]
+        )
+
+        return flow_data

activity_browser/controllers/database.py

@@ -9,13 +9,15 @@
 from ..bwutils.strategies import relink_exchanges_existing_db
 from ..ui.widgets import (
     CopyDatabaseDialog, DatabaseLinkingDialog, DefaultBiosphereDialog,
-    BiosphereUpdater, DatabaseLinkingResultsDialog
+    BiosphereUpdater, DatabaseLinkingResultsDialog, EcoinventVersionDialog
 )
 from ..ui.wizards.db_export_wizard import DatabaseExportWizard
 from ..ui.wizards.db_import_wizard import DatabaseImportWizard
 from ..settings import project_settings
 from ..signals import signals
 from .project import ProjectController
+from ..info import __ei_versions__
+from ..utils import sort_semantic_versions
 
 import logging
 from activity_browser.logger import ABHandler
@@ -66,23 +68,43 @@ def ensure_sqlite_indices(self):
 
     @Slot(name="bw2Setup")
     def install_default_data(self) -> None:
-        dialog = DefaultBiosphereDialog(self.window)
+
+        # let user choose version
+        version_dialog = EcoinventVersionDialog(self.window)
+        if version_dialog.exec_() != EcoinventVersionDialog.Accepted: return
+        version = version_dialog.options.currentText()
+
+        dialog = DefaultBiosphereDialog(version[:3], self.window)  # only read Major/Minor part of version
         dialog.show()
 
     @Slot(name="updateBiosphereDialog")
     def update_biosphere(self) -> None:
         """ Open a popup with progression bar and run through the different
         functions for adding ecoinvent biosphere flows.
         """
-        ok = QtWidgets.QMessageBox.question(
+        # warn user of consequences of updating
+        warn_dialog = QtWidgets.QMessageBox.question(
             self.window, "Update biosphere3?",
-            "Updating the biosphere3 database cannot be undone!",
+            'Newer versions of the biosphere database may not\n'
+            'always be compatible with older ecoinvent versions.\n'
+            '\nUpdating the biosphere3 database cannot be undone!\n',
             QtWidgets.QMessageBox.Ok | QtWidgets.QMessageBox.Abort,
             QtWidgets.QMessageBox.Abort
         )
-        if ok == QtWidgets.QMessageBox.Ok:
-            dialog = BiosphereUpdater(self.window)
-            dialog.show()
+        if warn_dialog is not QtWidgets.QMessageBox.Ok: return
+
+        # let user choose version
+        version_dialog = EcoinventVersionDialog(self.window)
+        if version_dialog.exec_() != EcoinventVersionDialog.Accepted: return
+        version = version_dialog.options.currentText()
+
+        # reduce biosphere update list up to the selected version
+        sorted_versions = sort_semantic_versions(__ei_versions__, highest_to_lowest=False)
+        ei_versions = sorted_versions[:sorted_versions.index(version) + 1]
+
+        # show updating dialog
+        dialog = BiosphereUpdater(ei_versions, self.window)
+        dialog.show()
 
     @Slot(name="addDatabase")
     def add_database(self):

activity_browser/info.py

@@ -1,6 +1,62 @@
+import ast
+import os.path
 from importlib.metadata import version, PackageNotFoundError
+from .utils import safe_link_fetch, sort_semantic_versions
 
+import logging
+from .logger import ABHandler
+logger = logging.getLogger('ab_logs')
+log = ABHandler.setup_with_logger(logger, __name__)
+
+# get AB version
 try:
     __version__ = version(__package__)
 except PackageNotFoundError:
     __version__ = "0.0.0"
+
+
+def get_compatible_versions() -> list:
+    """Get compatible versions of ecoinvent for this AB version.
+
+    Reads this file on github repo: activity-browser/better_biosphere_handling/compatible_ei_versions.txt'.
+    Converts file content to available ecoinvent versions for each version of AB.
+    Finds the correct available versions for this AB version, if failing to read version,
+    the lowest version in the file is chosen.
+    """
+    try:
+        # read versions
+        versions_URL = 'https://raw.githubusercontent.com/LCA-ActivityBrowser/activity-browser/master/activity_browser/bwutils/ecoinvent_biosphere_versions/compatible_ei_versions.txt'
+        page, error = safe_link_fetch(versions_URL)
+        if not error:
+            file = page.text
+        else:
+            # silently try a local fallback:
+            log.debug(f'Reading online compatible ecoinvent versions failed '
+                      f'-attempting local fallback- with this error: {error}')
+            file_path = os.path.join(os.path.dirname(__file__),
+                                     'bwutils',
+                                     'ecoinvent_biosphere_versions',
+                                     'compatible_ei_versions.txt')
+            with open(file_path, 'r') as f:
+                file = f.read()
+        all_versions = ast.literal_eval(file)
+
+        # select either the latest lower version available or if none available the lowest version for safety
+        sorted_versions = sort_semantic_versions(all_versions.keys())
+        for ab_version in sorted_versions:
+            if sort_semantic_versions([__version__, ab_version])[0] == __version__:
+                # current version is higher than or equal to tested AB version:
+                ei_versions = all_versions[ab_version]
+                break
+        else:
+            ei_versions = all_versions[sorted_versions[-1]]
+
+        log.debug(f'Following versions of ecoinvent are compatible with AB {__version__}: {ei_versions}')
+        return ei_versions
+
+    except Exception as error:
+        log.debug(f'Reading local fallback failed with: {error}')
+        return ['3.4', '3.5', '3.6', '3.7', '3.7.1', '3.8', '3.9', '3.9.1']
+
+
+__ei_versions__ = get_compatible_versions()

activity_browser/ui/tables/models/lca_setup.py

@@ -94,6 +94,7 @@ def activities(self) -> list:
         return [{x["key"]: x["Amount"]} for x in selection]
 
     def check_activities(self, db):
+        if db == 'biosphere3': return  # if biosphere changed, we don't need to check as it doesn't have activities.
         databases = [list(k.keys())[0][0] for k in self.activities]
         if db in databases:
             self.sync()

activity_browser/ui/widgets/__init__.py

@@ -9,7 +9,7 @@
     DatabaseLinkingDialog, DefaultBiosphereDialog,
     DatabaseLinkingResultsDialog, ActivityLinkingDialog,
     ActivityLinkingResultsDialog, ProjectDeletionDialog,
-    ScenarioDatabaseDialog, LocationLinkingDialog
+    ScenarioDatabaseDialog, LocationLinkingDialog, EcoinventVersionDialog
 )
 from .line_edit import (SignalledPlainTextEdit, SignalledComboEdit,
                         SignalledLineEdit)

activity_browser/ui/widgets/biosphere_update.py

@@ -14,14 +14,14 @@
 
 
 class BiosphereUpdater(QtWidgets.QProgressDialog):
-    def __init__(self, parent=None):
+    def __init__(self, ei_versions, parent=None):
         super().__init__(parent=parent)
         self.setWindowTitle("Updating '{}' database".format(bw.config.biosphere))
         self.setLabelText("Adding new flows to biosphere database")
         self.setRange(0, 0)
         self.show()
 
-        self.thread = UpdateBiosphereThread(self)
+        self.thread = UpdateBiosphereThread(ei_versions, self)
         self.setMaximum(self.thread.total_patches)
         self.thread.progress.connect(self.update_progress)
         self.thread.finished.connect(self.finished)
@@ -43,16 +43,21 @@ def update_progress(self, current: int):
 class UpdateBiosphereThread(QtCore.QThread):
     PATCHES = [patch for patch in dir(data) if patch.startswith('add_ecoinvent') and patch.endswith('biosphere_flows')]
     progress = Signal(int)
-    def __init__(self, parent=None):
+    def __init__(self, ei_versions, parent=None):
         super().__init__(parent)
+
+        # reduce the patches list to only compatible versions for this AB version
+        self.PATCHES = [p for p in self.PATCHES if any(v.replace('.', '') in p for v in ei_versions)]
+
         self.total_patches = len(self.PATCHES)
 
     def run(self):
         try:
             for i, patch in enumerate(self.PATCHES):
                 self.progress.emit(i)
+                log.debug(f'Applying biosphere patch: {patch}')
                 update_bio = getattr(data, patch)
                 update_bio()
         except ValidityError as e:
-            log.error("Could not patch biosphere: {}".format(str(e)))
+            log.error(f'Could not patch biosphere: {str(e)}')
             self.exit(1)

activity_browser/ui/widgets/dialog.py

@@ -11,6 +11,10 @@
 from activity_browser.signals import signals
 from ..style import style_group_box, vertical_line
 from ...ui.icons import qicons
+from ...ui.widgets import BiosphereUpdater
+from ...info import __ei_versions__
+from ...bwutils.ecoinvent_biosphere_versions.ecospold2biosphereimporter import create_default_biosphere3
+from ...utils import sort_semantic_versions
 
 class ForceInputDialog(QtWidgets.QDialog):
     """ Due to QInputDialog not allowing 'ok' button to be disabled when
@@ -498,18 +502,23 @@ def open_activity(self):
 
 
 class DefaultBiosphereDialog(QtWidgets.QProgressDialog):
-    def __init__(self, parent=None):
+    def __init__(self, version, parent=None):
         super().__init__(parent=parent)
-        self.setWindowTitle("Biosphere and impact categories")
+        self.setWindowTitle('Biosphere and impact categories')
         self.setRange(0, 3)
         self.setModal(Qt.ApplicationModal)
 
-        self.biosphere_thread = DefaultBiosphereThread(self)
+        self.version = version
+
+        self.biosphere_thread = DefaultBiosphereThread(self.version, self)
         self.biosphere_thread.update.connect(self.update_progress)
         self.biosphere_thread.finished.connect(self.finished)
         self.biosphere_thread.start()
 
-    @Slot(int, str, name="updateThread")
+        # finally, check if patches are available for this version and apply them
+        self.check_patches()
+
+    @Slot(int, str, name='updateThread')
     def update_progress(self, current: int, text: str) -> None:
         self.setValue(current)
         self.setLabelText(text)
@@ -520,15 +529,29 @@ def finished(self, result: int = None) -> None:
         signals.change_project.emit(bw.projects.current)
         signals.project_selected.emit()
 
+    def check_patches(self):
+        """Apply any relevant biosphere patches if available."""
+        # reduce biosphere update list up to the selected version
+        sorted_versions = sort_semantic_versions(__ei_versions__, highest_to_lowest=False)
+        ei_versions = sorted_versions[:sorted_versions.index(self.version) + 1]
+
+        # show updating dialog
+        dialog = BiosphereUpdater(ei_versions, self)
+        dialog.show()
+
 
 class DefaultBiosphereThread(QThread):
     update = Signal(int, str)
 
+    def __init__(self, version, parent=None):
+        super().__init__(parent=parent)
+        self.version = version
+
     def run(self):
         project = "<b>{}</b>".format(bw.projects.current)
         if "biosphere3" not in bw.databases:
             self.update.emit(0, "Creating default biosphere for {}".format(project))
-            bw.create_default_biosphere3()
+            create_default_biosphere3(self.version)
             project_settings.add_db("biosphere3")
         if not len(bw.methods):
             self.update.emit(1, "Creating default LCIA methods for {}".format(project))
@@ -1233,3 +1256,29 @@ def relink_location(cls, act_name: str, options: List[Tuple[str, List[str]]],
                         parent=None) -> 'LocationLinkingDialog':
         label = "Relinking exchanges from activity '{}' to a new location.".format(act_name)
         return cls.construct_dialog(label, options, parent)
+
+
+class EcoinventVersionDialog(QtWidgets.QDialog):
+    def __init__(self, parent=None):
+        super(EcoinventVersionDialog, self).__init__(parent)
+
+        self.setWindowTitle("Choose a biosphere version")
+
+        self.buttons = QtWidgets.QDialogButtonBox(
+            QtWidgets.QDialogButtonBox.Ok | QtWidgets.QDialogButtonBox.Cancel,
+        )
+        self.buttons.accepted.connect(self.accept)
+        self.buttons.rejected.connect(self.reject)
+
+        self.layout = QtWidgets.QVBoxLayout()
+        self.label = QtWidgets.QLabel('Choose which biosphere version\n'
+                                      'you would like to use')
+        self.options = QtWidgets.QComboBox()
+
+        # Add available ecoinvent versions to the combobox
+        self.options.addItems(sort_semantic_versions(__ei_versions__))
+
+        self.layout.addWidget(self.label)
+        self.layout.addWidget(self.options)
+        self.layout.addWidget(self.buttons)
+        self.setLayout(self.layout)

activity_browser/ui/wizards/db_import_wizard.py

@@ -21,6 +21,8 @@
 from ...signals import signals
 from ..style import style_group_box
 from ..widgets import DatabaseLinkingDialog
+from ...info import __ei_versions__
+from ...utils import sort_semantic_versions
 
 import logging
 from activity_browser.logger import ABHandler
@@ -989,15 +991,16 @@ def initializePage(self):
             self.db_dict = self.wizard.downloader.db_dict
         self.system_models = {
             version: sorted({k[1] for k in self.db_dict.keys() if k[0] == version}, reverse=True)
-            for version in sorted({k[0] for k in self.db_dict.keys()}, reverse=True)
+            for version in sorted({k[0] for k in self.db_dict.keys() if k[0] in __ei_versions__}, reverse=True)
         }
         # Catch for incorrect 'universal' key presence
         # (introduced in version 3.6 of ecoinvent)
         if "universal" in self.system_models:
             del self.system_models["universal"]
         self.version_combobox.clear()
         self.system_model_combobox.clear()
-        self.version_combobox.addItems(list(self.system_models.keys()))
+        versions = sort_semantic_versions(self.system_models.keys())
+        self.version_combobox.addItems(versions)
         if bool(self.version_combobox.count()):
             # Adding the items will cause system_model_combobox to update
             # and show the correct list, this is just to be sure.