Issue 33 add helper get xxx method

.github/workflows/ci.yml

@@ -25,15 +25,13 @@ jobs:
           - "3.9"
         os:
           - ubuntu-latest
-          - macos-latest
-          - windows-latest
     steps:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
 
       - name: Setup python for test ${{ matrix.py }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.py }}
 

README.md

@@ -167,6 +167,11 @@ The client library documentation is hosted on [GitHub Pages](https://oceannetwor
 For documentation and examples about Oceans 3.0 API, visit the [wiki](https://wiki.oceannetworks.ca/display/O2A/Oceans+3.0+API+Home)
 and [OpenAPI](https://data.oceannetworks.ca/OpenAPI) page on the Oceans 3.0 Data Portal website.
 
+# Multithreading issue
+
+We kindly ask users to **not** use too many threads when using threading/multiprocessing libraries on download tasks.
+It can cause issues for both server and client and may not appreciably increase download speeds.
+
 # Contributing
 
 All contributions are welcome and appreciated!

doc/source/API_Guide.md

@@ -45,15 +45,15 @@ Use discovery methods to:
 
 :::
 
-|                          Method                           |             Description              |                                   API Endpoint                                   |
-| :-------------------------------------------------------: | :----------------------------------: | :------------------------------------------------------------------------------: |
-|         [getLocations](#onc.onc.ONC.getLocations)         |          Returns locations           |        [/locations](https://data.oceannetworks.ca/OpenAPI#get-/locations)        |
-| [getLocationHierarchy](#onc.onc.ONC.getLocationHierarchy) |       Returns a location tree        |   [/locations/tree](https://data.oceannetworks.ca/OpenAPI#get-/locations/tree)   |
-|       [getDeployments](#onc.onc.ONC.getDeployments)       | Returns a list of device deployments |      [/deployments](https://data.oceannetworks.ca/OpenAPI#get-/deployments)      |
-|  [getDeviceCategories](#onc.onc.ONC.getDeviceCategories)  | Returns a list of device categories  | [/deviceCategories](https://data.oceannetworks.ca/OpenAPI#get-/deviceCategories) |
-|           [getDevices](#onc.onc.ONC.getDevices)           |      Returns a list of devices       |          [/devices](https://data.oceannetworks.ca/OpenAPI#get-/devices)          |
-|        [getProperties](#onc.onc.ONC.getProperties)        |     Returns a list of properties     |       [/properties](https://data.oceannetworks.ca/OpenAPI#get-/properties)       |
-|      [getDataProducts](#onc.onc.ONC.getDataProducts)      |   Returns a list of data products    |     [/dataProducts](https://data.oceannetworks.ca/OpenAPI#get-/dataProducts)     |
+|                         Method                          |             Description             |                                   API Endpoint                                   |
+| :-----------------------------------------------------: | :---------------------------------: | :------------------------------------------------------------------------------: |
+|        [getLocations](#onc.onc.ONC.getLocations)        |          Return locations           |        [/locations](https://data.oceannetworks.ca/OpenAPI#get-/locations)        |
+|    [getLocationsTree](#onc.onc.ONC.getLocationsTree)    |       Return a location tree        |   [/locations/tree](https://data.oceannetworks.ca/OpenAPI#get-/locations/tree)   |
+|      [getDeployments](#onc.onc.ONC.getDeployments)      | Return a list of device deployments |      [/deployments](https://data.oceannetworks.ca/OpenAPI#get-/deployments)      |
+| [getDeviceCategories](#onc.onc.ONC.getDeviceCategories) | Return a list of device categories  | [/deviceCategories](https://data.oceannetworks.ca/OpenAPI#get-/deviceCategories) |
+|          [getDevices](#onc.onc.ONC.getDevices)          |      Return a list of devices       |          [/devices](https://data.oceannetworks.ca/OpenAPI#get-/devices)          |
+|       [getProperties](#onc.onc.ONC.getProperties)       |     Return a list of properties     |       [/properties](https://data.oceannetworks.ca/OpenAPI#get-/properties)       |
+|     [getDataProducts](#onc.onc.ONC.getDataProducts)     |   Return a list of data products    |     [/dataProducts](https://data.oceannetworks.ca/OpenAPI#get-/dataProducts)     |
 
 ## Data product download methods
 
@@ -112,18 +112,20 @@ Use the _allPages_ parameter to automatically download all pages required for yo
 
 :::
 
-|                            Method                             |                              Description                              |                                      API Endpoint                                      |
-| :-----------------------------------------------------------: | :-------------------------------------------------------------------: | :------------------------------------------------------------------------------------: |
-|    [getDirectByLocation](#onc.onc.ONC.getDirectByLocation)    | Returns scalar data <br> from a specific location and device category | [/scalardata/location](https://data.oceannetworks.ca/OpenAPI#get-/scalardata/location) |
-|      [getDirectByDevice](#onc.onc.ONC.getDirectByDevice)      |              Returns scalar data from a specific device               |   [/scalardata/device](https://data.oceannetworks.ca/OpenAPI#get-/scalardata/device)   |
-| [getDirectRawByLocation](#onc.onc.ONC.getDirectRawByLocation) |  Returns raw data <br> from a specific location and device category   |    [/rawdata/location](https://data.oceannetworks.ca/OpenAPI#get-/rawdata/location)    |
-|   [getDirectRawByDevice](#onc.onc.ONC.getDirectRawByDevice)   |                Returns raw data from a specific device                |      [/rawdata/device](https://data.oceannetworks.ca/OpenAPI#get-/rawdata/device)      |
+|                             Method                              |                             Description                              |                                      API Endpoint                                      |
+| :-------------------------------------------------------------: | :------------------------------------------------------------------: | :------------------------------------------------------------------------------------: |
+| [getScalardataByLocation](#onc.onc.ONC.getScalardataByLocation) | Return scalar data <br> from a specific location and device category | [/scalardata/location](https://data.oceannetworks.ca/OpenAPI#get-/scalardata/location) |
+|   [getScalardataByDevice](#onc.onc.ONC.getScalardataByDevice)   |              Return scalar data from a specific device               |   [/scalardata/device](https://data.oceannetworks.ca/OpenAPI#get-/scalardata/device)   |
+|    [getRawdataByLocation](#onc.onc.ONC.getRawdataByLocation)    |  Return raw data <br> from a specific location and device category   |    [/rawdata/location](https://data.oceannetworks.ca/OpenAPI#get-/rawdata/location)    |
+|      [getRawdataByDevice](#onc.onc.ONC.getRawdataByDevice)      |                Return raw data from a specific device                |      [/rawdata/device](https://data.oceannetworks.ca/OpenAPI#get-/rawdata/device)      |
 
 Helper methods are listed below.
 
-|                            Method                             |                                      Description                                      |
-| :-----------------------------------------------------------: | :-----------------------------------------------------------------------------------: |
-| [getSensorCategoryCodes](#onc.onc.ONC.getSensorCategoryCodes) | Returns a list of sensor category codes <br> prior to querying the scalardata service |
+|                            Method                             |                                     Description                                      |
+| :-----------------------------------------------------------: | :----------------------------------------------------------------------------------: |
+| [getSensorCategoryCodes](#onc.onc.ONC.getSensorCategoryCodes) | Return a list of sensor category codes <br> prior to querying the scalardata service |
+|          [getScalardata](#onc.onc.ONC.getScalardata)          |                                  Return scalar data                                  |
+|             [getRawdata](#onc.onc.ONC.getRawdata)             |                                   Return raw data                                    |
 
 ## Archive file download methods
 
@@ -148,14 +150,15 @@ Due to security regulations, some very recent files (e.g. hydrophone.wav files i
 
 :::
 
-|                       Method                        |                                         Description                                         |                                       API Endpoint                                       |
-| :-------------------------------------------------: | :-----------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------: |
-| [getListByLocation](#onc.onc.ONC.getListByLocation) | Returns a list of available archive files <br> from a specific location and device category | [/archivefile/location](https://data.oceannetworks.ca/OpenAPI#get-/archivefile/location) |
-|   [getListByDevice](#onc.onc.ONC.getListByDevice)   |            Returns a list of available archive files <br> from a specific device            |   [/archivefile/device](https://data.oceannetworks.ca/OpenAPI#get-/archivefile/device)   |
-|           [getFile](#onc.onc.ONC.getFile)           |                                  Download an archive file                                   | [/archivefile/download](https://data.oceannetworks.ca/OpenAPI#get-/archivefile/download) |
+|                              Method                               |                                        Description                                         |                                       API Endpoint                                       |
+| :---------------------------------------------------------------: | :----------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------: |
+| [getArchivefileByLocation](#onc.onc.ONC.getArchivefileByLocation) | Return a list of available archive files <br> from a specific location and device category | [/archivefile/location](https://data.oceannetworks.ca/OpenAPI#get-/archivefile/location) |
+|   [getArchivefileByDevice](#onc.onc.ONC.getArchivefileByDevice)   |            Return a list of available archive files <br> from a specific device            |   [/archivefile/device](https://data.oceannetworks.ca/OpenAPI#get-/archivefile/device)   |
+|      [downloadArchivefile](#onc.onc.ONC.downloadArchivefile)      |                                  Download an archive file                                  | [/archivefile/download](https://data.oceannetworks.ca/OpenAPI#get-/archivefile/download) |
 
 Helper methods are listed below.
 
-|                    Method                     |                            Description                            |
-| :-------------------------------------------: | :---------------------------------------------------------------: |
-| [getDirectFiles](#onc.onc.ONC.getDirectFiles) | Download a list of archived files that match the filters provided |
+|                               Method                                |                            Description                            |
+| :-----------------------------------------------------------------: | :---------------------------------------------------------------: |
+| [downloadDirectArchivefile](#onc.onc.ONC.downloadDirectArchivefile) | Download a list of archived files that match the filters provided |
+|            [getArchivefile](#onc.onc.ONC.getArchivefile)            |             Return a list of available archive files              |

doc/source/conf.py

@@ -51,3 +51,33 @@
 autoapi_dirs = ["../../src"]
 autoapi_ignore = ["*modules*"]
 suppress_warnings = ["autoapi.python_import_resolution"]
+
+
+def skip_rules(app, what, name, obj, skip, options):
+    # 1. skip aliases in ONC class
+    aliases = [
+        "getLocationHierarchy",
+        "getDirectByLocation",
+        "getDirectByDevice",
+        "getDirectRawByLocation",
+        "getDirectRawByDevice",
+        "getListByLocation",
+        "getListByDevice",
+        "getFile",
+        "getDirectFiles",
+    ]
+    onc_aliases = {f"onc.ONC.{alias}" for alias in aliases}
+
+    if name in onc_aliases:
+        skip = True
+
+    # 2. skip submodules onc.onc.ONC
+    if what == "module":
+        skip = True
+
+    return skip
+
+
+def setup(sphinx):
+    # sphinx.connect("autoapi-skip-member", skip_aliases_in_ONC_class)
+    sphinx.connect("autoapi-skip-member", skip_rules)

src/onc/modules/_OncArchive.py

@@ -18,23 +18,31 @@ class _OncArchive(_OncService):
     def __init__(self, parent: object):
         super().__init__(parent)
 
-    def getListByLocation(self, filters: dict = None, allPages: bool = False):
+    def getArchivefileByLocation(self, filters: dict, allPages: bool):
         """
         Return a list of archived files for a device category in a location.
 
         The filenames obtained can be used to download files using the getFile() method.
         """
         return self._getList(filters, by="location", allPages=allPages)
 
-    def getListByDevice(self, filters: dict = None, allPages: bool = False):
+    def getArchivefileByDevice(self, filters: dict, allPages: bool):
         """
         Return a list of archived files from a specific device.
 
         The filenames obtained can be used to download files using the getFile() method.
         """
         return self._getList(filters, by="device", allPages=allPages)
 
-    def getFile(self, filename: str = "", overwrite: bool = False):
+    def getArchivefile(self, filters: dict, allPages: bool):
+        return self._delegateByFilters(
+            byDevice=self.getArchivefileByDevice,
+            byLocation=self.getArchivefileByLocation,
+            filters=filters,
+            allPages=allPages,
+        )
+
+    def downloadArchivefile(self, filename: str = "", overwrite: bool = False):
         url = self._serviceUrl("archivefiles")
 
         filters = {
@@ -71,7 +79,7 @@ def getFile(self, filename: str = "", overwrite: bool = False):
             "file": filename,
         }
 
-    def getDirectFiles(
+    def downloadDirectArchivefile(
         self, filters: dict, overwrite: bool = False, allPages: bool = False
     ):
         """
@@ -88,16 +96,7 @@ def getDirectFiles(
             del filters["returnOptions"]
 
         # Get a list of files
-        if "locationCode" in filters and "deviceCategoryCode" in filters:
-            dataRows = self.getListByLocation(filters=filters, allPages=allPages)
-        elif "deviceCode" in filters:
-            dataRows = self.getListByDevice(filters=filters, allPages=allPages)
-        else:
-            raise ValueError(
-                "getDirectFiles filters require either a combination of "
-                '"locationCode" and "deviceCategoryCode", '
-                'or a "deviceCode" present.'
-            )
+        dataRows = self.getArchivefile(filters, allPages)
 
         n = len(dataRows["files"])
         print(f"Obtained a list of {n} files to download.")
@@ -116,7 +115,7 @@ def getDirectFiles(
 
             if (not fileExists) or (fileExists and overwrite):
                 print(f'   ({tries} of {n}) Downloading file: "{filename}"')
-                downInfo = self.getFile(filename, overwrite)
+                downInfo = self.downloadArchivefile(filename, overwrite)
                 size += downInfo["size"]
                 time += downInfo["downloadTime"]
                 downInfos.append(downInfo)

src/onc/modules/_OncRealTime.py

@@ -12,7 +12,7 @@ class _OncRealTime(_OncService):
     def __init__(self, config: dict):
         super().__init__(config)
 
-    def getDirectByLocation(self, filters: dict, allPages: bool):
+    def getScalardataByLocation(self, filters: dict, allPages: bool):
         """
         Return scalar data readings from a device category in a location.
 
@@ -21,7 +21,7 @@ def getDirectByLocation(self, filters: dict, allPages: bool):
         """
         return self._getDirectAllPages(filters, "scalardata", "getByLocation", allPages)
 
-    def getDirectByDevice(self, filters: dict, allPages: bool):
+    def getScalardataByDevice(self, filters: dict, allPages: bool):
         """
         Return scalar data readings from a device.
 
@@ -30,7 +30,15 @@ def getDirectByDevice(self, filters: dict, allPages: bool):
         """
         return self._getDirectAllPages(filters, "scalardata", "getByDevice", allPages)
 
-    def getDirectRawByLocation(self, filters: dict, allPages: bool):
+    def getScalardata(self, filters: dict, allPages: bool):
+        return self._delegateByFilters(
+            byDevice=self.getScalardataByDevice,
+            byLocation=self.getScalardataByLocation,
+            filters=filters,
+            allPages=allPages,
+        )
+
+    def getRawdataByLocation(self, filters: dict, allPages: bool):
         """
         Return raw data readings from a device category in a location.
 
@@ -39,7 +47,7 @@ def getDirectRawByLocation(self, filters: dict, allPages: bool):
         """
         return self._getDirectAllPages(filters, "rawdata", "getByLocation", allPages)
 
-    def getDirectRawByDevice(self, filters: dict, allPages: bool):
+    def getRawdataByDevice(self, filters: dict, allPages: bool):
         """
         Return raw data readings from an device.
 
@@ -48,12 +56,17 @@ def getDirectRawByDevice(self, filters: dict, allPages: bool):
         """
         return self._getDirectAllPages(filters, "rawdata", "getByDevice", allPages)
 
+    def getRawdata(self, filters: dict, allPages: bool):
+        return self._delegateByFilters(
+            byDevice=self.getRawdataByDevice,
+            byLocation=self.getRawdataByLocation,
+            filters=filters,
+            allPages=allPages,
+        )
+
     def getSensorCategoryCodes(self, filters: dict):
         updated_filters = filters | {"returnOptions": "excludeScalarData"}
-        if "deviceCode" in filters:
-            return self.getDirectByDevice(updated_filters, False)["sensorData"]
-        else:
-            return self.getDirectByLocation(updated_filters, False)["sensorData"]
+        return self.getScalardata(updated_filters, False)["sensorData"]
 
     def _getDirectAllPages(
         self, filters: dict, service: str, method: str, allPages: bool

src/onc/modules/_OncService.py

@@ -97,3 +97,20 @@ def _config(self, key: str):
         Returns a property from the parent (ONC class)
         """
         return getattr(self.parent(), key)
+
+    def _delegateByFilters(self, byDevice, byLocation, **kwargs):
+        """
+        Delegate getX helper methods into getXByDevice or getXByLocation methods.
+        """
+        filters = kwargs["filters"]
+
+        if "deviceCode" in filters:
+            return byDevice(**kwargs)
+        elif "locationCode" in filters and "deviceCategoryCode" in filters:
+            return byLocation(**kwargs)
+        else:
+            raise ValueError(
+                "Query parameters require either a combination of "
+                "'locationCode' and 'deviceCategoryCode', "
+                "or a 'deviceCode' present."
+            )