Issue 26 cancel restart data product

doc/source/API_Guide.md

@@ -12,9 +12,17 @@ Check [here](https://wiki.oceannetworks.ca/display/O2A/Glossary+of+Terms) for mo
 
 The [ONC](#onc.onc.ONC) class provides a wrapper for Oceans 3.0 API requests.
 All the client library's functionality is provided as methods of this class.
+Each [Oceans 3.0 public API](https://data.oceannetworks.ca/OpenAPI) has a corresponding public method in this class.
+In addition, the ONC class provides some useful helper methods that involve multiple APIs to simplify the workflow.
 
 Create an ONC object to access this library's functionalities.
 
+```python
+from onc import ONC
+
+onc = ONC("YOUR_TOKEN_HERE")
+```
+
 ## Discovery methods
 
 Discovery methods can be used to search for available locations, deployments, device categories, devices, properties, and data products.
@@ -65,12 +73,20 @@ If the data product requested doesn't exist in our archive, it will be generated
 
 :::
 
-|                         Method                          |                  Description                   |                                               API Endpoint                                               |
-| :-----------------------------------------------------: | :--------------------------------------------: | :------------------------------------------------------------------------------------------------------: |
-|    [orderDataProduct](#onc.onc.ONC.orderDataProduct)    | Request, run, and download <br> a data product |                                                                                                          |
-|  [requestDataProduct](#onc.onc.ONC.requestDataProduct)  |             Request a data product             |  [/dataProductDelivery/request](https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/request)  |
-|      [runDataProduct](#onc.onc.ONC.runDataProduct)      |          Run a requested data product          |      [/dataProductDelivery/run](https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/run)      |
-| [downloadDataProduct](#onc.onc.ONC.downloadDataProduct) |            Download a data product             | [/dataProductDelivery/download](https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/download) |
+|                         Method                          |                  Description                  |                                               API Endpoint                                               |
+| :-----------------------------------------------------: | :-------------------------------------------: | :------------------------------------------------------------------------------------------------------: |
+|  [requestDataProduct](#onc.onc.ONC.requestDataProduct)  |            Request a data product             |  [/dataProductDelivery/request](https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/request)  |
+|    [checkDataProduct](#onc.onc.ONC.checkDataProduct)    | Check status of a <br> requested data product |   [/dataProductDelivery/status](https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/status)   |
+|      [runDataProduct](#onc.onc.ONC.runDataProduct)      |         Run a requested data product          |      [/dataProductDelivery/run](https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/run)      |
+|   [cancelDataProduct](#onc.onc.ONC.cancelDataProduct)   |         Cancel a running data product         |   [/dataProductDelivery/cancel](https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/cancel)   |
+|  [restartDataProduct](#onc.onc.ONC.restartDataProduct)  |       Restart a cancelled data product        |  [/dataProductDelivery/restart](https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/restart)  |
+| [downloadDataProduct](#onc.onc.ONC.downloadDataProduct) |            Download a data product            | [/dataProductDelivery/download](https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/download) |
+
+Helper methods are listed below.
+
+|                      Method                       |                Description                |
+| :-----------------------------------------------: | :---------------------------------------: |
+| [orderDataProduct](#onc.onc.ONC.orderDataProduct) | Request, run, and download a data product |
 
 ## Near real-time data access methods
 
@@ -103,6 +119,12 @@ Use the _allPages_ parameter to automatically download all pages required for yo
 | [getDirectRawByLocation](#onc.onc.ONC.getDirectRawByLocation) |  Returns raw data <br> from a specific location and device category   |    [/rawdata/location](https://data.oceannetworks.ca/OpenAPI#get-/rawdata/location)    |
 |   [getDirectRawByDevice](#onc.onc.ONC.getDirectRawByDevice)   |                Returns raw data from a specific device                |      [/rawdata/device](https://data.oceannetworks.ca/OpenAPI#get-/rawdata/device)      |
 
+Helper methods are listed below.
+
+|                            Method                             |                                      Description                                      |
+| :-----------------------------------------------------------: | :-----------------------------------------------------------------------------------: |
+| [getSensorCategoryCodes](#onc.onc.ONC.getSensorCategoryCodes) | Returns a list of sensor category codes <br> prior to querying the scalardata service |
+
 ## Archive file download methods
 
 These methods allow users to directly download previously generated data product files from our archive.
@@ -131,4 +153,9 @@ Due to security regulations, some very recent files (e.g. hydrophone.wav files i
 | [getListByLocation](#onc.onc.ONC.getListByLocation) | Returns a list of available archive files <br> from a specific location and device category | [/archivefile/location](https://data.oceannetworks.ca/OpenAPI#get-/archivefile/location) |
 |   [getListByDevice](#onc.onc.ONC.getListByDevice)   |            Returns a list of available archive files <br> from a specific device            |   [/archivefile/device](https://data.oceannetworks.ca/OpenAPI#get-/archivefile/device)   |
 |           [getFile](#onc.onc.ONC.getFile)           |                                  Download an archive file                                   | [/archivefile/download](https://data.oceannetworks.ca/OpenAPI#get-/archivefile/download) |
-|    [getDirectFiles](#onc.onc.ONC.getDirectFiles)    |           Download a list of archived files <br> that match the filters provided            |                                                                                          |
+
+Helper methods are listed below.
+
+|                    Method                     |                            Description                            |
+| :-------------------------------------------: | :---------------------------------------------------------------: |
+| [getDirectFiles](#onc.onc.ONC.getDirectFiles) | Download a list of archived files that match the filters provided |

src/onc/modules/_OncDelivery.py

@@ -74,6 +74,13 @@ def requestDataProduct(self, filters: dict):
         self._printProductRequest(response)
         return response
 
+    def checkDataProduct(self, dpRequestId: int):
+        url = f"{self._config('baseUrl')}api/dataProductDelivery/status"
+        filters = {
+            "dpRequestId": dpRequestId,
+        }
+        return self._doRequest(url, filters)
+
     def runDataProduct(self, dpRequestId: int, waitComplete: bool):
         """
         Run a product request.
@@ -83,6 +90,9 @@ def runDataProduct(self, dpRequestId: int, waitComplete: bool):
         """
         status = ""
         log = _PollLog(True)
+        print(
+            f"To cancel the running data product, run 'onc.cancelDataProduct({dpRequestId})'"  # noqa: E501
+        )
         url = f"{self._config('baseUrl')}api/dataProductDelivery"
         runResult = {"runIds": [], "fileCount": 0, "runTime": 0, "requestCount": 0}
 
@@ -108,6 +118,8 @@ def runDataProduct(self, dpRequestId: int, waitComplete: bool):
             if waitComplete:
                 status = data[0]["status"]
                 log.logMessage(data)
+                if status == "cancelled":
+                    break
                 if code != 200:
                     sleep(self.pollPeriod)
             else:
@@ -128,6 +140,24 @@ def runDataProduct(self, dpRequestId: int, waitComplete: bool):
 
         return runResult
 
+    def cancelDataProduct(self, dpRequestId: int):
+        url = f"{self._config('baseUrl')}api/dataProductDelivery/cancel"
+        filters = {
+            "dpRequestId": dpRequestId,
+        }
+        return self._doRequest(url, filters)
+
+    def restartDataProduct(self, dpRequestId: int, waitComplete: bool):
+        url = f"{self._config('baseUrl')}api/dataProductDelivery/restart"
+        filters = {
+            "dpRequestId": dpRequestId,
+        }
+        data = self._doRequest(url, filters)
+        if waitComplete:
+            return self.runDataProduct(dpRequestId, True)
+        else:
+            return data
+
     def downloadDataProduct(
         self,
         runId: int,

src/onc/modules/_OncRealTime.py

@@ -1,3 +1,5 @@
+from typing import Any
+
 from ._MultiPage import _MultiPage
 from ._OncService import _OncService
 
@@ -46,9 +48,16 @@ def getDirectRawByDevice(self, filters: dict, allPages: bool):
         """
         return self._getDirectAllPages(filters, "rawdata", "getByDevice", allPages)
 
+    def getSensorCategoryCodes(self, filters: dict):
+        updated_filters = filters | {"returnOptions": "excludeScalarData"}
+        if "deviceCode" in filters:
+            return self.getDirectByDevice(updated_filters, False)["sensorData"]
+        else:
+            return self.getDirectByLocation(updated_filters, False)["sensorData"]
+
     def _getDirectAllPages(
         self, filters: dict, service: str, method: str, allPages: bool
-    ):
+    ) -> Any:
         """
         Keeps downloading all scalar or raw data pages until finished.
 

src/onc/modules/_OncService.py

@@ -39,6 +39,7 @@ def _doRequest(self, url: str, filters: dict | None = None, getTime: bool = Fals
         """
         if filters is None:
             filters = {}
+        filters["token"] = self._config("token")
         timeout = self._config("timeout")
 
         txtParams = parse.unquote(parse.urlencode(filters))

src/onc/onc.py

@@ -782,9 +782,80 @@ def orderDataProduct(
     def requestDataProduct(self, filters: dict):
         return self.delivery.requestDataProduct(filters)
 
+    def checkDataProduct(self, dpRequestId: int):
+        """
+        Check status of a requested data product.
+
+        The API endpoint is ``/dataProductDelivery/status``.
+
+        See https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/status
+        for usage.
+
+        Parameters
+        ----------
+        dpRequestId : int
+            A dpRequestId returned from calling ``requestDataProduct``.
+
+        Returns
+        -------
+        dict
+            API response.
+        """
+        return self.delivery.checkDataProduct(dpRequestId)
+
     def runDataProduct(self, dpRequestId: int, waitComplete: bool = True):
         return self.delivery.runDataProduct(dpRequestId, waitComplete)
 
+    def cancelDataProduct(self, dpRequestId: int):
+        """
+        Cancel a running data product.
+
+        The API endpoint is ``/dataProductDelivery/cancel``.
+
+        See https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/cancel
+        for usage.
+
+        Parameters
+        ----------
+        dpRequestId : int
+            A dpRequestId returned from calling ``requestDataProduct``.
+
+        Returns
+        -------
+        list of dict
+            API response. Each status returned in the list is a dict with the following structure.
+
+            - dpRunId: int
+            - status: str
+        """  # noqa: E501
+        return self.delivery.cancelDataProduct(dpRequestId)
+
+    def restartDataProduct(self, dpRequestId: int, waitComplete: bool = True):
+        """
+        Restart a cancelled data product.
+
+        The API endpoint is ``/dataProductDelivery/restart``.
+
+        Restart searches cancelled by calling the ``cancelDataProduct`` method.
+
+        See https://data.oceannetworks.ca/OpenAPI#get-/dataProductDelivery/restart
+        for usage.
+
+        Parameters
+        ----------
+        dpRequestId : int
+            A dpRequestId returned from calling ``requestDataProduct``.
+
+        Returns
+        -------
+        list of dict
+            API response. Each status returned in the list is a dict with the following structure.
+
+            - dpRunId: int
+            - status: str
+        """  # noqa: E501
+        return self.delivery.restartDataProduct(dpRequestId, waitComplete)
+
     def downloadDataProduct(
         self,
         runId: int,
@@ -815,6 +886,71 @@ def getDirectRawByLocation(self, filters: dict = None, allPages: bool = False):
     def getDirectRawByDevice(self, filters: dict = None, allPages: bool = False):
         return self.realTime.getDirectRawByDevice(filters, allPages)
 
+    def getSensorCategoryCodes(self, filters: dict):
+        """
+        Return a list of sensor category codes.
+
+        A helper method for narrowing down the sensorCategoryCodes that are of interest
+        prior to the use of the scalardata service.
+
+        Parameters
+        ----------
+        filters : dict
+            Query string parameters in the API request.
+            Use the same filters for calling ``getDirectByLocation`` or ``getDirectByDevice``.
+
+        Returns
+        -------
+        list of dict
+            API response. Each sensor category code returned in the list is a dict with the following structure.
+
+            - outputFormat: str
+            - sensorCategoryCode: str
+            - sensorCode: str
+            - sensorName: str
+            - unitOfMeasure: str
+
+        Examples
+        --------
+        >>> params = {
+        ...     "locationCode": "NCBC",
+        ...     "deviceCategoryCode": "BPR",
+        ...     "propertyCode": "seawatertemperature,totalpressure",
+        ... }  # doctest: +SKIP
+        >>> onc.getSensorCategoryCodes(params)  # doctest: +SKIP
+        [
+            {
+                "outputFormat": "array",
+                "sensorCategoryCode": "pressure",
+                "sensorCode": "Pressure",
+                "sensorName": "Seafloor Pressure",
+                "unitOfMeasure": "decibar",
+            },
+            {
+                "outputFormat": "array",
+                "sensorCategoryCode": "temperature",
+                "sensorCode": "Temperature",
+                "sensorName": "Housing Temperature",
+                "unitOfMeasure": "C",
+            },
+            {
+                "outputFormat": "array",
+                "sensorCategoryCode": "temperature1",
+                "sensorCode": "temperature1",
+                "sensorName": "Temperature",
+                "unitOfMeasure": "C",
+            },
+            {
+                "outputFormat": "array",
+                "sensorCategoryCode": "temperature2",
+                "sensorCode": "temperature2",
+                "sensorName": "P-Sensor Temperature",
+                "unitOfMeasure": "C",
+            },
+        ]
+        """  # noqa: E501
+        return self.realTime.getSensorCategoryCodes(filters)
+
     # Archive file methods
 
     def getListByLocation(self, filters: dict = None, allPages: bool = False):

tests/data_product_delivery/conftest.py

@@ -13,3 +13,17 @@ def params() -> dict:
         "dpo_qualityControl": 1,
         "dpo_resample": "none",
     }
+
+
+@pytest.fixture()
+def expected_keys_download_results() -> dict:
+    return {
+        "url": str,
+        "status": str,
+        "size": int,
+        "file": str,
+        "index": str,
+        "downloaded": bool,
+        "requestCount": int,
+        "fileDownloadTime": float,
+    }

tests/data_product_delivery/test_data_product_delivery_cancel.py

@@ -0,0 +1,7 @@
+import pytest
+import requests
+
+
+def test_invalid_request_id(requester):
+    with pytest.raises(requests.HTTPError, match=r"API Error 127"):
+        requester.cancelDataProduct(1234567890)

tests/data_product_delivery/test_data_product_delivery_order.py

@@ -2,20 +2,6 @@
 import requests
 
 
-@pytest.fixture()
-def expected_keys_download_results() -> dict:
-    return {
-        "url": str,
-        "status": str,
-        "size": int,
-        "file": str,
-        "index": str,
-        "downloaded": bool,
-        "requestCount": int,
-        "fileDownloadTime": float,
-    }
-
-
 def test_invalid_param_value(requester, params):
     params_invalid_param_value = params | {"dataProductCode": "XYZ123"}
     with pytest.raises(requests.HTTPError, match=r"API Error 127"):
@@ -83,25 +69,3 @@ def test_valid_results_only(requester, params, expected_keys_download_results, u
     util.assert_dict_key_types(
         data["downloadResults"][0], expected_keys_download_results
     )
-
-
-def test_valid_manual(requester, params, expected_keys_download_results, util):
-    request_id = requester.requestDataProduct(params)["dpRequestId"]
-    run_id = requester.runDataProduct(request_id)["runIds"][0]
-    data = requester.downloadDataProduct(run_id)
-
-    assert (
-        len(data) == 3
-    ), "The first two are png files, and the third one is the metadata."
-
-    assert data[0]["status"] == "complete"
-    assert data[0]["index"] == "1"
-    assert data[0]["downloaded"] is True
-
-    assert data[2]["status"] == "complete"
-    assert data[2]["index"] == "meta"
-    assert data[2]["downloaded"] is True
-
-    assert util.get_download_files_num(requester) == 3
-
-    util.assert_dict_key_types(data[0], expected_keys_download_results)

tests/data_product_delivery/test_data_product_delivery_restart.py

@@ -0,0 +1,7 @@
+import pytest
+import requests
+
+
+def test_invalid_request_id(requester):
+    with pytest.raises(requests.HTTPError, match=r"API Error 127"):
+        requester.restartDataProduct(1234567890)