From f9ef107cfa8d9dc4fd2e28c949e5c84bb3e07594 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 13:13:27 -0600 Subject: [PATCH 01/35] Doc changes --- docs/source/data-access-api/index.rst | 18 ++++++++++++++++-- docs/source/data-access-api/openapi.yml | 7 +++++++ 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 3daf1d246..e3d1ad22e 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -11,14 +11,21 @@ The SDC provides a REST API that allows users to upload and download files, as well as query for file metadata. The following documentation describes the various endpoints that are supported and how to use them. -The API can be accessed from the following URL: https://api.dev.imap-mission.com - +The API can be accessed from the following URL [WIP]: https://api.dev.imap-mission.com +*Note: Several sections and links begin with* [WIP]. *As development on the API is ongoing, this indicates +that the full implementation of the functionality is yet to be completed.* .. openapi:: openapi.yml :group: :include: /upload +When uploading files to the API, ensure these files are stored properly in a ``data`` directory. Then, +ensure your working directory is one level above the ``data`` directory in order to properly upload files. + +[WIP] Certain ancillary files can also be uploaded to the API. For more specific information regarding these files, visit +`Ancillary Files `_ + **Example Usage:** .. code-block:: bash @@ -45,6 +52,13 @@ The API can be accessed from the following URL: https://api.dev.imap-mission.com :group: :include: /download +It is important to note that your working directory will be established as the default directory. I.e, the ``data`` +directory--which files are downloaded to--will automatically be placed in this file path. Choose your working directory +accordingly to suit your desires. + +When downloading a file from the API, different folders within the ``data`` directory will be made to better +organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts`` + **Example Usage:** .. code-block:: bash diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index bc2ba78be..1f6156245 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -52,6 +52,7 @@ paths: items: type: string + '/download/{filepath}': get: tags: @@ -153,6 +154,12 @@ paths: required: false schema: type: string + - in: query + name: [WIP] filename + description: The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). + required: false + schema: + type: string responses: '200': description: Successful query From d41584666c2dad73d392ac06a1e7e893f57944ed Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 13:43:33 -0600 Subject: [PATCH 02/35] Updating --- docs/source/data-access-api/openapi.yml | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index 1f6156245..f29ac4b12 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -144,7 +144,8 @@ paths: type: string - in: query name: version - description: The version of data product in the format ``vXX-YY`` (e.g. ``v01-01``). + description: The version of data product in the format ``vXX-YY`` (e.g. ``v01-01``). You can also choose to + query ``--version latest`` in order to receive the most recent version of a file. required: false schema: type: string @@ -160,6 +161,18 @@ paths: required: false schema: type: string + - in: query + name: [ WIP ] N days ago + description: Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). + required: false + schema: + type: string + - in: query + name: [ WIP ] today + description: Returns all files acquired in the last 24 hours. + required: false + schema: + type: string responses: '200': description: Successful query From 26135b0b1b4a6d0653ce8d078a61c56d395c3b78 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 13:49:15 -0600 Subject: [PATCH 03/35] Fixing --- docs/source/data-access-api/openapi.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index f29ac4b12..a0a6e70ba 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -156,19 +156,19 @@ paths: schema: type: string - in: query - name: [WIP] filename + name: "[WIP] filename" description: The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). required: false schema: type: string - in: query - name: [ WIP ] N days ago + name: "[WIP] N days ago" description: Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). required: false schema: type: string - in: query - name: [ WIP ] today + name: "[WIP] today" description: Returns all files acquired in the last 24 hours. required: false schema: From 4376bd7efa1dfa0d44a60ed524fea4bd224833d0 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 13:56:08 -0600 Subject: [PATCH 04/35] Fixing look --- docs/source/data-access-api/openapi.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index a0a6e70ba..f52ce1084 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -156,19 +156,19 @@ paths: schema: type: string - in: query - name: "[WIP] filename" + name: WIP filename description: The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). required: false schema: type: string - in: query - name: "[WIP] N days ago" + name: WIP N days ago description: Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). required: false schema: type: string - in: query - name: "[WIP] today" + name: WIP today description: Returns all files acquired in the last 24 hours. required: false schema: From 9e06ca6b8580a2ee40e6e25bd8d6f00c77a3160b Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 14:04:01 -0600 Subject: [PATCH 05/35] Playing with format --- docs/source/data-access-api/openapi.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index f52ce1084..9caf56113 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -156,19 +156,19 @@ paths: schema: type: string - in: query - name: WIP filename + name: filename (WIP) description: The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). required: false schema: type: string - in: query - name: WIP N days ago + name: N days ago (WIP) description: Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). required: false schema: type: string - in: query - name: WIP today + name: today (WIP) description: Returns all files acquired in the last 24 hours. required: false schema: From 5b9f4c6c156bee113e9e84104a6584d6e5b99bb4 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 14:10:20 -0600 Subject: [PATCH 06/35] Fixing again --- docs/source/data-access-api/openapi.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index 9caf56113..263ab16a3 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -156,19 +156,19 @@ paths: schema: type: string - in: query - name: filename (WIP) + name: filename WIP description: The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). required: false schema: type: string - in: query - name: N days ago (WIP) + name: N days ago WIP description: Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). required: false schema: type: string - in: query - name: today (WIP) + name: today WIP description: Returns all files acquired in the last 24 hours. required: false schema: From 1deb7aa392a5f0b6a237f7a447f0780a4d83604f Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 14:11:05 -0600 Subject: [PATCH 07/35] Fixing again --- docs/source/data-access-api/openapi.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index 263ab16a3..9435094a7 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -156,7 +156,7 @@ paths: schema: type: string - in: query - name: filename WIP + name: filename (WIP) description: The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). required: false schema: From 713f0dd578ae4f1a9bc395a9b3aaf1cf3ee50db1 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 14:17:31 -0600 Subject: [PATCH 08/35] Fixing again --- docs/source/data-access-api/index.rst | 2 ++ docs/source/data-access-api/openapi.yml | 6 +++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index e3d1ad22e..9a6f9a77c 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -78,6 +78,8 @@ organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_s :group: :include: /query +*Note: ``filename``, ``N days ago``, and ``today`` are all [WIP].* + **Example Usage:** .. code-block:: bash diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index 9435094a7..3da7d0abb 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -156,19 +156,19 @@ paths: schema: type: string - in: query - name: filename (WIP) + name: filename description: The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). required: false schema: type: string - in: query - name: N days ago WIP + name: N days ago description: Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). required: false schema: type: string - in: query - name: today WIP + name: today description: Returns all files acquired in the last 24 hours. required: false schema: From 8f3218ac34c87d67adbcc094d95ec4541f70b38d Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 14:21:55 -0600 Subject: [PATCH 09/35] Fixing again --- docs/source/data-access-api/index.rst | 2 +- docs/source/data-access-api/openapi.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 9a6f9a77c..55b5e3707 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -78,7 +78,7 @@ organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_s :group: :include: /query -*Note: ``filename``, ``N days ago``, and ``today`` are all [WIP].* +*Note:* ``filename``, ``N days ago``, and ``today`` *are all [WIP].* **Example Usage:** diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index 3da7d0abb..a1d11a963 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -162,7 +162,7 @@ paths: schema: type: string - in: query - name: N days ago + name: N_days_ago description: Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). required: false schema: From 923d74b942ff3fd947cc118000f5a5077813ec01 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 14:24:50 -0600 Subject: [PATCH 10/35] Fixing again --- docs/source/data-access-api/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 55b5e3707..429162555 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -78,7 +78,7 @@ organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_s :group: :include: /query -*Note:* ``filename``, ``N days ago``, and ``today`` *are all [WIP].* +*Note:* ``filename``, ``N days ago``*, and* ``today`` *are all [WIP].* **Example Usage:** From 41952cdaa0cc3ccbf3b1600a09825e2ba12a6a67 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 26 Jul 2024 14:38:24 -0600 Subject: [PATCH 11/35] Fixing again --- docs/source/data-access-api/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 429162555..402d60fa3 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -78,7 +78,7 @@ organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_s :group: :include: /query -*Note:* ``filename``, ``N days ago``*, and* ``today`` *are all [WIP].* +*Note:* ``filename``, ``N days ago`` *, and* ``today`` *are all [WIP].* **Example Usage:** From ecf5ca8cb84e2792f486169a9b5d0546731beefc Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Tue, 30 Jul 2024 15:03:07 -0600 Subject: [PATCH 12/35] Small change --- docs/source/data-access-api/index.rst | 2 -- docs/source/data-access-api/openapi.yml | 6 +++--- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 402d60fa3..e3d1ad22e 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -78,8 +78,6 @@ organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_s :group: :include: /query -*Note:* ``filename``, ``N days ago`` *, and* ``today`` *are all [WIP].* - **Example Usage:** .. code-block:: bash diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index a1d11a963..fe3323c4e 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -157,19 +157,19 @@ paths: type: string - in: query name: filename - description: The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). + description: [WIP] The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). required: false schema: type: string - in: query name: N_days_ago - description: Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). + description: [WIP] Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). required: false schema: type: string - in: query name: today - description: Returns all files acquired in the last 24 hours. + description: [WIP] Returns all files acquired in the last 24 hours. required: false schema: type: string From 73c5dce0dbc087ef2035a5dc6be3390e1748dee7 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Tue, 30 Jul 2024 15:26:53 -0600 Subject: [PATCH 13/35] More formatting --- docs/source/data-access-api/openapi.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index fe3323c4e..5f418ba40 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -157,19 +157,19 @@ paths: type: string - in: query name: filename - description: [WIP] The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). + description: WIP -- The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). required: false schema: type: string - in: query name: N_days_ago - description: [WIP] Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). + description: WIP -- Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). required: false schema: type: string - in: query name: today - description: [WIP] Returns all files acquired in the last 24 hours. + description: WIP -- Returns all files acquired in the last 24 hours. required: false schema: type: string From e9e3cedfbe4dfe86caf5a9b7830797113a1b679d Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Wed, 7 Aug 2024 09:30:52 -0600 Subject: [PATCH 14/35] PR updates --- docs/source/data-access-api/index.rst | 4 +-- docs/source/data-access-api/openapi.yml | 38 ++++++++++++------------- 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index e3d1ad22e..57cbc6bed 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -11,11 +11,11 @@ The SDC provides a REST API that allows users to upload and download files, as well as query for file metadata. The following documentation describes the various endpoints that are supported and how to use them. -The API can be accessed from the following URL [WIP]: https://api.dev.imap-mission.com - *Note: Several sections and links begin with* [WIP]. *As development on the API is ongoing, this indicates that the full implementation of the functionality is yet to be completed.* +The API can be accessed from the following URL [WIP]: https://api.dev.imap-mission.com + .. openapi:: openapi.yml :group: :include: /upload diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index 5f418ba40..5018fd61c 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -144,7 +144,7 @@ paths: type: string - in: query name: version - description: The version of data product in the format ``vXX-YY`` (e.g. ``v01-01``). You can also choose to + description: The version of data product in the format ``vNNN`` (e.g. ``v001``). You can also choose to query ``--version latest`` in order to receive the most recent version of a file. required: false schema: @@ -155,24 +155,24 @@ paths: required: false schema: type: string - - in: query - name: filename - description: WIP -- The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). - required: false - schema: - type: string - - in: query - name: N_days_ago - description: WIP -- Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). - required: false - schema: - type: string - - in: query - name: today - description: WIP -- Returns all files acquired in the last 24 hours. - required: false - schema: - type: string +# - in: query +# name: filename +# description: WIP -- The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). +# required: false +# schema: +# type: string +# - in: query +# name: N_days_ago +# description: WIP -- Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). +# required: false +# schema: +# type: string +# - in: query +# name: today +# description: WIP -- Returns all files acquired in the last 24 hours. +# required: false +# schema: +# type: string responses: '200': description: Successful query From 8d2a93a9d1477fc6834978b770343b4498582c83 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Wed, 7 Aug 2024 09:34:14 -0600 Subject: [PATCH 15/35] I changed my mind on format --- docs/source/data-access-api/openapi.yml | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/docs/source/data-access-api/openapi.yml b/docs/source/data-access-api/openapi.yml index 5018fd61c..18952ed46 100644 --- a/docs/source/data-access-api/openapi.yml +++ b/docs/source/data-access-api/openapi.yml @@ -155,24 +155,6 @@ paths: required: false schema: type: string -# - in: query -# name: filename -# description: WIP -- The full filename of the desired file (e.g. ``imap_mag_l1a_burst_20240105_20240105_v01-01.pkts``). -# required: false -# schema: -# type: string -# - in: query -# name: N_days_ago -# description: WIP -- Returns all files acquired in the last 'N' days (with 'N' representing a numeric value). -# required: false -# schema: -# type: string -# - in: query -# name: today -# description: WIP -- Returns all files acquired in the last 24 hours. -# required: false -# schema: -# type: string responses: '200': description: Successful query From 3211b725acd5b40a0cd31665c9097336cb9d00e0 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Wed, 7 Aug 2024 11:46:36 -0600 Subject: [PATCH 16/35] Inital Changes --- docs/source/data-access-api/index.rst | 120 ++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 57cbc6bed..fa5629801 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -16,6 +16,24 @@ that the full implementation of the functionality is yet to be completed.* The API can be accessed from the following URL [WIP]: https://api.dev.imap-mission.com +To Install +========== + .. code-block:: bash + + pip install imap-data-access + +Base Command Arguments +====================== + .. code-block:: bash + + imap-data-access -h # or + imap-data-access query # or + imap-data-access download # or + imap-data-access upload + + Add the -h flag to helo with any base command for more information. + + .. openapi:: openapi.yml :group: :include: /upload @@ -91,6 +109,108 @@ organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_s {"statusCode": 200, "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}, "body": [{"file_path": "imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-05.pkts", "instrument": "swe", "data_level": "l0", "descriptor": "sci", "start_date": "20240105", "end_date": "20240105", "version": "v00-05", "extension": "pkts"}]} {"statusCode": 400, "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}, "body": " is not a valid query parameter. Valid query parameters are: ['file_path', 'instrument', 'data_level', 'descriptor', 'start_date', 'end_date', 'version', 'extension']"} +Importing as a Package +====================== + .. code-block:: bash + + import imap_data_access + + # Search for files + results = imap_data_access.query(instrument="mag", data_level="l0") + # results is a list of dictionaries + # [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105','version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] + + # Download a file that was returned from the search + imap_data_access.download("imap/mag/l0/2024/01/imap_mag_l0_raw_202040101_v001.pkts") + + # Upload a calibration file that exists locally + imap_data_access.upload("imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf") + +Configuration +============= +**Data Directory** + +The folder structure for data files within the IMAP SDC is rigidly defined, so the data access will mimic that structure to make sure all data is stored in the same hierarchical structure as the SDC. This will enable seamless transition between a user's local system and the SDC. This is only used for downloads. +A user's root data location can be specified as an environment variable IMAP_DATA_DIR or through a configuration dictionary within the package itself imap_data_access.config["DATA_DIR"]. If the IMAP_DATA_DIR variable is not set, the program defaults to the user's current working directory + data/. +The following is the directory structure the IMAP SDC uses. + + .. code-block:: bash + + / + imap/ + / + / + / + / + + +for example, with ``IMAP_DATA_DIR=/data:`` + + .. code-block:: bash + + /data/ + imap/ + swe/ + l0/ + 2024/ + 01/ + imap_swe_l0_sci_20240105_v001.pkts + +**Data Access URL** + +To change the default URL that the package accesses, you can set the environment variable ``IMAP_DATA_ACCESS_URL`` or within the package ``imap_data_access.config["DATA_ACCESS_URL"]``. The default is the development server https://api.dev.imap-mission.com. + +Troubleshooting +=============== +**Network Issues** + +**SSL** + +If you encounter SSL errors similar to the following: + + .. code-block:: bash + + urllib.error.URLError: + +That generally means the Python environment you're using is not finding your system's root certificates properly. This means you need to tell Python how to find those certificates with the following potential solutions. + +#. Upgrade the certifi package + + .. code-block:: bash + + pip install --upgrade certifi + +#. Install system certificates -- Depending on the Python version you installed the program with the command will look something like this: + + .. code-block:: bash + + /Applications/Python\ 3.10/Install\ Certificates.command + +**HTTP Error 502: BadGateway** + +This could mean that the service is temporarily down. If you continue to encounter this, reach out to the IMAP SDC at imap-sdc@lasp.colorado.edu. + +**FileNotFoundError** + +This could mean that the local data directory is not set up with the same paths as the SDC. See the data directory section for an example of how to set this up. + +File Validation +=============== + +This package validates filenames and paths to check they follow our standards, as defined by the filename conventions. There is also a class available for use by other packages to create filepaths and filenames that follow the IMAP SDC conventions. + +To use this class, use ``imap_data_access.ScienceFilePath``. + +Usage: + + .. code-block:: bash + + science_file = imap_data_access.ScienceFilePath("imap_swe_l0_sci_20240101_v001.pkts") + + # Filepath = /imap/swe/l0/2024/01/imap_swe_l0_sci_20240101_v001.pkts + filepath = science_file.construct_path() + + Other pages =========== From dc3f9372e542ed5104338ddefc31cb09e3b3dd12 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Wed, 7 Aug 2024 13:25:15 -0600 Subject: [PATCH 17/35] Some small changes --- docs/source/data-access-api/index.rst | 29 +++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index fa5629801..53004c63f 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -31,7 +31,7 @@ Base Command Arguments imap-data-access download # or imap-data-access upload - Add the -h flag to helo with any base command for more information. +Add the -h flag to helo with any base command for more information. .. openapi:: openapi.yml @@ -39,16 +39,23 @@ Base Command Arguments :include: /upload When uploading files to the API, ensure these files are stored properly in a ``data`` directory. Then, -ensure your working directory is one level above the ``data`` directory in order to properly upload files. +ensure your working directory is one level above the ``data`` directory in order to properly upload files (see data directory section for more detail). [WIP] Certain ancillary files can also be uploaded to the API. For more specific information regarding these files, visit `Ancillary Files `_ **Example Usage:** -.. code-block:: bash + .. code-block:: bash - curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/upload/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts + $ imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf + Successfully uploaded the file to the IMAP SDC + +See also: + + .. code-block:: bash + + curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/upload/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts **Possible Responses:** @@ -79,6 +86,13 @@ organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_s **Example Usage:** +.. code-block:: bash + + imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf + Successfully uploaded the file to the IMAP SDC + +See also: + .. code-block:: bash curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/download/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts @@ -98,6 +112,13 @@ organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_s **Example Usage:** +.. code-block:: bash + + imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json + [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] + +See also: + .. code-block:: bash curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/query?instrument=swe&data_level=l0&descriptor=sci&start_date=20240105&end_date=20240105&extension=pkts From d46c20ad7cb19d72fce58b15b8be981b595dfe27 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Wed, 7 Aug 2024 15:14:04 -0600 Subject: [PATCH 18/35] Little changes --- docs/source/data-access-api/index.rst | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 53004c63f..ea3dc3ebf 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -31,25 +31,24 @@ Base Command Arguments imap-data-access download # or imap-data-access upload -Add the -h flag to helo with any base command for more information. +Add the -h flag with any base command for more information on use and functionality. .. openapi:: openapi.yml :group: :include: /upload -When uploading files to the API, ensure these files are stored properly in a ``data`` directory. Then, -ensure your working directory is one level above the ``data`` directory in order to properly upload files (see data directory section for more detail). +When uploading files to the API, ensure these files are stored properly in a ``data`` directory (see the Data Directory section below for more information). Then, +ensure your working directory is one level above ``data`` in order to properly upload files. -[WIP] Certain ancillary files can also be uploaded to the API. For more specific information regarding these files, visit +[WIP] Certain ancillary files can also be uploaded to the API. For more specific information regarding this, visit `Ancillary Files `_ **Example Usage:** .. code-block:: bash - $ imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf - Successfully uploaded the file to the IMAP SDC + imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf See also: @@ -78,18 +77,16 @@ See also: :include: /download It is important to note that your working directory will be established as the default directory. I.e, the ``data`` -directory--which files are downloaded to--will automatically be placed in this file path. Choose your working directory +directory ( to which files are downloaded) will automatically be placed in this file path. Choose your working directory accordingly to suit your desires. -When downloading a file from the API, different folders within the ``data`` directory will be made to better -organize the files. See the example file path: ``data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts`` +When downloading a file from the API, different folders within the ``data`` directory will be created for proper storage. See the example file path: ``data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts`` **Example Usage:** .. code-block:: bash imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf - Successfully uploaded the file to the IMAP SDC See also: @@ -115,6 +112,7 @@ See also: .. code-block:: bash imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json + # The following line is returned: [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] See also: @@ -132,6 +130,10 @@ See also: Importing as a Package ====================== +Imap data access can also be imported and used as a python package if desired. + +**Example Usage:** + .. code-block:: bash import imap_data_access From 304e779a2282eb021465b0f27d77e975682960c4 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Wed, 7 Aug 2024 16:12:28 -0600 Subject: [PATCH 19/35] Fixing docs error --- docs/source/data-access-api/index.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index c97ba0d7f..c45402651 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -32,7 +32,6 @@ Base Command Arguments imap-data-access upload Add the -h flag with any base command for more information on use and functionality. -======= .. openapi:: openapi.yml :group: From 2af4df7670e1ad0da98cee5a62a44644c18ddeb9 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Mon, 12 Aug 2024 11:43:01 -0600 Subject: [PATCH 20/35] Order changes --- docs/source/data-access-api/index.rst | 189 +++++++++++++++----------- 1 file changed, 113 insertions(+), 76 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index c45402651..a2a30e79c 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -7,23 +7,33 @@ The `imap-data-access `_ +.. openapi:: openapi.yml + :group: + :include: /query **Example Usage:** - .. code-block:: bash - - imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf - -See also: - - .. code-block:: bash +.. code-block:: bash - curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/upload/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts + imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json + # The following line is returned: + [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] -**Possible Responses:** - -.. code-block:: json - - {"statusCode": 200, "body": "https://sds-data-.s3.amazon.com/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts?"} - {"statusCode": 400, "body": "Invalid filename. Expected - ______."} - {"statusCode": 400, "body": "Invalid mission."} - {"statusCode": 400, "body": "Invalid instrument. Please choose from {'ultra-45', 'codice', 'glows', 'hit', 'lo', 'mag', 'swe', 'hi-45', 'idex', 'ultra-90', 'hi-90', 'swapi'}"} - {"statusCode": 400, "body": "Invalid data level. Please choose from {'l0', 'l1', 'l1a', 'l1b', 'l1c', 'l1d', 'l2'}"} - {"statusCode": 400, "body": "Invalid start date format. Please use YYYYMMDD format."} - {"statusCode": 400, "body": "Invalid end date format. Please use YYYYMMDD format."} - {"statusCode": 400, "body": "Invalid version format. Please use vxx-xx format."} - {"statusCode": 400, "body": "Invalid extension. Extension should be pkts for data level l0 and cdf for data level higher than l0"} - {"statusCode": 409, "body": "https://sds-data-.s3.amazon.com/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts already exists."} +Download a file +--------------- +You can download files form the database using the command ``download``. .. openapi:: openapi.yml :group: @@ -88,47 +82,28 @@ organize the downloaded files. See the example path: ``data/imap/swe/l0/2024/01/ imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf -See also: - -.. code-block:: bash - - curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/download/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts - -**Possible Responses:** - -.. code-block:: json - - {"statusCode": 302, "headers": {"Content-Type": "text/html", "Location": "s3://sds-data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01"}, "body": {"download_url": "s3://sds-data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01"}} - {"statusCode": 400, "body": "No file requested for download. Please provide a filename in the path. Eg. /download/path/to/file/filename.pkts"} - {"statusCode": 404, "body": "File not found, make sure you include the full path to the file in the request, e.g. /download/path/to/file/filename.pkts"} +Upload a file +------------- +Similarly, you may upload a file to the data base using the command ``upload``. .. openapi:: openapi.yml :group: - :include: /query - -**Example Usage:** - -.. code-block:: bash - - imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json - # The following line is returned: - [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] - -See also: + :include: /upload -.. code-block:: bash +When uploading files to the API, ensure these files are stored properly in a ``data`` directory (see the Data Directory section below for more information). Then, +ensure your working directory is one level above ``data`` in order to properly upload files. - curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/query?instrument=swe&data_level=l0&descriptor=sci&start_date=20240105&end_date=20240105&extension=pkts +[WIP] Certain ancillary files can also be uploaded to the API. For more specific information regarding these files, visit +`Ancillary Files `_ -**Possible Responses:** +**Example Usage:** -.. code-block:: json + .. code-block:: bash - {"statusCode": 200, "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}, "body": [{"file_path": "imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-05.pkts", "instrument": "swe", "data_level": "l0", "descriptor": "sci", "start_date": "20240105", "end_date": "20240105", "version": "v00-05", "extension": "pkts"}]} - {"statusCode": 400, "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}, "body": " is not a valid query parameter. Valid query parameters are: ['file_path', 'instrument', 'data_level', 'descriptor', 'start_date', 'end_date', 'version', 'extension']"} + imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf -Importing as a Package +Importing as a package ====================== Imap data access can also be imported and used as a python package if desired. @@ -151,7 +126,8 @@ Imap data access can also be imported and used as a python package if desired. Configuration ============= -**Data Directory** +Data Directory +-------------- The folder structure for data files within the IMAP SDC is rigidly defined, so the data access will mimic that structure to make sure all data is stored in the same hierarchical structure as the SDC. This will enable seamless transition between a user's local system and the SDC. This is only used for downloads. A user's root data location can be specified as an environment variable IMAP_DATA_DIR or through a configuration dictionary within the package itself imap_data_access.config["DATA_DIR"]. If the IMAP_DATA_DIR variable is not set, the program defaults to the user's current working directory + data/. @@ -179,13 +155,30 @@ for example, with ``IMAP_DATA_DIR=/data:`` 01/ imap_swe_l0_sci_20240105_v001.pkts -**Data Access URL** +Data Access URL +--------------- To change the default URL that the package accesses, you can set the environment variable ``IMAP_DATA_ACCESS_URL`` or within the package ``imap_data_access.config["DATA_ACCESS_URL"]``. The default is the development server https://api.dev.imap-mission.com. +File Validation +=============== + +This package validates filenames and paths to check they follow our standards, as defined by the filename conventions. There is also a class available for use by other packages to create filepaths and filenames that follow the IMAP SDC conventions. +To use this class, use ``imap_data_access.ScienceFilePath``. + +Usage: + + .. code-block:: bash + + science_file = imap_data_access.ScienceFilePath("imap_swe_l0_sci_20240101_v001.pkts") + + # Filepath = /imap/swe/l0/2024/01/imap_swe_l0_sci_20240101_v001.pkts + filepath = science_file.construct_path() + Troubleshooting =============== -**Network Issues** +Network Issues +-------------- **SSL** @@ -213,25 +206,69 @@ That generally means the Python environment you're using is not finding your sys This could mean that the service is temporarily down. If you continue to encounter this, reach out to the IMAP SDC at imap-sdc@lasp.colorado.edu. -**FileNotFoundError** +FileNotFoundError +----------------- This could mean that the local data directory is not set up with the same paths as the SDC. See the data directory section for an example of how to set this up. -File Validation -=============== - -This package validates filenames and paths to check they follow our standards, as defined by the filename conventions. There is also a class available for use by other packages to create filepaths and filenames that follow the IMAP SDC conventions. -To use this class, use ``imap_data_access.ScienceFilePath``. +REST API Specification +====================== +Upload +------ -Usage: +**Example Usage:** .. code-block:: bash - science_file = imap_data_access.ScienceFilePath("imap_swe_l0_sci_20240101_v001.pkts") + curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/upload/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts - # Filepath = /imap/swe/l0/2024/01/imap_swe_l0_sci_20240101_v001.pkts - filepath = science_file.construct_path() +**Possible Responses:** +.. code-block:: json + + {"statusCode": 200, "body": "https://sds-data-.s3.amazon.com/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts?"} + {"statusCode": 400, "body": "Invalid filename. Expected - ______."} + {"statusCode": 400, "body": "Invalid mission."} + {"statusCode": 400, "body": "Invalid instrument. Please choose from {'ultra-45', 'codice', 'glows', 'hit', 'lo', 'mag', 'swe', 'hi-45', 'idex', 'ultra-90', 'hi-90', 'swapi'}"} + {"statusCode": 400, "body": "Invalid data level. Please choose from {'l0', 'l1', 'l1a', 'l1b', 'l1c', 'l1d', 'l2'}"} + {"statusCode": 400, "body": "Invalid start date format. Please use YYYYMMDD format."} + {"statusCode": 400, "body": "Invalid end date format. Please use YYYYMMDD format."} + {"statusCode": 400, "body": "Invalid version format. Please use vxx-xx format."} + {"statusCode": 400, "body": "Invalid extension. Extension should be pkts for data level l0 and cdf for data level higher than l0"} + {"statusCode": 409, "body": "https://sds-data-.s3.amazon.com/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts already exists."} + + +Download +-------- + +**Example Usage:** + +.. code-block:: bash + + curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/download/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts + +**Possible Responses:** + +.. code-block:: json + + {"statusCode": 302, "headers": {"Content-Type": "text/html", "Location": "s3://sds-data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01"}, "body": {"download_url": "s3://sds-data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01"}} + {"statusCode": 400, "body": "No file requested for download. Please provide a filename in the path. Eg. /download/path/to/file/filename.pkts"} + {"statusCode": 404, "body": "File not found, make sure you include the full path to the file in the request, e.g. /download/path/to/file/filename.pkts"} + +Query +------ +**Example Usage:** + +.. code-block:: bash + + curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/query?instrument=swe&data_level=l0&descriptor=sci&start_date=20240105&end_date=20240105&extension=pkts + +**Possible Responses:** + +.. code-block:: json + + {"statusCode": 200, "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}, "body": [{"file_path": "imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-05.pkts", "instrument": "swe", "data_level": "l0", "descriptor": "sci", "start_date": "20240105", "end_date": "20240105", "version": "v00-05", "extension": "pkts"}]} + {"statusCode": 400, "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}, "body": " is not a valid query parameter. Valid query parameters are: ['file_path', 'instrument', 'data_level', 'descriptor', 'start_date', 'end_date', 'version', 'extension']"} Other pages =========== From a4e15e06b16cbee7467d37de1bc7e839fe08a497 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Mon, 12 Aug 2024 11:43:53 -0600 Subject: [PATCH 21/35] Order changes --- docs/source/data-access-api/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index a2a30e79c..b35d2b266 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -1,5 +1,6 @@ .. _data-access-api: + Data Access API =============== From 8952a5a8acd94fd1357e4161b83ea8abea8ac222 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Mon, 12 Aug 2024 11:47:02 -0600 Subject: [PATCH 22/35] Pre-commit --- docs/source/data-access-api/index.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index b35d2b266..a2a30e79c 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -1,6 +1,5 @@ .. _data-access-api: - Data Access API =============== From 1182258a85a19d555d6b33633bee9c83d66cdc88 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Wed, 14 Aug 2024 13:24:12 -0600 Subject: [PATCH 23/35] Small changes --- docs/source/data-access-api/index.rst | 20 ++------------------ 1 file changed, 2 insertions(+), 18 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index a2a30e79c..6d81ac08d 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -11,8 +11,6 @@ The SDC further provides a REST API that allows users to upload and download fil well as query for file metadata. The following documentation describes the various endpoints that are supported and how to use them. -The following documentation covers both functionalities starting with the CLI use, and moving to the REST API usage afterwards. - *Note: Several sections and links begin with* [WIP]. *As development on the API is ongoing, this indicates that the full implementation of the functionality is yet to be completed.* @@ -23,7 +21,7 @@ Command Line Utility To Install ---------- -Run the following command to use the data access API CLI. +Run the following command to use the API CLI. .. code-block:: bash @@ -32,7 +30,7 @@ Run the following command to use the data access API CLI. Base Command Arguments ---------------------- -The following are the base command arguments for the CLI: +The following are base command arguments for the CLI: .. code-block:: bash @@ -43,11 +41,6 @@ The following are the base command arguments for the CLI: Add the -h flag with any base command for more information on use and functionality. -Query to search for data ------------------------- - -You can search for files in the database using the command ``query`` and different parameters either alone or in conjunction with each other. - .. openapi:: openapi.yml :group: :include: /query @@ -60,10 +53,6 @@ You can search for files in the database using the command ``query`` and differe # The following line is returned: [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] -Download a file ---------------- - -You can download files form the database using the command ``download``. .. openapi:: openapi.yml :group: @@ -82,11 +71,6 @@ organize the downloaded files. See the example path: ``data/imap/swe/l0/2024/01/ imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf -Upload a file -------------- - -Similarly, you may upload a file to the data base using the command ``upload``. - .. openapi:: openapi.yml :group: :include: /upload From 510f5510259cf0fb8d064cb61e9ed52e26bebbbd Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Thu, 15 Aug 2024 15:51:16 -0600 Subject: [PATCH 24/35] More PR comments --- docs/source/data-access-api/index.rst | 62 +++++++++++++-------------- 1 file changed, 29 insertions(+), 33 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 6d81ac08d..e07d1fd72 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -17,9 +17,9 @@ that the full implementation of the functionality is yet to be completed.* The API can be accessed from the following URL [WIP]: https://api.dev.imap-mission.com Command Line Utility -==================== +-------------------- To Install ----------- +^^^^^^^^^^ Run the following command to use the API CLI. @@ -28,7 +28,7 @@ Run the following command to use the API CLI. pip install imap-data-access Base Command Arguments ----------------------- +^^^^^^^^^^^^^^^^^^^^^^ The following are base command arguments for the CLI: @@ -41,10 +41,8 @@ The following are base command arguments for the CLI: Add the -h flag with any base command for more information on use and functionality. -.. openapi:: openapi.yml - :group: - :include: /query - +Query +^^^^^ **Example Usage:** .. code-block:: bash @@ -53,11 +51,8 @@ Add the -h flag with any base command for more information on use and functional # The following line is returned: [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] - -.. openapi:: openapi.yml - :group: - :include: /download - +Download +^^^^^^^^ It is important to note that your working directory will be established as the default directory. I.e, the ``data`` directory (to which files are downloaded) will automatically be placed in this file path. Choose your working directory accordingly to suit your desires. @@ -71,10 +66,8 @@ organize the downloaded files. See the example path: ``data/imap/swe/l0/2024/01/ imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf -.. openapi:: openapi.yml - :group: - :include: /upload - +Upload +^^^^^^ When uploading files to the API, ensure these files are stored properly in a ``data`` directory (see the Data Directory section below for more information). Then, ensure your working directory is one level above ``data`` in order to properly upload files. @@ -88,7 +81,7 @@ ensure your working directory is one level above ``data`` in order to properly u imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf Importing as a package -====================== +---------------------- Imap data access can also be imported and used as a python package if desired. **Example Usage:** @@ -109,9 +102,9 @@ Imap data access can also be imported and used as a python package if desired. imap_data_access.upload("imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf") Configuration -============= -Data Directory -------------- +Data Directory +^^^^^^^^^^^^^^ The folder structure for data files within the IMAP SDC is rigidly defined, so the data access will mimic that structure to make sure all data is stored in the same hierarchical structure as the SDC. This will enable seamless transition between a user's local system and the SDC. This is only used for downloads. A user's root data location can be specified as an environment variable IMAP_DATA_DIR or through a configuration dictionary within the package itself imap_data_access.config["DATA_DIR"]. If the IMAP_DATA_DIR variable is not set, the program defaults to the user's current working directory + data/. @@ -140,12 +133,12 @@ for example, with ``IMAP_DATA_DIR=/data:`` imap_swe_l0_sci_20240105_v001.pkts Data Access URL ---------------- +^^^^^^^^^^^^^^^ To change the default URL that the package accesses, you can set the environment variable ``IMAP_DATA_ACCESS_URL`` or within the package ``imap_data_access.config["DATA_ACCESS_URL"]``. The default is the development server https://api.dev.imap-mission.com. File Validation -=============== +--------------- This package validates filenames and paths to check they follow our standards, as defined by the filename conventions. There is also a class available for use by other packages to create filepaths and filenames that follow the IMAP SDC conventions. To use this class, use ``imap_data_access.ScienceFilePath``. @@ -160,9 +153,9 @@ Usage: filepath = science_file.construct_path() Troubleshooting -=============== +--------------- Network Issues --------------- +^^^^^^^^^^^^^^ **SSL** @@ -191,14 +184,15 @@ That generally means the Python environment you're using is not finding your sys This could mean that the service is temporarily down. If you continue to encounter this, reach out to the IMAP SDC at imap-sdc@lasp.colorado.edu. FileNotFoundError ------------------ +^^^^^^^^^^^^^^^^^ This could mean that the local data directory is not set up with the same paths as the SDC. See the data directory section for an example of how to set this up. REST API Specification -====================== -Upload ------- +---------------------- +.. openapi:: openapi.yml + :group: + :include: /upload **Example Usage:** @@ -221,9 +215,9 @@ Upload {"statusCode": 400, "body": "Invalid extension. Extension should be pkts for data level l0 and cdf for data level higher than l0"} {"statusCode": 409, "body": "https://sds-data-.s3.amazon.com/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts already exists."} - -Download --------- +.. openapi:: openapi.yml + :group: + :include: /download **Example Usage:** @@ -239,8 +233,10 @@ Download {"statusCode": 400, "body": "No file requested for download. Please provide a filename in the path. Eg. /download/path/to/file/filename.pkts"} {"statusCode": 404, "body": "File not found, make sure you include the full path to the file in the request, e.g. /download/path/to/file/filename.pkts"} -Query ------- +.. openapi:: openapi.yml + :group: + :include: /query + **Example Usage:** .. code-block:: bash @@ -255,7 +251,7 @@ Query {"statusCode": 400, "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}, "body": " is not a valid query parameter. Valid query parameters are: ['file_path', 'instrument', 'data_level', 'descriptor', 'start_date', 'end_date', 'version', 'extension']"} Other pages -=========== +----------- .. toctree:: :maxdepth: 1 From e0858a52e292c9ac823f63a9a696cb93d8d99159 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 16 Aug 2024 11:11:34 -0600 Subject: [PATCH 25/35] Adding descriptions --- docs/source/data-access-api/index.rst | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index e07d1fd72..ecb049e6d 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -4,10 +4,10 @@ Data Access API =============== The `imap-data-access `_ -repository provides programmatic access and a command-line utility for +repository provides programmatic access, and a command-line utility for interacting with the API. It is the preferred way to use the API. -The SDC further provides a REST API that allows users to upload and download files, as +The SDC also provides a REST API that allows users to upload and download files as well as query for file metadata. The following documentation describes the various endpoints that are supported and how to use them. @@ -20,7 +20,6 @@ Command Line Utility -------------------- To Install ^^^^^^^^^^ - Run the following command to use the API CLI. .. code-block:: bash @@ -29,7 +28,6 @@ Run the following command to use the API CLI. Base Command Arguments ^^^^^^^^^^^^^^^^^^^^^^ - The following are base command arguments for the CLI: .. code-block:: bash @@ -43,6 +41,15 @@ Add the -h flag with any base command for more information on use and functional Query ^^^^^ + +To query for files, you can use several parameters: + + --instrument, + --data-level, + --descriptor, + etc. +Further information is found in in the ``query -h`` menu. You can use parameters alone, or in combination. + **Example Usage:** .. code-block:: bash @@ -53,8 +60,11 @@ Query Download ^^^^^^^^ + +To download files using the CLI tool, use the command ``download``. These will be placed in a ``data`` directory. + It is important to note that your working directory will be established as the default directory. I.e, the ``data`` -directory (to which files are downloaded) will automatically be placed in this file path. Choose your working directory +directory will automatically be placed in this file path. Choose your working directory accordingly to suit your desires. When downloading a file from the API, different folders within the ``data`` directory will be made to better @@ -64,10 +74,14 @@ organize the downloaded files. See the example path: ``data/imap/swe/l0/2024/01/ .. code-block:: bash - imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf + imap-data-access download imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts + Upload ^^^^^^ + +Similarly, files can be uploaded to the API using the command ``upload``. + When uploading files to the API, ensure these files are stored properly in a ``data`` directory (see the Data Directory section below for more information). Then, ensure your working directory is one level above ``data`` in order to properly upload files. From 1b76a68b5706b2eeaf3feafaa02d0fc6f27cf2ff Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 16 Aug 2024 11:15:42 -0600 Subject: [PATCH 26/35] Hopefully fixing my mistake --- docs/source/data-access-api/index.rst | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index ecb049e6d..4737ce262 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -20,6 +20,7 @@ Command Line Utility -------------------- To Install ^^^^^^^^^^ + Run the following command to use the API CLI. .. code-block:: bash @@ -28,6 +29,7 @@ Run the following command to use the API CLI. Base Command Arguments ^^^^^^^^^^^^^^^^^^^^^^ + The following are base command arguments for the CLI: .. code-block:: bash @@ -42,21 +44,17 @@ Add the -h flag with any base command for more information on use and functional Query ^^^^^ -To query for files, you can use several parameters: +To query for files, you can use several parameters: ``--instrument``, ``--data-level``, ``--descriptor``, etc. - --instrument, - --data-level, - --descriptor, - etc. Further information is found in in the ``query -h`` menu. You can use parameters alone, or in combination. **Example Usage:** -.. code-block:: bash + .. code-block:: bash - imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json - # The following line is returned: - [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] + imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json + # The following line is returned: + [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] Download ^^^^^^^^ @@ -72,9 +70,9 @@ organize the downloaded files. See the example path: ``data/imap/swe/l0/2024/01/ **Example Usage:** -.. code-block:: bash + .. code-block:: bash - imap-data-access download imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts + imap-data-access download imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts Upload From 56e626f78f6b74e86382c2123911ee80c2900a81 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Fri, 16 Aug 2024 11:20:52 -0600 Subject: [PATCH 27/35] More small changes --- docs/source/data-access-api/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 4737ce262..5c6997146 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -21,7 +21,7 @@ Command Line Utility To Install ^^^^^^^^^^ -Run the following command to use the API CLI. +Run the following command to use the API CLI: .. code-block:: bash From 8598f0bcda404cb8ad230ca76a3827d8392099da Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Thu, 29 Aug 2024 12:46:42 -0600 Subject: [PATCH 28/35] Testing ref links --- docs/source/data-access-api/index.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 5c6997146..524c89038 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -4,12 +4,11 @@ Data Access API =============== The `imap-data-access `_ -repository provides programmatic access, and a command-line utility for -interacting with the API. It is the preferred way to use the API. +repository provides a command-line utility and python package for +interacting with the API programmatically. It is the preferred way to use the API. -The SDC also provides a REST API that allows users to upload and download files as -well as query for file metadata. The following documentation describes the -various endpoints that are supported and how to use them. +Users may also download, upload, and query via the REST API directly through the browser, or `curl` commands. +The :ref:`REST API Specification ` section below describes the various endpoints that are supported and how to use them. *Note: Several sections and links begin with* [WIP]. *As development on the API is ongoing, this indicates that the full implementation of the functionality is yet to be completed.* From 93295a6a3fb52b94c7e3b12cc29e48583823dc5b Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Thu, 29 Aug 2024 12:53:45 -0600 Subject: [PATCH 29/35] Testing ref links AGAIN --- docs/source/data-access-api/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 524c89038..6dc93205f 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -8,7 +8,7 @@ repository provides a command-line utility and python package for interacting with the API programmatically. It is the preferred way to use the API. Users may also download, upload, and query via the REST API directly through the browser, or `curl` commands. -The :ref:`REST API Specification ` section below describes the various endpoints that are supported and how to use them. +The :ref:`REST API Specification ` section below describes the various endpoints that are supported and how to use them. *Note: Several sections and links begin with* [WIP]. *As development on the API is ongoing, this indicates that the full implementation of the functionality is yet to be completed.* From b1f13ec441f8ee12226f685bf17a6ec2f1428339 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Thu, 29 Aug 2024 12:59:26 -0600 Subject: [PATCH 30/35] More ref testing --- docs/source/data-access-api/index.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 6dc93205f..86718d2cb 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -8,7 +8,7 @@ repository provides a command-line utility and python package for interacting with the API programmatically. It is the preferred way to use the API. Users may also download, upload, and query via the REST API directly through the browser, or `curl` commands. -The :ref:`REST API Specification ` section below describes the various endpoints that are supported and how to use them. +The `REST API Specification`_ section below describes the various endpoints that are supported and how to use them. *Note: Several sections and links begin with* [WIP]. *As development on the API is ongoing, this indicates that the full implementation of the functionality is yet to be completed.* @@ -199,6 +199,8 @@ FileNotFoundError This could mean that the local data directory is not set up with the same paths as the SDC. See the data directory section for an example of how to set this up. +.. _rest-api-specification: + REST API Specification ---------------------- .. openapi:: openapi.yml From 33be89933ab85d34e35e8ff96016f6057a77053f Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Thu, 29 Aug 2024 13:36:31 -0600 Subject: [PATCH 31/35] Checking formatting --- docs/source/data-access-api/index.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 86718d2cb..c411b1166 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -53,7 +53,10 @@ Further information is found in in the ``query -h`` menu. You can use parameters imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json # The following line is returned: - [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] + [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', + 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, + {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', + 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] Download ^^^^^^^^ From 67600f16353ad3bb70c00e07c6a57877c6570ef1 Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Thu, 29 Aug 2024 13:43:53 -0600 Subject: [PATCH 32/35] Small changes --- docs/source/data-access-api/index.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index c411b1166..4d868d32a 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -61,14 +61,15 @@ Further information is found in in the ``query -h`` menu. You can use parameters Download ^^^^^^^^ -To download files using the CLI tool, use the command ``download``. These will be placed in a ``data`` directory. +To download files using the CLI tool, use the command ``download``. The downloaded files will be placed in a ``data`` directory. It is important to note that your working directory will be established as the default directory. I.e, the ``data`` directory will automatically be placed in this file path. Choose your working directory accordingly to suit your desires. When downloading a file from the API, different folders within the ``data`` directory will be made to better -organize the downloaded files. See the example path: ``data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts`` +organize the downloaded files. See the example path: ``data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts``. +The ``data`` directory and its structure is further described here: `Data Directory`_ **Example Usage:** @@ -117,6 +118,7 @@ Imap data access can also be imported and used as a python package if desired. Configuration -------------- +.. _data-directory: Data Directory ^^^^^^^^^^^^^^ From 51bba5b06d2f163d875ba04c151a0c3da13582ee Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Thu, 29 Aug 2024 13:46:34 -0600 Subject: [PATCH 33/35] Small change --- docs/source/data-access-api/index.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 4d868d32a..acf61830b 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -118,7 +118,9 @@ Imap data access can also be imported and used as a python package if desired. Configuration -------------- + .. _data-directory: + Data Directory ^^^^^^^^^^^^^^ From a42e4f96e255f0af1547327e430546f8115acd1c Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Thu, 29 Aug 2024 16:24:54 -0600 Subject: [PATCH 34/35] Final comment changes --- docs/source/data-access-api/index.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index acf61830b..4fdb50295 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -108,7 +108,10 @@ Imap data access can also be imported and used as a python package if desired. # Search for files results = imap_data_access.query(instrument="mag", data_level="l0") # results is a list of dictionaries - # [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105','version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] + # [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', + 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105','version': 'v001', 'extension': 'pkts'}, + {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', + 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}] # Download a file that was returned from the search imap_data_access.download("imap/mag/l0/2024/01/imap_mag_l0_raw_202040101_v001.pkts") @@ -125,7 +128,7 @@ Data Directory ^^^^^^^^^^^^^^ The folder structure for data files within the IMAP SDC is rigidly defined, so the data access will mimic that structure to make sure all data is stored in the same hierarchical structure as the SDC. This will enable seamless transition between a user's local system and the SDC. This is only used for downloads. -A user's root data location can be specified as an environment variable IMAP_DATA_DIR or through a configuration dictionary within the package itself imap_data_access.config["DATA_DIR"]. If the IMAP_DATA_DIR variable is not set, the program defaults to the user's current working directory + data/. +A user's root data location can be specified as an environment variable ``IMAP_DATA_DIR`` or through a configuration dictionary within the package itself (``imap_data_access.config["DATA_DIR"]``). If the ``IMAP_DATA_DIR`` variable is not set, the program defaults to the user's current working directory + ``data/``. The following is the directory structure the IMAP SDC uses. .. code-block:: bash @@ -153,12 +156,12 @@ for example, with ``IMAP_DATA_DIR=/data:`` Data Access URL ^^^^^^^^^^^^^^^ -To change the default URL that the package accesses, you can set the environment variable ``IMAP_DATA_ACCESS_URL`` or within the package ``imap_data_access.config["DATA_ACCESS_URL"]``. The default is the development server https://api.dev.imap-mission.com. +To change the default URL that the package accesses, you can set the environment variable ``IMAP_DATA_ACCESS_URL`` or within the package ``imap_data_access.config["DATA_ACCESS_URL"]``. The default is the development server (``https://api.dev.imap-mission.com``). File Validation --------------- -This package validates filenames and paths to check they follow our standards, as defined by the filename conventions. There is also a class available for use by other packages to create filepaths and filenames that follow the IMAP SDC conventions. +This package validates filenames and paths to check they follow our standards, as defined by the [filename conventions](https://imap-processing.readthedocs.io/en/latest/development-guide/style-guide/naming-conventions.html). There is also a class available for use by other packages to create filepaths and filenames that follow the IMAP SDC conventions. To use this class, use ``imap_data_access.ScienceFilePath``. Usage: From 9e0bba6f1fccd1161d447c103615bea3ec9bd92c Mon Sep 17 00:00:00 2001 From: Ana Manica Date: Thu, 29 Aug 2024 16:50:12 -0600 Subject: [PATCH 35/35] Final touches --- docs/source/data-access-api/index.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/source/data-access-api/index.rst b/docs/source/data-access-api/index.rst index 4fdb50295..68de9a1eb 100644 --- a/docs/source/data-access-api/index.rst +++ b/docs/source/data-access-api/index.rst @@ -7,8 +7,8 @@ The `imap-data-access `_. There is also a class available for use by other packages to create filepaths and filenames that follow the IMAP SDC conventions. To use this class, use ``imap_data_access.ScienceFilePath``. Usage: @@ -194,7 +194,7 @@ That generally means the Python environment you're using is not finding your sys pip install --upgrade certifi -#. Install system certificates -- Depending on the Python version you installed the program with the command will look something like this: +#. Install system certificates -- Depending on the Python version you installed the program with, the command will look something like this: .. code-block:: bash @@ -207,7 +207,7 @@ This could mean that the service is temporarily down. If you continue to encount FileNotFoundError ^^^^^^^^^^^^^^^^^ -This could mean that the local data directory is not set up with the same paths as the SDC. See the data directory section for an example of how to set this up. +This could mean that the local data directory is not set up with the same paths as the SDC. See the `Data Directory`_ section for an example of how to set this up. .. _rest-api-specification: