Merge pull request #34 from semuconsulting/RC-1.0.5

RC 1.0.5

.github/workflows/checkpr.yml

@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.9, "3.10", "3.11", "3.12", "3.13.0-rc.3"]
+        python-version: [3.9, "3.10", "3.11", "3.12", "3.13"]
 
     steps:
       - uses: actions/checkout@v4

.github/workflows/main.yml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.9, "3.10", "3.11", "3.12", "3.13.0-rc.3"]
+        python-version: [3.9, "3.10", "3.11", "3.12", "3.13"]
 
     steps:
       - uses: actions/checkout@v4

.vscode/settings.json

@@ -5,5 +5,5 @@
     "python3.8InterpreterPath": "/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8",
     "modulename": "${workspaceFolderBasename}",
     "distname": "${workspaceFolderBasename}",
-    "moduleversion": "1.0.4"
+    "moduleversion": "1.0.5"
 }

README.md

@@ -129,9 +129,15 @@ with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as stream:
       print(parsed_data)
 ```
 
-#### Encrypted Payloads
+### Encrypted Payloads
 
-Some proprietary SPARTN message sources (e.g. Thingstream PointPerfect © MQTT) use encrypted payloads (`eaf=1`). In order to decrypt and decode these payloads, the user must set `decode=1` and provide a valid decryption `key`. Keys are typically 32-character hexadecimal strings valid for a 4 week period. If the datastream contains messages with ambiguous 16-bit `gnssTimetag` (`timeTagtype=0`) - which generally includes all GAD messages and some OCB messages - a nominal `basedate` is also required, representing the UTC datetime on which the datastream was originally created to the nearest half day. If you're parsing data in real time, this can be left at the default `datetime.now(timezone.utc)`. If you're parsing historical data, you will need to provide a basedate representing the UTC datetime on which the datastream was originally created to the nearest half day. `pyspartn` can derive the requisite `basedate` from any 32-bit `gnssTimetag` for the same message subtype, but this is dependent on the datastream containing such 32-bit timetags. See examples below.
+At time of writing, most proprietary SPARTN message sources (e.g. Thingstream PointPerfect © MQTT) use encrypted payloads (`eaf=1`). In order to decrypt and decode these payloads, a valid decryption `key` is required. Keys are typically 32-character hexadecimal strings valid for a 4 week period.
+
+In addition to the key, the SPARTN decryption algorithm requires a 32-bit `gnssTimeTag` value. The provision of this 32-bit `gnssTimeTag` depends on the incoming data stream:
+- Some SPARTN message types (*e.g. HPAC and a few OCB messages*) include the requisite 32-bit `gnssTimeTag` in the message header (denoted by `timeTagtype=1`). Others (*e.g. GAD and most OCB messages*) use an ambiguous 16-bit `gnssTimeTag` value for reasons of brevity (denoted by `timeTagtype=0`). In these circumstances, a nominal 'basedate' must be provided by the user, representing the UTC datetime on which the datastream was originally created to the nearest half day, in order to convert the 16-bit `gnssTimeTag` to an unambiguous 32-bit value.
+- If you're parsing data in real time, this basedate can be left at the default `datetime.now(timezone.utc)`.
+- If you're parsing historical data, you will need to provide a basedate representing the UTC datetime on which the data stream was originally created, to the nearest half day.
+- If a nominal basedate of `TIMEBASE` (`datetime(2010, 1, 1, 0, 0, tzinfo=timezone.utc)`) is provided, `pyspartn.SPARTNReader` can *attempt* to derive the requisite `gnssTimeTag` value from any 32-bit `gnssTimetag` in a preceding message of the same subtype in the same data stream, but *unless and until this eventuality occurs (e.g. unless an HPAC message precedes an OCB message of the same subtype), decryption may fail*. Always set the `quitonerror` argument to `ERRLOG` or `ERRIGNORE` to log or ignore such initial failures.
 
 The current decryption key can also be set via environment variable `MQTTKEY`, but bear in mind this will need updating every 4 weeks.
 
@@ -145,7 +151,7 @@ with Serial('/dev/tty.usbmodem14101', 9600, timeout=3) as stream:
       print(parsed_data)
 ```
 
-Example - Historical file input with decryption.
+Example - Historical file input with decryption, using an known basedate:
 ```python
 from datetime import datetime, timezone
 from pyspartn import SPARTNReader
@@ -157,6 +163,24 @@ with open('spartndata.log', 'rb') as stream:
 
 ```
 
+Example - Historical file input with decryption, using a nominal TIMEBASE basedate:
+```python
+from datetime import datetime, timezone
+from pyspartn import SPARTNReader, TIMEBASE, ERRLOG
+
+with open('spartndata.log', 'rb') as stream:
+   spr = SPARTNReader(stream, decode=1, key="930d847b779b126863c8b3b2766ae7cc", basedate=TIMEBASE, quitonerror=ERRLOG)
+   for raw_data, parsed_data in spr:
+      print(parsed_data)
+
+```
+```
+... (first few messages may fail decryption, until we find a usable 32-bit gnssTimeTag ...)
+"Message type SPARTN-1X-OCB-GPS timetag 33190 not successfully decrypted - check key and basedate"
+"Message type SPARTN-1X-OCB-GLO timetag 31234 not successfully decrypted - check key and basedate"
+... (but the rest should be decrypted OK ...)
+```
+
 ---
 ## <a name="parsing">Parsing</a>
 
@@ -284,9 +308,8 @@ b's\x00\x12\xe2\x00|\x10[\x12H\xf5\t\xa0\xb4+\x99\x02\x15\xe2\x05\x85\xb7\x83\xc
 The following examples are available in the /examples folder:
 
 1. `spartnparser.py` - illustrates how to parse SPARTN transport layer data from a binary SPARTN datastream.
-1. `spartn_decrypt.py` - illustrates how to decrypt and parse a binary SPARTN log file (e.g. from the `spartn_mqtt_client.py` or `spartn_ntrip_client.py` examples below).
-1. `spartn_mqtt_client.py` - implements a simple SPARTN MQTT client using the [`pygnssutils.GNSSMQTTClient`](https://github.com/semuconsulting/pygnssutils?tab=readme-ov-file#gnssmqttclient) class. **NB**: requires a valid ClientID for a
-SPARTN MQTT service e.g. u-blox Thingstream PointPerfect MQTT.
+1. `spartn_decrypt.py` - illustrates how to decrypt and decode a binary SPARTN log file (e.g. from the `spartn_mqtt_client.py` or `spartn_ntrip_client.py` examples below).
+1. `spartn_mqtt_client.py` - implements a simple SPARTN MQTT client using the [`pygnssutils.GNSSMQTTClient`](https://github.com/semuconsulting/pygnssutils?tab=readme-ov-file#gnssmqttclient) class. **NB**: requires a valid ClientID for a SPARTN MQTT service e.g. u-blox Thingstream PointPerfect MQTT.
 1. `spartn_ntrip_client.py` - implements a simple SPARTN NTRIP client using the [`pygnssutils.GNSSNTRIPClient`](https://github.com/semuconsulting/pygnssutils?tab=readme-ov-file#gnssntripclient) class. **NB**: requires a valid user and password for a
 SPARTN NTRIP service e.g. u-blox Thingstream PointPerfect NTRIP.
 1. `rxmpmp_extract_spartn.py` - ilustrates how to extract individual SPARTN messages from the accumulated UBX-RXM-PMP data output by an NEO-D9S L-band correction receiver.

RELEASE_NOTES.md

@@ -1,6 +1,16 @@
 # pyspartn Release Notes
 
-### RELEASE 1.0.3
+### RELEASE 1.0.5
+
+CHANGES:
+
+1. Add new optional `timetags` argument to SPARTNReader & SPARTNMessage, to allow them to use any available 32-bit gnssTimeTag values from the incoming datastream in order to decrypt messages (*rather than having to provide an explicit basedate*). The `timetags` argument is a dict of the format `{0: 495763673, 1: 485866844, 3: 410283479}` where the key represents the message subType (0 = GPS, 1 = GLO, 2 = GAL, etc.), and the value represents the 32-bit gnssTimeTag value to use.
+   - If a nominal decryption basedate of `TIMEBASE` (`datetime(2010, 1, 1, 0, 0, tzinfo=timezone.utc)`), or integer `0`, is passed to SPARTNReader, it will endeavour to capture 32-bit `gnssTimeTag` values for each `msgSubtype` from the incoming data stream and pass these to SPARTNMessage to decrypt messages of the same `msgSubtype` with 16-bit gnssTimeTags (`timeTagtype=0`). 
+   - **NB:** this will only work if the data stream contains valid 32-bit `gnssTimeTag` values for the same `msgSubtype` e.g. if an HPAC message for a given `msgSubtype` precedes a GAD or OCB message for the same `msgSubType` - *until such an eventuality occurs, decryption of GAD or OCB messages may fail!*
+   - Always use `quitonerror=ERRLOG` or `quitonerror=ERRIGNORE` when setting basedate to `TIMEBASE`.
+2. SPARTMMessage will now return explicit `SPARTNDecryptionError` if unable to successfully decrypt/decode message using key and basedate provided.
+
+### RELEASE 1.0.4
 
 CHANGES:
 

docs/conf.py

@@ -57,9 +57,6 @@
 # so a file named "default.css" will overwrite the builtin "default.css".
 # html_static_path = ["_static"]
 html_last_updated_fmt = "%b %d %Y"
-html_theme_options = {
-    "display_version": True,
-}
 
 autodoc_default_options = {
     "members": True,

examples/spartn_decrypt.py

@@ -1,28 +1,53 @@
 """
 spart_decrypt.py
 
-Illustration of how to read, decrypt and parse the contents
+Illustration of how to read, decrypt and decode the contents
 of a binary SPARTN log file e.g. from an Thingstream PointPerfect
 SPARTN MQTT or NTRIP service.
 
-NB: decryption requires the key and basedate applicable at the
-time the SPARTN log was originally captured.
+NB: data stream must ONLY contain SPARTN protocol messages. If you're
+capturing a log from a Thingstream PointPerfect SPARTN MQTT service,
+disable the 'Key' and 'Assist' topics, as these return UBX protocol
+messages.
+
+At time of writing, the MQTT service uses encrypted payloads (`eaf=1`).
+In order to decrypt and decode these payloads, a valid decryption `key`
+is required. Keys are typically 32-character hexadecimal strings valid
+for a 4 week period e.g. "bc75cdd919406d61c3df9e26c2f7e77a"
+
+In addition to the key, the SPARTN decryption algorithm requires a 32-bit
+`gnssTimeTag` value. The provision of this 32-bit `gnssTimeTag` depends on
+the incoming data stream:
+- Some SPARTN message types (*e.g. HPAC and a few OCB messages*) include
+  the requisite 32-bit `gnssTimeTag` in the message header (denoted by
+  `timeTagtype=1`). Others (*e.g. GAD and most OCB messages*) use an
+  ambiguous 16-bit `gnssTimeTag` value for reasons of brevity (denoted by
+  `timeTagtype=0`). In these circumstances, a nominal 'basedate' must be
+  provided by the user, representing the UTC datetime on which the datastream
+  was originally created to the nearest half day, in order to convert the
+  16-bit `gnssTimeTag` to an unambiguous 32-bit value.
+- If you're parsing data in real time, this basedate can be set to `None` and
+  will default to the current UTC datetime `datetime.now(timezone.utc)`.
+- If you're parsing historical data, you will need to provide a basedate
+  representing the UTC datetime on which the data stream was originally
+  created, to the nearest half day, or...
+- If a nominal basedate of `TIMEBASE` (`datetime(2010, 1, 1, 0, 0, tzinfo=timezone.utc)`)
+  is provided, `pyspartn.SPARTNReader` can *attempt* to derive the requisite `gnssTimeTag`
+  value from any 32-bit `gnssTimetag` in a preceding message of the same subtype in the
+  same data stream, but *unless and until this eventuality occurs (e.g. unless an HPAC
+  message precedes an OCB message of the same subtype), decryption may fail*. Always set
+  the `quitonerror` argument to `ERRLOG` or `ERRIGNORE` to log or ignore such initial
+  failures.
 
 Usage:
 
-python3 spartn_decrypt.py infile="inputfilename.log" key="bc75cdd919406d61c3df9e26c2f7e77a", basedate=431287200
+python3 spartn_decrypt.py infile="inputfilename.log" key="bc75cdd919406d61c3df9e26c2f7e77a", \
+    basedate=431287200
 
 Basedate must be in 32-bit gnssTimeTag integer format - use date2timetag() to convert datetime.
 
 Run from /examples folder. Example is set up to use 'd9s_spartn_data.bin' file by default.
 
-FYI: SPARTNMessage objects implement a protected attribute `_padding`,
-which represents the number of redundant bits added to the payload
-content in order to byte-align the payload with the exact number of
-bytes specified in the transport layer payload length nData. If the
-payload has been successfully decrypted and decoded, the value of
-_padding should always be between 0 and 8.
-
 Created on 12 Feb 2023
 
 :author: semuadmin
@@ -33,20 +58,40 @@
 from datetime import datetime, timezone
 from sys import argv
 
-from pyspartn import SPARTNReader
+from pyspartn import SPARTNDecryptionError, SPARTNReader, TIMEBASE
+
+DEMO_DATASTREAM = "d9s_spartn_data.bin"
+DEMO_KEY = "bc75cdd919406d61c3df9e26c2f7e77a"
+
+# UNCOMMENT ONE OF THE FOLLOWING DEMO BASEDATES TO SEE THE
+# EFFECTS OF USING DIFFERENT BASEDATE VALUES ...
+
+# None (will default to current utc datetime)
+# (this will fail for any messages where timeTagtype=0)
+# DEMO_BASEDATE = None
+
+# attempt to get 32-bit gnssTimeTag from data stream
+# (this may fail for some early messages where timeTagtype=0,
+# but should work for all messages thereafter)
+DEMO_BASEDATE = TIMEBASE
+
+# the actual original basedate for the demo d9s_spartn_data.bin data stream,
+# equivalent to a 32-bit gnssTimeTag = 431287200
+# (this should work for all incoming messages)
+# DEMO_BASEDATE = datetime(2023, 9, 1, 18, 0, 0, 0, timezone.utc)
 
 
 def main(**kwargs):
     """
     Read, decrypt and decode SPARTN log file.
     """
 
-    infile = kwargs.get("infile", "d9s_spartn_data.bin")
-    key = kwargs.get("key", "bc75cdd919406d61c3df9e26c2f7e77a")
-    basedate = int(kwargs.get("basedate", 431287200))
-    if basedate == 0:  # default to now()
-        basedate = datetime.now(tz=timezone.utc)
-    counts = {"OCB": 0, "HPAC": 0, "GAD": 0}
+    infile = kwargs.get("infile", DEMO_DATASTREAM)
+    key = kwargs.get("key", DEMO_KEY)
+    basedate = kwargs.get("basedate", DEMO_BASEDATE)
+    if basedate is not None and not isinstance(basedate, datetime):
+        basedate = int(basedate)
+    counts = {"OCB": 0, "HPAC": 0, "GAD": 0, "TOTAL": 0, "ERRORS": 0}
 
     with open(infile, "rb") as stream:
         spr = SPARTNReader(
@@ -56,15 +101,33 @@ def main(**kwargs):
             basedate=basedate,
             quitonerror=0,
         )
-        for _, parsed in spr:
-            for key in counts:
-                if key in parsed.identity:
-                    counts[key] += 1
-            # print(parsed)
-            # uncomment this line for an informal check on successful decryption...
-            print(f"{parsed.identity} - Decrypted OK? {0 <= parsed._padding <= 8}")
-
-    print(f"SPARTN messages read from {infile}: {str(counts).strip('{}')}")
+        eof = False
+        while not eof:
+            try:
+                # not using iterator as we want to capture Exceptions...
+                raw, parsed = spr.read()
+                if parsed is not None:
+                    for key in counts:
+                        if key in parsed.identity:
+                            counts[key] += 1
+                    print(
+                        (
+                            f"{parsed.identity=}, {parsed.eaf=}, "
+                            f"{parsed.timeTagtype=}, {parsed.gnssTimeTag=}"
+                        )
+                    )
+                    counts["TOTAL"] += 1
+                else:
+                    eof = True
+
+            except SPARTNDecryptionError as err:
+                print(err)
+                counts["ERRORS"] += 1
+                continue
+
+    print(
+        f"\nSPARTN messages decrypted and decoded from {infile}:\n{str(counts).strip("{}")}"
+    )
 
 
 if __name__ == "__main__":

pyproject.toml

@@ -7,7 +7,7 @@ name = "pyspartn"
 authors = [{ name = "semuadmin", email = "semuadmin@semuconsulting.com" }]
 maintainers = [{ name = "semuadmin", email = "semuadmin@semuconsulting.com" }]
 description = "SPARTN protocol parser"
-version = "1.0.4"
+version = "1.0.5"
 license = { file = "LICENSE" }
 readme = "README.md"
 requires-python = ">=3.9"
@@ -85,7 +85,7 @@ disable = """
 
 [tool.pytest.ini_options]
 minversion = "7.0"
-addopts = "--cov --cov-report html --cov-fail-under 90"
+addopts = "--cov --cov-report html --cov-fail-under 95"
 pythonpath = ["src"]
 
 [tool.coverage.run]

src/pyspartn/__init__.py

@@ -9,6 +9,7 @@
 from pyspartn._version import __version__
 from pyspartn.exceptions import (
     ParameterError,
+    SPARTNDecryptionError,
     SPARTNMessageError,
     SPARTNParseError,
     SPARTNStreamError,

src/pyspartn/_version.py

@@ -8,4 +8,4 @@
 :license: BSD 3-Clause
 """
 
-__version__ = "1.0.4"
+__version__ = "1.0.5"

src/pyspartn/exceptions.py

@@ -19,6 +19,12 @@ class SPARTNParseError(Exception):
     """
 
 
+class SPARTNDecryptionError(Exception):
+    """
+    SPARTN Decryption error.
+    """
+
+
 class SPARTNStreamError(Exception):
     """
     SPATRTN Streaming error.