Nightscout CGM Import Settings fails with error

[kskandis]

Describe the bug
I know Nightscout is not recommended for CGM but I thought I would test it. Trio connects fine to my Nightscout instance but Nightscout Config - Import Settings fails with the following error:

"The data couldn't be read because it isn't in the correct format."

To Reproduce
Steps to reproduce the behavior:

1. Go to Settings
2. Click on CGM
3. Scroll down to Nightscout under Type
4. After entering URL and Secret, confirm Connect
5. Tap on Import Settings, accept prompt to allow overwrite of existing settings
6. See error

Expected behavior
Nightscout settings should be imported.

Screenshots
If applicable, add screenshots to help explain your problem.

Smartphone (please complete the following information):
iPhone SE 3rd Generation iOS 17.4.1

Setup Information (please complete the following information):

• Pump type: none yet, plan to use OmniPod Dash
• CGM type and CGM app: Nightscout 15.0.2
• Trio version, branch and git reference (see Trio Settings): dev 5faf162 Trio v0.1.0

Additional context
Unable to provide screenshot as the Nightscout CGM cannot be deleted and re-added. Configuring Nightscout seems to be a one-time only process. It would be nice to have a Delete option to allow a re-configuration of Nightscout CGM.

[aug0211]

Does your selection of mg/dl vs mmol match your NS configuration?

My hunch is perhaps your Trio preferences still has mmol set (default starting point) and your NS is using mg/dl.

If this is the case, the fix is to set your preferences in Trio to match NS. It would also indicate that the error message in Trio isn’t clear enough.

[aug0211]

Forr the nice to have part, can you delete your NS connection in the NS settings section of the app (not in the CGM section)?

[kskandis]

Does your selection of mg/dl vs mmol match your NS configuration?

Yes, Preferences - Glucose Units is mg/dL which matches my NS configuration. I retried again after following your instructions to Delete the Nightscout service and then re-added it, connected, then Import Settings. It still shows:

Import Error
The data couldn't be read because it isn't in the correct format.

I put a breakpoint in NightcoutConfigStateModel.swift ImportSettings() and the error is caught at line 281:

[dsnallfot]

Very strange. I have trouble locating that particular or any similar Import error text string in the Trio code ("The data couldn't be read because it isn't in the correct format.").

A screenshot of the Import error alert would be of great help for further troubleshooting so I can see the exact phrasing.

And also, you wrote "Pump type: none yet, plan to use OmniPod Dash". Do you run a pump simulator? (The reason im asking is that there needs to be a pump simulator or actual pump set up for the import settings to work, since basal settings are written to the pump. If that fails, it should however show another import error message than the one you're getting..

EDIT: Thank you for providing the screenshot in your post above

[kskandis]

Do you run a pump simulator?

Yes, I am using the pump simulator. I just ran Trio in the Simulator on my Mac, and it is failing in the code above.

[MikePlante1]

Have you used this Nightscout URL with iAPS or Loop before?

[dsnallfot]

And one more thing. You do not need to delete nightscout to be able to do a new import. When it works as it should, the "import setting" can be run anytime you like (if you for instance make changes to the CR/ISF/Basals In nightscout, it can be imported by Trio). However very important to always double check that its imported as expected as stated in the alert

[kskandis]

Have you used this Nightscout URL with iAPS or Loop before?

Yes, I am using DIY Loop for several years with my Nightscout URL.

Here is another screenshot of that same file for Import Settings:

[kskandis]

A screenshot of the Import error alert would be of great help for further troubleshooting so I can see the exact phrasing.

This is from my iPhone SE

[dsnallfot]

Ok. I think Mike can answer this better, but the alert is a default alert for parsing errors (that's why I didn't find that specific string anywhere in the code). I see that now.

And as stated in the alert and in the log it's a formatting mismatch (maybe because previously saved entries from loop doesn't fit 1:1 with formatting in Trio)

[kskandis]

It errors out right after it attempts to decode the JSON. Here is the parsingError from the above catch statement:
Printing description of parsingError:
▿ DecodingError
▿ typeMismatch : 2 elements
- .0 : __C.NSDecimal
▿ .1 : Context
▿ codingPath : 2 elements
▿ 0 : _JSONKey(stringValue: "Index 0", intValue: 0)
▿ rep : Rep
- index : 0
- 1 : CodingKeys(stringValue: "mills", intValue: nil)
- debugDescription : ""
- underlyingError : nil

[kskandis]

And as stated in the alert and in the log it's a formatting mismatch

I guess that means I cannot import settings from Nightscout then? I just wanted you to know this was occurring and may impact others. Thank you so much for your help!!!

[dsnallfot]

And as stated in the alert and in the log it's a formatting mismatch

I guess that means I cannot import settings from Nightscout then? I just wanted you to know this was occurring and may impact others. Thank you so much for your help!!!

We need to dig deeper in what could or could not be done to fix this.

And as you say, if this affects all users coming from Loop to Trio it is very good to know (and communicate in onboarding documentation) that it's a known limitation with the import function. That you need to have made a first upload of settings to NS from Trio (or previously iAPS) for the import function to work correctly.

But I don't know for sure. We need to look into this more.

[kskandis]

I downloaded the profile.json file from Nightscout and jsonlint.com shows that it is valid. There are several elements with "mills" and they are strings not integers or Decimal (as defined in RawFetchedProfile.swift below). Could this be the issue?

{
    "_id": "6655fb828f969503514dfdef",
    "defaultProfile": "Default",
    **"mills": "1716910977635",**
    "enteredBy": "Loop",
    "startDate": "2024-05-28T15:42:57Z",,

NightscoutConfigStateModel.swift uses:
let fetchedProfileStore = try jsonDecoder.decode([FetchedNightscoutProfileStore].self, from: data)

Defined in RawFetchedProfile.swift:
struct FetchedNightscoutProfileStore: JSON {
let _id: String
let defaultProfile: String
let startDate: String
let mills: Decimal

In NightscoutStatus.swfit it is an Int:
struct NightscoutProfileStore: JSON {
let defaultProfile: String
let startDate: Date
let mills: Int

[dsnallfot]

Ok, good catch.
I jumped into my own MongoDB library and extracted an (really) old Loop profile file, and a fresh iAPS/Trio profile file and run it through GPT4o to get help in pointing out any differences in fields and formatting, and there are a few mainly regarding number formatting, timezone and units. See attached chat answer if you like.

And as input to further investigations between devs in how to handle the user migration from loop->Trio scenarios in the best way going forward.

https://chatgpt.com/share/99f6aaea-dce3-48d0-8a0b-d1e371a1d782

Hm. The link maybe dont work. I pasted the answer below. (The first entry is Loop profile, the second is Trio)

GPT code analysis

Here are the differences in formatting of field values between the two database entries:

General Structure

• IDs:
  • First Entry: "_id":{"$oid":"605abb1bcaf7cc000486c4b1"}
  • Second Entry: "_id":{"$oid":"66569cdfec93970002d5f5a1"}

Date and Time

• startDate:
  • First Entry: "startDate":"2021-03-24T04:07:55Z"
  • Second Entry: "startDate":"2024-05-29T03:11:27.287Z"

Units

• defaultProfile:

  • First Entry: "defaultProfile":"Default"
  • Second Entry: "defaultProfile":"default"

• mills:

  • First Entry: "mills":"1616558875697"
  • Second Entry: "mills":{"$numberDouble":"1716952287000.0"}

• units:

  • First Entry: "units":"mmol/L"
  • Second Entry: "units":"mmol"

Entered By

• enteredBy:
  • First Entry: "enteredBy":"Loop"
  • Second Entry: "enteredBy":"iAPS"

Loop Settings

• loopSettings: (Only present in the first entry)
  • First Entry: Contains loopSettings with detailed configuration.
  • Second Entry: Does not have loopSettings.

Store Values

• store timezone:

  • First Entry: "timezone":"ETC/GMT-1"
  • Second Entry: "timezone":"Europe/Stockholm"

• sensitivity values (sens):

  • First Entry: Sensitivity values range from 18 to 20 with fewer time points.
  • Second Entry: Sensitivity values range from 7 to 10 with more detailed time points.

• basal values:

  • First Entry: Values range from 0.05 to 0.25 with fewer time points.
  • Second Entry: Values range from 0.2 to 0.5 with more detailed time points.

• carbratio values:

  • First Entry: Values range from 30 to 50 with fewer time points.
  • Second Entry: Values range from 15 to 25 with more detailed time points.

• target_high values:

  • First Entry: Targets have fewer time points and values are 5.5 and 6.
  • Second Entry: Target is consistently 5.5 with only one time point.

• target_low values:

  • First Entry: Values are 5 and 5.5 with two time points.
  • Second Entry: Value is consistently 5.5 with only one time point.

• delay values:

  • First Entry: "delay":"0"
  • Second Entry: "delay":{"$numberInt":"0"}

• dia values:

  • First Entry: "dia":{"$numberDouble":"6.166666666666667"}
  • Second Entry: "dia":{"$numberInt":"6"}

• carbs_hr:

  • First Entry: "carbs_hr":"0"
  • Second Entry: "carbs_hr":{"$numberInt":"23"}

Additional Fields

• created_at: (Only present in the second entry)
  • Second Entry: "created_at":"2024-05-29T03:11:27.486Z"

Key Observations

• Numeric Formatting: The first entry often uses strings for numeric values (e.g., "mills":"1616558875697"), while the second entry uses objects for numeric values (e.g., "mills":{"$numberDouble":"1716952287000.0"}).
• Units: There's a slight difference in units, with the first entry using "mmol/L" and the second entry using "mmol".
• Complex Fields: The first entry contains a more complex loopSettings field which is absent in the second entry.
• Timezone and Profile Differences: The second entry uses a different timezone and slightly different naming conventions for profiles (e.g., "defaultProfile":"default" instead of "Default").

These differences indicate variations in how the data is structured, stored, and possibly the schema version or application generating these entries.

[kskandis]

After I changed mills to a String in RawFetchedProfile.swift, "Import Settings" got further but errored out on:

"carbs_hr" which is a String on Nightscout, but Int in FetchedNightscoutProfileStore,
"Store" which is JSON on Nightscout and String: ScheduledNightscoutProfile on FetchedNightscoutProfileStore.
"delay" is also a String in Nightscout
"created_at" does not exist in Nightscout
.store["default"] does not exist in Nightscout, it is .store["Default"] //uppercase Default

Swift.DecodingError.Context(codingPath: [_JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "store", intValue: nil), _JSONKey(stringValue: "Default", intValue: nil), CodingKeys(stringValue: "carbs_hr", intValue: nil)], debugDescription: "Expected to decode Int but found a string instead.", underlyingError: nil))

keyNotFound(CodingKeys(stringValue: "created_at", intValue: nil), Swift.DecodingError.Context(codingPath: [_JSONKey(stringValue: "Index 0", intValue: 0)], debugDescription: "No value associated with key CodingKeys(stringValue: "created_at", intValue: nil) ("created_at").", underlyingError: nil))

profile.json:
"dia": 6,
"carbs_hr": "0",
"delay": "0"

    "mills": "1716742417215",
    "units": "mg/dL",
    **"store": {**
        **"Default": {**
            "target_low": [
                {
                    "time": "00:00",
                    "timeAsSeconds": 0,

Changing these in RawFetchedProfile.swift and NightscoutConfigStateModel.swift to match Nightscout resolves the "Import Settings" issue for "enteredBy": "Loop" Nightscout profiles.

[kskandis]

iAPS/Trio profile file and run it through GPT4o to get help in pointing out any differences

That is a great idea to use chatGPT for the diff!! It matched basically what I found in testing.

[dsnallfot]

Changing these in RawFetchedProfile.swift and NightscoutConfigStateModel.swift to match Nightscout resolves the "Import Settings" issue for "enteredBy": "Loop" Nightscout profiles.

One idea could be to make a conditional check for enteredBy: Trio or Loop (or not Trio) when importing and adjust the formatting accordingly. Loop->Trio isn't however the only user migration case. The previous NS profile entries could of course have been made by AAPS or previous iAPS, and there maybe isn't possible to find a proper way to handle all scenarios.

But this issue is registered and lives on this board, and some of our devs can look into this further as soon as the higher prioritized issues right now are resolved. (if you don't want to make a Pull request draft on this yourself that is?, it seems you have quickly identified the issue and possible solutions😊 )

[kskandis]

The previous NS profile entries could of course have been made by AAPS or previous iAPS, and there maybe isn't possible to find a proper way to handle all scenarios.

Yes, and it could be an old version of Nightscout with totally different data, but this scenario probably isn't too likely. Perhaps we could just re-try the JSON decoder on catch with the Loop data types? I'm not sure what format AAPS uses but I think it may be the same as Loop?

Yes, I can look into making a PR draft for this issue.

[bjornoleh]

That’s great @kskandis. You should be able to find users here with examples of both Loop and AAPS data in their NS databases.