non-activity tracked data (i.e. steps, resting heart rate)

[jperryhouts]

First of all, thank you for your work on this! This project is one of the main reasons I bought a tomtom watch in the first place!

My question is whether or not it's possible to download tracked data that isn't associated with an activity, like steps/day, heart rate, etc. I know it's stored on the watch because the android TomTom app can access it (I downloaded the app specifically to check this).

If it's possible and hasn't been implemented could you point me in the right direction? I'd be happy to contribute.

Thanks!
-Jonathan

[ryanbinns]

I assume that the information is stored in a file on the watch. I have an original Multisport watch that doesn't support this, so I can't check it myself. I guess the way to check would be to download a list of files on the watch and check the IDs against the list of file masks in ttwatch.h. Check the ones that don't match any of the known file masks with a hex editor and see if you can spot any patterns that match with what information the app tells you is there. You can get a list of files by making sure you're compiling with the unsafe functions enabled, then run ttwatch --list.

[jperryhouts]

Thanks. Do you expect that there would be ~150 files which don't match any mask in ttwatch? That seems like a lot. This is from a Spark Cardio + Music watch, if that makes a difference.

Some of them do include the relevant words (i.e. "steps", "calories", etc) but look like they're just metadata. Considering how cryptic the activity files look, even in a hex editor, I'm not sure I would recognize useful data if I saw it. Do you have any tips for poking around in these files?

[scubajorgen]

If I change the 'tracker' settings of my adventurer, I see changes in the files 0x00b1xxxx, 0x00b2xxxx, 0x00b3xxxx. Maybe 0x00b4xxxx files may be also involved . These files seem to be in the google protobuf format, just like the uploadable trails (0x00b8xxxx files). At least 0x00b1xxxx and 0x00b3xxxx seem to contain multiple records.
I deduced the google protobuf definition file for the uploadable trails. A similar reverse engineering procedure must be followed for the other files, which is a tedious job...
Like ttwatch (I guess) my software (http://blog.studioblueplanet.net/?page_id=566) contain debug functions to list the files on the watch and download all or individual files.

[jperryhouts]

You just beat me to it. I was just posting that I would try just doing a diff of all the files one day apart and see what's happened.

Thanks for the tip on protobuf format. That's a step in the right direction. I guess this is going to wait until this weekend when I have some spare time. Thanks!!

[scubajorgen]

I am interested in your findings :-)

[scubajorgen]

Use https://github.com/jmendeth/protobuf-inspector to peek into the files. Gives you the values. You just have to find out the varables... :-) Did this by hand once for the route files, not knowing the format :-(

[jperryhouts]

Thanks. I found that the other day, now I can get heartrate, timestamps, and one other field that looks like data but I havent identified yet. Wont habe time for it today but tomorrow I will probably be able to make some progress on it. Thanks again for the tips!!

[scubajorgen]

What I found out so far:
0x00b1nnnn files contain the activity tracking data records, together with some other data (device info, summaries(?), heartrates(?),...) . Each time you have connected your watch to your PC, a next file is created. So, if the latest file is 0x00b10003 during one session, the next session you find 0x00b10004 the latest file.
During the PC connection no records are added to the latest 0x00b1nnnn. Instead, records are added to 0x00b20000. This appears to be a temporary file. When the watch is detached, the data from 0x00b20000 is transferred to the next/latest 0x00b1nnnn file and this becomes the active activity log file.
When connecting to TomTom MySports, all 0x00b1nnnn files are uploaded and erased from the watch.

Apart from the 0x00b1nnnn files, the activity records (without the added data) are also stored in 0x00b3000n (A<=n<=F). I guess these form a kind of sliding buffers, one file per day. 0x00b3000a always contains the oldest data, 0x00b3000f contains todays data.. So when a day passes, 0x00b3000a is scratched, 0x00b3000b becomes 0x00b3000a, etc. I assume the watch uses these files for displaying the activity tracking (those fancy charts).

I added the protobuf definition file that can be used to decode the files. Now we've to find out the meaning of the values and unknown.... (note: you have to modify the options for the language you're using and use protoc to generate the source code)
tracker.zip

[jperryhouts]

Those look promising. I've been looking at 0x00a60003. It seems to be appended frequently.

It's huge (like 21MB, and growing) and has timestamps going back to August 1 at 10:40 am. I have no idea why that particular time is significant. Does the adventurer have that file? If so, what do you make of it? I attached mine here if you want to poke around in it.

0x00a60003.gz

[scubajorgen]

These 0x00a6nnnn files are present on my Adventurer (0x00a60000 and 0x00a60001; I cannot open your file, because it is too big; protobuf-inspector gives an out-of-memory). To me they seem internal log files or so, I cannot make sense of the values in it.
I've reverse engineered the 0x00b1nnnn/0x00b3nnnn files and adapted my application (https://github.com/scubajorgen/TomTomWatch) to list the content of all the 0x00b1nnnn files present on the watch (tracked activity, heart rates, sleep periods; note that these files are erased when using TomTom MySports Connector!!). I found out the algoritm TomTom uses to calculate the daily sleep period out of the raw data. (A sleep mode indicates the 1st hour of sleep (mode=3) and the subsequent hours of sleep (mode=4). Sum sleep seconds per blocks that contain 3 as well as 4; small sleep periods (only mode=3) are not counted).
Protobuf definition file attached explaining the meaning of the files.
tracker.zip

[derickr]

Good work - I've always wanted to look at all the other data that the watch (Runner 2) stores, and which I know you can get off. I suspected these files to contain heartrate for the whole day etc.

[jperryhouts]

Wow, fantastic! I'm sorry I disappeared for a bit, I've been traveling for work and haven't had time for much else. I'll check out your protobuf definition when I have a chance and maybe add export capability to ttwatch also. Thanks for doing that!

[derickr]

@scubajorgen — I had a look today at the 0x00b3nnnn files and deduced many of the fields:

These are the summaries of each field over a whole day if you could up the values of all records in the files, for the 1—8—1—2 records:

4: 52189 (time in day)
5: 10228 (steps)
6: 4999 (activity duration in seconds)
7: 8716 (distance in meters)
8: 1901 (calories)
9: 1089
10: 80
11: 14

I haven't figured out what 9 to 11 are yet, and how "points" are calculated. The watch shows "204" with the summary data from above.

For each record, the values:
1: seems to be a sequential number
2: the unix timestamp for when that record was made
4: the duration in seconds for which the current record stores information

[scubajorgen]

@derickr Thanx mate! I found out this already. I just discovered I did not describe this record in my reverse engineering document on my blog: https://blog.studioblueplanet.net/?page_id=627. I will add it one of these days.

1. Record number
2. Time
3. Timezone I guess (difference between time and time in UTC?)
4. Interval of this record. Often 900s=15 minutes
5. steps
6. activity duration in s
7. distance in m
8. kcal
9. kcal if the user was in rest (i.e. not performing activity)
10. ? Is this in the record??
11. ?
12. Sleep time
13. Sleep mode 1. Active 2. Charging TomTom 3. 1st hour of sleep 4. subsequent hours of sleep

[derickr]

I found out about 8 and 9 already, but I see a 10 once in a while in the summary files (0x00b3xxxx). I haven't looked at the 0x00b1xxxx files yet, as I had no need. In any case, I did just submit this PR: #153 , which uses the tracker.proto file you attached at #132 (comment)

I'm intending to look at the 0x00b1xxxx files next.

[davipatti]

I would like to contribute to this if possible. Has anyone made any more progress since the last post?

[derickr]

Yes - many parts of this were merged through #153